Candidates About

 

 Koushik A

Sr. Big Data Architect

Summary

v  Above 9+ years of Experience in Data Analysis, Data Modeling, Data Architect, Big Data/Hadoop professional with applied information Technology.

v  Proficient in Data Architecture/ DW/ Bigdata/ Hadoop/ Data Integration/ Master Data Management, Data Migration and Operational Data Store, BI Reporting projects with a deep focus in design, development and deployment of BI and data solutions using custom, open source and off the shelf BI tools

v  Experienced in Technical consulting and end-to-end delivery with architecture, data modeling, data governance and design - development - implementation of solutions.

v  Have experience in Apache Spark, Spark Streaming, Spark SQL and NoSQL databases like HBase, Cassandra, and MongoDB.

v  Expertise in configuring the monitoring and alerting tools according to the requirement like AWS CloudWatch. 

v  Proficiency in Big Data Practices and Technologies like HDFS, MapReduce, Hive, Pig, HBase, Sqoop, Oozie, Flume, Spark, Kafka.

v  Data Warehousing: Full life-cycle project leadership, business-driven requirements , capacity planning, gathering, feasibility analysis, enterprise and solution architecture, design, construction, data quality, profiling and cleansing, source-target mapping, gap analysis, data integration/ETL, SOA, ODA, data marts, Inman/Kimball methodology, Data Modeling for OLTP, canonical modeling, Dimension Modeling for data ware house star/snowflake design.

v  Experience in BI/DW solution (ETL,OLAP, Data mart), Informatica ,BI Reporting tool like Tableau and Qlikview and also experienced leading the team of application, ETL, BI developers, Testing team .

v  Experience in developing, support and maintenance for the ETL (Extract, Transform and Load) processes using Talend Integration Suite.

v  Worked on Informatica Power Center tools-Designer, Repository Manager, Workflow Manager.

v  Proficiency in multiple databases like MongoDB, Cassandra, MySQL, ORACLE and MS SQL Server.

v  Extensive use of Talend ELT, database, data set, HBase, Hive, PIG, HDFS and SCOOP components.

v  Experience in Installation, Configuration, and Administration of Informatica Power Center 8.x, 9.1 Client/Server. 

v  Experienced on Hadoop Ecosystem and Big Data components including Apache Spark, Scala, Python , HDFS, Map Reduce, KAFKA. 

v  Expertise in reading and writing data from and to multiple source systems such as oracle, HDFS, XML, delimited files, Excel, Positional and CSV files.

v  Experience in Business Intelligence (BI) project Development and implementation using Microstrategy product suits including Microstrategy Desktop/Developer, Web, Architect, OLAP Services, Administrator and Intelligence server. 

v  Logical and physical database designing like Tables, Constraints, Index, etc. using Erwin, ER Studio, TOAD Modeler and SQL Modeler.

v  Good understanding and hands on experience with AWS S3 and EC2.

v  Good experience on programming languages Python, Scala.

v  Experience in Performance tuning of Informatica (sources, mappings, targets and sessions) and tuning the SQL queries.

v  Excellent knowledge on creating reports on SAP Business Objects, Webi reports for multiple data providers. 

v  Created and maintained UDB DDL for databases, table spaces, tables, views, triggers, and stored procedures. Resolved lock escalations, lock-waits and deadlocks.

v  Experience in working with business intelligence and data warehouse software, including SSAS, Pentaho, Cognos Database, Amazon Redshift, or Azure Data Warehouse.

v  Worked on Informatica Power Center tools-Designer, Repository Manager, Workflow Manager.

v  Logical and physical database designing like Tables, Constraints, Index, etc. using Erwin, ER Studio, TOAD Modeler and SQL Modeler.

v  Extensive ETL testing experience using Informatica 9x/8x, Talend, Pentaho.

v  Experience in Dimensional Data Modeling, Star/Snowflake schema, FACT & Dimension tables.

v  Experience with relational (3NF) and dimensional data architectures. Experience in leading cross-functional, culturally diverse teams to meet strategic, tactical and operational goals and objectives.

v  Good exposure to BI reporting Microstrategy 8i, 9i & Tableau & SQL programming, RDBMS - Teradata, Oracle, and SQL server. 

v  Expertise on Relational Data modeling (3NF) and Dimensional data modeling.

v  Experience in developing Map Reduce Programs using Apache Hadoop for analyzing the big data as per the requirement.

v  Practical understanding of the Data modeling (Dimensional & Relational) concepts like Star-Schema Modeling, Snowflake Schema Modeling, Fact and Dimension tables.

 

Technical Skills:
Big Data Hadoop :	Map Reduce, HDFS, Hive, Pig, HBase, Zookeeper, Sqoop, Oozie, Flume, Scala, Akka, Kafka, Storm, Mongo DB.
Data Analysis/ Modeling Tools:	Erwin R6/R9, Rational System Architect, IBM Infosphere Data Architect, ER Studio and Oracle Designer.
Database Tools:	Microsoft SQL Server12.0 , Teradata 15.0, Oracle 11g/9i/12c and  MS Access
BI Tools:	Tableau 7.0/8.2, Tableau server 8.2, Tableau Reader 8.1,SAP  Business Objects, Crystal Reports Packages: Microsoft Office 2010, Microsoft Project 2010, SAP and Microsoft Visio, Share point Portal Server
Version Tool:	VSS, SVN, CVS.
Project Execution Methodologies:	Agile, Ralph Kimball and BillInmon data warehousing methodology, Rational Unified Process (RUP), Rapid Application Development (RAD), Joint Application Development (JAD) .
ETL/Data warehouse Tools:	Informatica 9.6/9.1/8.6.1/8.1, SAP Business Objects XIR3.1/XIR2, Web Intelligence, Talend, Pentaho.
Tools:	OBIE 10g/11g/12c, SAP_ECC6 EHP5, Go to meeting, Docusign, Insidesales.com, Share point, Mat-lab.
AWS:	AWS, EC2, S3, SQS.
Operating System:	Windows, Unix, Sun Solaris
RDBMS:	Microsoft SQL Server14.0 , Teradata 15.0, Oracle 12c/11g and MS Access.
Other Tools:	TOAD, SQL PLUS, SQL LOADER, MS Project, MS Visio and MS Office, Have worked on C++, UNIX, PL/SQL.

 

 

RESPONSIBILITIES:

v  Design/ architected and implemented complex projects dealing with the considerable data size (GB/ PB) and with high complexity.

v  Developed Scala scripts, UDF's using both Data frames/SQL and RDD/MapReduce in Spark for Data Aggregation, queries and writing data back into RDBMS through Sqoop.

v  Implemented solutions for ingesting data from various sources and processing the Data-at-Rest utilizing Big Data technologies such as Hadoop, Map Reduce Frameworks, HBase, Hive .

v  Implementation of Big Data ecosystem(Hive, Impala, Sqoop, Flume, Spark, Lambda)with Cloud Architecture.

v  Generate metadata, create Talend etl jobs, mappings to load data warehouse, data lake.

v  Developed Spark code using Scala and Spark-SQL/Streaming for faster processing of data.

v  Used Talend for Big data Integration using Spark and Hadoop.

v  Used Microsoft Windows server and authenticated client server relationship via Kerbros protocol.

v  Experience on BI reporting with At Scale OLAP for Big Data.

v  Loaded and transformed large sets of structured, semi structured and unstructured data using Hadoop/Big Data concepts.

v  Designed data analysis and visualization using BigSQL DSM and IBM Big Sheet.

v  Designed and Developed Real time Stream processing Application using Spark, Kafka, Scala and Hive to perform Streaming ETL and apply Machine Learning.

v  Identify query duplication, complexity and dependency to minimize migration efforts 
Technology stack: Oracle, Hortonworks HDP cluster, Attunity Visibility, Cloudera Navigator Optimizer, AWS Cloud and Dynamo DB.

v  Experience in AWS, implementing solutions using services like (EC2, S3, RDS, Redshift, VPC)

v  Worked on Talend Magic Quadrant for performing fast integration tasks.

v  Worked as a Hadoop consultant on (Map Reduce/Pig/HIVE/Sqoop).

v  Worked with Spark and Python.

v  Worked using Apache Hadoop ecosystem components like HDFS, Hive, Sqoop, Pig, and Map Reduce.

v  Lead architecture and design of data processing, warehousing and analytics initiatives.

v  Worked with AWS to implement the client-side encryption as Dynamo DB does not support at rest encryption at this time.

v  Exploring with the Spark for improving the performance and optimization of the existing algorithms in Hadoop using Spark Context, Spark-SQL, Data Frame, Pair RDD's, Spark YARN.

v  Used Data Frame API in Scala for converting the distributed collection of data organized into named columns.

v  Performed data profiling and transformation on the raw data using Pig and Python.

v  Experienced with batch processing of data sources using Apache Spark.

v  Developing predictive analytic using Apache Spark Scala APIs.

v  Involved in working of big data analysis using Pig and User defined functions (UDF).

v  Created Hive External tables and loaded the data into tables and query data using HQL.

v  Used Sqoop to efficiently transfer data between databases and HDFS and used Flume to stream the log data from servers.

v  Implement enterprise grade platform(mark logic) for ETL from mainframe to NOSQL(cassandra).

v  Experience on BI reporting with At Scale OLAP for Big Data.

v  Responsible for importing log files from various sources into HDFS using Flume.

v  Worked on tools Flume, Storm and Spark.

v  Assigned name to each of the columns using case class option in Scala.

v  Enhancements to traditional data warehouse based on STAR schema, update data models, perform Data Analytics and Reporting using Tableau.

v  Expert in performing business analytical scripts using Hive SQL.

v  Implemented continuous integration & deployment (CICD) through Jenkins for Hadoop jobs.

v  Worked in writing Hadoop Jobs for analyzing data using Hive, Pig accessing Text format files, sequence files, Parquet files.

v  Experience in different Hadoop distributions like Cloudera (CDH3 & CDH4) and Horton Works Distributions (HDP) and MapR.

v  Experience in integrating oozie logs to kibana dashboard.

v  Extracted the data from MySQL, AWS RedShift into HDFS using Sqoop. 

v  Developed Spark code using Scala and Spark-SQL for faster testing and data processing.

v  Imported millions of structured data from relational databases using Sqoop import to process using Spark and stored the data into HDFS in CSV format.

v  Developed Spark streaming application to pull data from cloud to Hive table.

v  Used Spark SQL to process the huge amount of structured data.

 

Environment: Spark, YARN, HIVE, Pig, Scala, Python, Hadoop, AWS, Dynamo DB, Kibana, Cloudera, EMR, JDBC, Redshift, NOSQL, Sqoop, MYSQL, Star Schema, Flume, Scala, oozie.

 

 

 

 

 

RESPONSIBILITIES:

v  Gathered the business requirements from the Business Partners and Subject Matter Experts.

v  Involved in installing Hadoop Ecosystem components. 

v  Responsible for Big data initiatives and engagement including analysis, brainstorming, POC, and architecture.

v  Wrote Sqoop scripts for moving data between Relational DBs, HDFS, and S3 storage and automated the workflow using Oozie.

v  Proof-of-concept to determine feasibility and product evaluation of Big Data products

v  Writing Hive join query to fetch info from multiple tables, writing multiple Map Reduce jobs to collect output from Hive.

v  Created high level and detail data models for Azure SQL Databases, NonSQL databases, as well as the use of storages for logging and data movement between on-premise data warehouse and cloud vNets. 

v  Designed both 3NF data models for ODS, OLTP, OLAP systems and dimensional data models using star and snow flake Schemas.

v  Developed and designed data integration and migration solutions in Azure. 

v  Implemented Spark GraphX application to analyze guest behavior for data science segments.

v  Ingest data into Hadoop / Hive/HDFS from different data sources. 

v  Used Hive to analyze data ingested into HBase by using Hive-HBase integration and compute various metrics for reporting on the dashboard.

v  Enhancements to traditional data warehouse based on STAR schema, update data models, perform Data Analytics and Reporting using Tableau

v  Involved in developing Map-reduce framework, writing queries scheduling map-reduce

v  Developed the code for Importing and exporting data into HDFS and Hive using Sqoop

v  Installed and configured Hadoop and responsible for maintaining cluster and managing and reviewing Hadoop log files.

v  Wrote ETL jobs to read from web apis using REST and HTTP calls and loaded into HDFS using Talend.

v  Generated comprehensive analytical reports by running SQL queries against current databases to conduct data analysis.  

v  Developed Shell, Perl and Python scripts to automate and provide Control flow to Pig scripts.

v  Imported data frequently from MySQL to HDFS using Sqoop. 

v  Created Hive tables and working on them using Hive QL .

v  Developed logistic regression and clustering models to predict usage consumption and exceptions for utility company on Azure HDInsight with Spark using R. 

v  Used many azure services including Azure ADFS, Azure AD, Azure AD B2C, SAML 2.0, OAuth 2.0, Data Factory, Service Bus, Application Insight, Redis Cashe, Azure SQL DB, Multi Factor Authentication, Traffic Manager, Storage, Azure App Service, Azure ASE, and many more. 

v  Designed ETL process using Talend Tool to load from Sources to Targets through data Transformations.

v  Design of Redshift Data model, Redshift Performance improvements/analysis

v  Continuous monitoring and managing the Hadoop cluster through Cloudera Manager.

v  Worked on configuring and managing disaster recovery and backup on Cassandra Data.

v  Utilized Oozie workflow to run Pig and Hive Jobs Extracted files from Mongo DB through Sqoop and placed in HDFS and processed.

v  Used Flume to collect, aggregate, and store the web log data from different sources like web servers, mobile and network devices and pushed to HDFS.

v  Involved in migration of data from existing RDBMS (oracle and SQL server) to Hadoop using Sqoop for processing data.

v  Advised the ETL and BI teams with design and architecture of the overall solution.

v  Created DDL scripts using ER Studio and source to target mappings to bring the data from source to the warehouse.

v  Work with Development, Storage and Network teams in installation and administration of MongoDB in the IT Enterprise Environment.

v   Wrote and executed various MYSQL database queries from Python using Python-MySQL connector and MySQLdb package. 

v  Developed Talend ESB services and deployed them on ESB servers on different instances. 

v  Designed and implemented the MongoDB schema. 

v   Created custom aggregations of client deliverables to meet requests using Python and pandas.

v  Developed shell scripts in UNIX environment to support scheduling of the Talend jobs. 

v  Effectively used Informatica parameter files for defining mapping variables, workflow variables, FTP connections and relational connections.

v  Implemented Change Data Capture technology in Talend in order to load deltas to a Data Warehouse.

v  Finalize the naming Standards for Data Elements and ETL Jobs and create a Data Dictionary for Meta Data Management.

v  Produced PL/SQL statement and stored procedures in DB2 for extracting as well as writing data. 

v  Wrote and executed SQL queries to verify that data has been moved from transactional system to DSS, Data warehouse, data mart reporting system in accordance with requirements.

v  Installed and configured Hadoop MapReduce, HDFS, Developed multiple MapReduce jobs for data cleaning and preprocessing.

v  Designed  development of various reports and dashboards that can be easily accessible through Tableau.

 

Environment: Pig, Sqoop, Kafka, Apache Cassandra, Oozie, Impala, Cloudera, AWS, AWS EMR, Redshift, Flume, Apache Hadoop, HDFS, Hive, Map Reduce, Cassandra, Zookeeper, MySQL, SQL Server 2008, SQL Server Analysis Services, Tableau, Oracle12c, Eclipse, Dynamo DB, PL/SQL and Python.

 

                                                                                                                 

RESPONSIBILITIES:

v  Analyzed the business requirements by dividing them into subject areas and understood the data flow within the organization 

v  Worked very close with Data Architectures and DBA team to implement data model changes in database in all environments. 

v  Developed data Mart for the base data in Star Schema, Snow-Flake Schema involved in developing the data warehouse for the database.

v  Developed enhancements to Mongo DB architecture to improve performance and scalability.

v  Created DDL scripts for implementing Data Modeling changes. Created ERWIN reports in HTML, RTF format depending upon the requirement, Published Data model in model mart, created naming convention files, co-coordinated with DBAs' to apply the data model changes. 

v  Developing predictive analytic using Apache Spark Scala APIs.

v  Assigned name to each of the columns using case class option in Scala.

v  Experienced with batch processing of data sources using Apache Spark.

v  Worked on Unit Testing for three reports and created SQL Test Scripts for each report as required.

v  Involved in developing Map-reduce framework, writing queries scheduling map-reduce. 

v  Involved in data modelling to define the table structure in MDM system.

v  Implement enterprise grade platform (mark logic) for ETL from mainframe to NoSQL (cassandra)

v  Designed and developed Data Mapping Application for 30+ disparate source systems (COBOL, MS Sql Server, Oracle, and Mainframe DB2), using MS Access, and UNIX Korn script.

v  Extensively used Erwin as the main tool for modeling along with Visio.

v  Installed, configured and administered JBOSS 4.0 server in various environments. 

v  Worked on Metadata Repository (MRM) for maintaining the definitions and mapping rules up to mark.

v  Created DDL scripts for implementing Data Modeling changes. Created ERWIN reports in HTML, RTF format depending upon the requirement, Published Data model in model mart, created naming convention files, co-coordinated with DBAs' to apply the data model changes. 

v  Extensively used Erwin as the main tool for modeling along with Visio 

v  Established and maintained comprehensive data model documentation including detailed descriptions of business entities, attributes, and data relationships. 

v  Worked on Metadata Repository (MRM) for maintaining the definitions and mapping rules up to mark.

v  Developed the Conceptual Data Models, Logical Data models and transformed them to creating schema using ERWIN.

v  Performed data cleaning and data manipulation activities using NZSQL utility.

v  Analyzed the physical data model to understand the relationship between existing tables.

v  Created a list of domains in Erwin and worked on building up the data dictionary for the company.

 

Environment: Erwin r8.2, Oracle SQL Developer, Oracle Data Modeler, Teradata 14, SSIS, Business Objects, SQL Server, ER/Studio Windows , MS Excel.

 

 

RESPONSIBILITIES:

v  Developed the logical data models a