Candidates About

 

Hetal Bhadaja

Summary:

§   Over eleven (11) years of experience in full life cycle Data Warehousing projects. 3+ years of work experience in designing and implementing complete end-to-end Big Data solution using Hadoop Ecosystem tools including HDFS, MapReduce, Hive, Pig, HBase, Zoo Keeper, Sqoop, Flume, Cassandra,Oozie, Phython, Spark and Kafka

§  Performed Infrastructure capacity planning considering various workload patterns and actively participated in hardware design for Edge, Master and DataNodes

§  Experience in installing and configuring Hadoop cluster using different distributions of Apache Hadoop like Cloudera and EMR

§  Good knowledge of Hadoop Architecture and various Hadoop 1.0 and Hadoop 2.0 components such as HDFS, MapReduce, NameNode, Secondary NameNode, Standby NameNode, DataNode, Job Tracker, Task Tracker, Resource Manager, Application Master, Node Manager and YARN

§  In depth exposure to HDFS concepts, blocks, master nodes, slave nodes, HDFS high-availability including both manual and automatic failover, command line interface, file reading and writing, copying a file from one cluster to another

§  Experience in analyzing data using Hive QL, Pig Latin and custom MapReduce programs in Java

§  Designed and Modeled Hive database using Partitioned and Bucketing tables with storing data in various file systems like Parquet, RC, ORC and Text File

§  Experience in importing and exporting batch data using Sqoop and real-time data using Flume from Relational Database Systems to HDFS and vice-versa

§  Extensively worked with Job/workflow scheduling tool Oozie and designed both time driven and data driven automated workflows

§  Created custom UDFs for Hive and Pig UDF's using Python for evaluation, filtering, loading and storing of data

§  Designed and implemented Hive and Pig Experience in monitoring infrastructure for Hadoop cluster using Nagios and Ganglia

§  Design and implemented spark cluster to analysis data in Cassandra.

§  Migrate table from SQL server to Cassnadra.

§  Good working experience in writing spark application in Scala using Spark Context, Spark SQL and Spark Streaming

§  Experience in Spark Streaming to receive real time data and to store the stream data into HDFS using Scala

§  Experienced in using Integrated Development environments like Eclipse, Intellij IDEA and Version Control tools like SVN, GIT and Build tools like Maven and SBT

§  Worked and migrated RDBMS databases into different NoSQL database

Experience working with CQL (Cassandra Query Language) on Cassandra file system.

§  Experience in Cluster maintenance processes which includes both HDFS, tasks like commissioning and decommissioning of datanode, File system check, balancing the cluster and Map Reduce tasks including commissioning and decommissioning a Task Tracker, killing of both Task Tracker and Job Tracker and dealing with blacklisted Task Tracker

§  Analyzing the clients existing Hadoop infrastructure and understand the performance bottlenecks and provide the performance tuning accordingly

§  Experience in deploying Hadoop cluster on Public and Private Cloud Environment like Amazon AWS

§  Experience in PL/SQL programming including SQL queries using stored procedures and triggers in Oracle, SQL Server using TOAD and Query Manager

§  Experience in integration of various data sources like Oracle9i/10g, DB2, Teradata, SQL Server, MS Access, XML and Flat files

§  Extensively worked on Data Warehouse design methodologies (Top-Down, Bottom-Up)

§  Solid experience in Design and Development of Extraction, Transformation, Load and Business                                                                      Intelligent processes

§  Excellent technical, communication, analytical and problem-solving skills and ability to get on well with people including cross-cultural backgrounds.

§  Extensively worked in leading ETL design, development, testing and implementation phases. Created logical and physical mapping documents and low level/high level ETL design documents using Info Sphere Information server business suites.

§  Experience working with Dimensional Modeling using Star Schema and Snow-Flake schema. Extensively worked in creating Conceptual, Logical and Physical Data Models and deployed on target databases using Erwin.

§  Experience working with Cognos Framework Manager, Reporting Studio, Analysis Studio, Query Studio and Cognos Connections

§  Developed Map/reduce streaming jobs using Python language that are implemented using Hive and Pig.

 

Skills:

§  Cloudera Enterprise CDH 5.4, Hadoop 2.6, Hive 1.1, HBase 1.0, Sqoop 2.0, Cassandra, Amazon AWS §  Spark 2.2,Spark streaming, Linux RHEL 6.x,Shell Programming, Aginity Workbench, Tableau, SAS, Oracle SQL Developer §  ETL Tools: IBM Information Server 9.1/8.7/8.5/8.1/7.5.2(EE/Server)/6.0,QualityStage        8.7/8.5/8.1/7.5x. SSIS 2008/2005, SQL*Loader & OWB §  Reporting Tools : Cognos 10.2, SSRS, OBIEE §  Dimensional Data Modeling:  Star Schema Modeling, Snow-Flake Modeling, FACT and Dimensions tables, physical and logical data modeling, Erwin 4.1.2/3.x and Oracle Designer §Programming Languages:  C, C, Python §   Databases:  MS SQL Server 2000/2005, Oracle 11g,10g/9i/8, PL/SQL, DB2 UDB,    Terdata 12/13 §    UNIX Tools:  Shell Scripts, C Shell, K-Shell Scripts, AWK and VI-editor, Perl Scripting. §  GUI:  Visual Basic 6.0/5.0/4.0, FrontPage 98/2000, Excel, Power Point, Visual Interdev 6.5/6.0 and Visual Age 3.5 §  Windows:  Windows Server 2008, Windows batch scripting

 

Project  Description:

 

New York Department of State, New York                                                                     Aug 2015 – Current

Sr. Hadoop Developer

 

§  Performed capacity planning for the Development, Test and Production Hadoop environments

§  Performed infrastructure planning and worked with Cloud vendor to setup Hadoop cluster as IaaS

§  Architected and Designed Hadoop cluster with up to 30 nodes consisting of Edge nodes, Management nodes and DataNodes

§  Extensively used Cloudera Manager to manage cluster nodes, services, administering cluster and assigning users, groups and roles for authorization

§  Configured Sqoop Client to import of data into HDFS from external Teradata / Oracle relational database management system

§  Set up and configured SFTP servers to import data from third party vendors

§  Designed and Developed Data ingestion process from various sources using Apache Sqoop, Flume, Hadoop shell commands and Kafka

§  Designed Data warehouse database on Hive

§  Created and Managed storage format strategies and Data Partitioning and Bucketing strategies for Hive and Impala tables

§  Extensively worked on extending Hive and Pig core functionality by writing custom UDF's

§  Performed Hive performance optimization using various techniques like Parallel execution, Partitioning, Bucketing, Indexing and Map-side Join

§  Designed and Developed Data Pipelines (ETL processes) in Apache Spark using Spark Context, Spark SQL and Spark Streaming

§  Used Spark Streaming to divide streaming data into batches as an input to spark engine for batch processing

§  Extensively used Intellij IDEA as integrated development environment, Git for version control and Maven for the project build

§  Hands on experience in writing spark application for ETL using various Spark APIs

§  Experience to build high performance algorithms for Big data sets

§  Extensively worked on RDD Lineage, Caching and Distributed persistence

§  Improved the performance of spark applications using various techniques like Broadcast Variable, Accumulators, Kryo Serialization, Repartition/Coalesce

§  Implemented multi-layer data security. Worked with Network security team to setup firewall rules. Worked with System Admin to setup LDAP integration for authentication

§  Designed and developed role based authorization and Data encryption at rest and in Motion

§  Certified seamless integration of third party tool like Aginity, Tableau and SAS with Hive and Impala

§  Created work flows with Oozie scripts to schedule and run jobs

§  Monitored Hadoop cluster using Ganglia

 

 Environment:

Cloudera Enterprise CDH 5.4, Hadoop 2.6, Hive 1.1, HBase 1.0, Sqoop 2.0, Spark 1.3, Linux RHEL 6.x, Oracle 11g, Shell Programming, Aginity Workbench, Cassandra,Tableau, SAS, Oracle SQL Developer

 

 New Jersey Department of Transportation, Trenton                                                April 2014 – July 2015

DataStage/ETL Lead

 

NJDOT was established in 1966 as the first State transportation agency in the United States. It has been responsible for maintaining and operating the State's highway and public road system, planning and developing transportation policy and assisting with rail, freight and intermodal transportation issues. The project is to build Enterprise Data Warehouse for integrating data from all heterogeneous sources and build Datamart for Transit Information.

 

Project 1:

·               Provided use case for Centralize Staging Area on Hadoop cluster

·               Provided proof of concept on virtual single node Hadoop instance

·               Performed capacity planning for Hadoop cluster for all environments (Dev, Test, Prod)

·               Planned the Hadoop cluster selecting hardware and operating system, Configuring disk and designing network

·               Participated in installing and configuring Cloudera (CDH4) SuSE package on 40 node cluster

·               Configured Rack awareness and Namenode high availability and federation

·               Performed Maintenance of Hadoop cluster - Starting and Stopping services, Adding and Decommissioning data nodes, Dealing with replication storm and Balancing HDFS block data

·               Designed and implemented Backup and Recovery strategy for Data and Namenode metadata

·               Extensively worked on Hadoop ecosystem products - HDFS, Map Reduce, Hive, Impala, Hbase and Oozie

·               Extensively worked on HiveQL to create and modify databases, create external and managed partitioned tables, Loading data into tables, Exporting data.

·               Created Hive views to reduce query complexity and restrict access to data based on conditions

·               Created Hive indexes on columns to speed some operations

·               Used Impala to run some ad-hoc queries

·               Wrote MapReduce programs in Java to process semi-structured xml and unstructured text files

·               Designed and Developed Shell Scripts and Sqoop Scripts to migrate data in and out of HDFS

·               Designed and Developed Oozie workflows to execute MapReduce jobs, Hive scripts, shell scripts and sending email notifications

 

Project 2:

·            Worked closely with the ETL Lead, Technical Lead, Data Modeler, Business Analysts to understand business requirements, providing expert knowledge and solutions on Data Warehousing, ensuring delivery of business needs in a timely cost-effective manner

·            Understood the Functional Requirements and Transformations on the source data and prepared Technical Specification

·            Prepared project estimation based on number of jobs in each interface and job complexities

·            Based on Technical Specification Document designed and created Datastage jobs to extract, transform and load data from source into Data warehouse and then into Datamarts

·            Loaded Datamart with PL/SQL only.

·            Worked on pipeline and partitioning parallelism techniques and ensured load balancing of data

·            Deployed different partitioning methods like Hash by field, Round Robin, Entire, Modulus, and Range for bulk data loading

·            Implemented Slowly Changing Dimension Type- 2 in Datastage

·            Extensively used DataStage Designer stages such as Aggregator, Transformer, Join, Dataset, Lookup, Funnel, Peek, Pivot

·            Implemented various shared containers for Re-using the business functionality

·            Followed Datastage coding standards and best practices

·            Performed Unit Testing in Data Stage and tracked results in unit Test cases (UTC)

·            Performed extensive performance tuning in Datastage, to handle big volume of data

·            Lead the other team members and guided the testing team members in executing the jobs, which helped them to develop their knowledge on the tool

·            Performed code-fix in Datastage for the defects rose from system testing and the Business System Analyst (BSA's)

·            Addressed defects on Datastage code added in Quality Center

·            Responsible for data analysis, requirements gathering, source-to-target mapping, process flow diagrams and documentation

·            Created UNIX shell scripts for File Transfer and File validation during parallel job execution

·            Tuned SQL queries for better performance for processing business logic in the database

 

Skills:

Hadoop 2.6, Hive 1.1, HBase 1.0, Sqoop 2.0, Spark 1.3, Linux RHEL 6.x, Oracle 11g, Shell Programming, Aginity Workbench, Tableau, SAS

IBM / Ascential DataStage 8.7,8.5, Oracle11g, SQL Server 2012, MS Word ,MS Access, Unix Windows 2007, Erwin 4.1.

 

 

Kaiser Permanente, Pleasanton, CA                                                                           Nov 2013 – Feb 2014

DataStage/ETL Lead

 

§  Prepared technical specifications and project estimates.

§  Designed and created DataStage jobs to extract, transform and load data from source into data warehouse and data marts.

§  Worked on pipeline and partitioning parallelism techniques and ensured load balancing of data.

§  Deployed different partitioning methods like Hash by field, Round Robin, Entire, Modulus, and Range for bulk data loading.

§  Implemented Slowly Changing Dimension Type- 2 in DataStage.

§  Extensively used DataStage Designer stages such as Aggregator, Transformer, Join, Dataset, Lookup, Funnel, Peek, and Pivot.

§  Implemented various shared containers for re-using the business functionality.

§  Involved in import and export using Data Stage Manager.

§  Responsible for data analysis, requirements gathering, source-to-target mapping, process flow diagrams, and documentation. 

§  Created UNIX shell scripts for File Transfer and File validation during parallel job execution.

§  Analyzed the Data Acquisition, Data Integration, Data Transformation, and  Data Delivery processes for Performance Assessment.

§  Modified existing ETL and BI architecture to achieve maximum parallel processing and reducing overhead on Database servers.

§  Tuned Database queries using various Oracle tuning techniques.

§  Partitioned and sub-partitioned tables choosing appropriate partition keys in Oracle.

§  Analyzed Job Monitor, Score Dump, Resource Estimation and Performance Analysis to identify bottlenecks in DataStage jobs.

§  Lead the implementation of performance improvement recommendations.

§  Use quality stages for data.

 

 

Johnson Controls Inc, Glendaled WI                                                                          Dec 2011 – Nov 2013 

DataStage/ETL Lead

 

§  Gathered user and business requirements through interviews, surveys, prototyping and observing.

§  Created conceptual model, business process diagrams, process and data flow diagrams and converted business requirements into logical models.

§  Converted logical models into physical models and created database script for implementing physical model into target database.

§  Used ERWIN 4.1/r7/r8 to develop conceptual, logical and physical data models and Implemented into target databases using complete compare and forward engineering.

§  Used Kimball Methodology (BUS Architecture) to design the data marts in dimensional modeling.

§  Created metadata repositories to maintain business, application, technical and process metadata.

§  Set up four layered ETL architecture including Functional Layer, Operational and Management Layer, Audit Balancing and Control (ABC) layer and Common component layer.

§  Analyzed existing informational sources and methods to identify problem areas and make recommendations for improvement.

§  Created mapping documents for architecture and ETL purpose.

§  Designed and developed Data stage Jobs to extract data from heterogeneous sources. Applied transform logic to extracted data and loaded into Data Warehouse Database.

§  Installed and configured DataStage on Grid Environment to achieve High Availability project requirement.

§  Migrated projects to new DataStage environment installed on Grid.

§  Modified DataStage jobs to add Grid configuration related parameters and modified datasage sequence to generated one configuration file at sequence level.

§  Wrote a routine to perform maintenance of grid to clean configuration files auto-generated by Resource Manager.

§  Designed and Documented actions to be performed in case of Head node failure and Compute node failure on Grid.

§  Designed and Developed Change Data Capture (CDC) process to extract delta data from ODS.

§  Implemented Federated architecture to lookup data from MDM. 

§  Used DataStage stages namely Sequential file, Transformer, Aggregate, Sort, Datasets, Join, Lookup, Change Capture, Funnel, Peek, Row Generator stages, Slowly Changing Dimension Type2, various Database stage to develop parallel jobs.

§  Used QualityStage to ensure consistency, removing data anomalies and spelling errors of the source information before being delivered for further processing.

 

 

State Street Bank, Boston MA                                                                                      Jan 2010 – Oct 2011

DataStage/ETL Developer

 

§  Developed Relational model for OLTP system, Demoralized model for Data Warehouse and Dimensional model (Star and Snow Flake) for different data marts using Erwin.

§  Identified and documented data sources and transformation rules required to populate and maintain data warehouse content and created source to target Logical Mapping Document.

§  </