Candidates About

                                                                        Raghav Mehta

                                                                                       Bigdata/Hadoop Developer

·         9+ Years of professional experience in IT which includes 4 years of comprehensive experience in working with Apache Hadoop Eco-components, Spark streaming and Amazon Web services(AWS).

·         Having Good Working Expertise on handling Terabytes of structured and unstructured data on significantly big Cluster Environment.

·         Good understanding of HDFS Designs, Daemons, HDFS High Availability (HA).          

·         Good understanding and working experience on Hadoop Distributions like Cloudera and Hortonworks.

·         Good Knowledge in creating event-processing data pipelines using flume, Kafka and Storm.

·         Expertise in data transformation & analysis using SPARK, PIG, HIVE

·         Used Apache Solr to Query Indexed Data for Analytics for boosting search catalyst operations.

·         Implemented MLib functions for training and building fully loaded classifiers models using Spark streaming, Spark SQL and Machine Learning APIs.

·         Worked on real time data integration using Kafka – Storm data pipeline, Spark streaming and HBase.

·         Worked on ingesting, reconciling, compacting and purging base table and incremental table data using Hive and HBaseand job scheduling through Oozie.

·         Hands on experience with AWS components like VPC, EC2, EBS, RedShift, CFT.

·         Worked on all major components of Hadoop Eco-components such as HDFS, HIVE, PIG, Oozie, Sqoop, Map Reduce and YARNon Cloudera, MapRand Hortonworks distributions. 

·         Worked on setting up AWS EMR, EC2 clusters andMulti-Node Hadoop Clusterinside developer environment.

·         Developed scripts and batch jobs to monitor and schedule various Spark jobs.

·         Worked on ingesting, reconciling, compacting and purging base table and incremental table data using Hive and HBase and job scheduling through Oozie.

·         Worked on importing and exporting data from different databases like Oracle, Teradata, MySQL, and IBMDB2 into HDFS and Hive usingSqoop. 

·         Worked on collecting stream data into HDFS using Kafka, Flume and Flink.

·         WrittenSpark SQL, Cassandra QL, Hive QL and PIG Latin queries for data analysis to meet the business requirements.

·         Created baseand incremented tables with partitioning, bucketing of table and creating UDF’s in Hive using optimisation and tuning.

·         Worked on NoSQL databases such as HBase, MongoDB and Cassandra.

·         Experience with Oozie Workflow Engine to automate and parallelize Hadoop Map/Reduce, Hive and PIG MR Frameworks.

·         Implemented automatic workflows and job scheduling using Oozie, Zookeeper and Ambari

·         Good working experience on Hadoop Cluster architecture and monitoring the cluster. In-depth understanding of Data Structure and Algorithms.

·         Build and configured Apache TEZ on Hive and PIG to achieve better responsive time while running MR Jobs.

·         Extending Hive and Pig core functionality by writing custom UDFs, UDTF and UDAFs.

·         Implemented Proofs of Concept on Hadoop stack and different big data analytic tools, migration from different databases to Hadoop

·         Experienced in working with spark eco system using Spark SQL and Scala queries on different formats like Text file, Avro Parquet files.

·         Worked on Software Development Life Cycle (SDLC)

·         Experienced in creating PL/SQL Stored Procedures, Functions, Cursors against Oracle (10g, 11g)

·         Key participant in all phases of software development life cycle with Analysis, Design, Development, Integration, Implementation, Debugging, and Testing of Software Applications in client server environment, Object Oriented Technology and Web based applications.

 

 

TECHNICAL SKILLS

BIGDATA Technologies	Apache Spark, ApacheSolr, HDFS, MapReduce, Pig, Sqoop, Hive, HBase, Kafka, Flume, OOZIE, YARN
AWS Components	VPC, EC2, EBS, RedShift, CFT, EMR, Cloudwatch
Programming Languages	Scala, Python, R and Java
Java Technologies	STRUTS, J2SE, JDBC, Java Script, HTML, XML
Servers	Web Logic and Tomcat
Java IDEs	Lucidworks Fusion, NetBeans, Eclipse.
Databases	Oracle10g, PL/SQL, MYSQL, HBase, MongoDB, NOSQL, Cassandra
Operating Systems	Windows7, Windows XP, Unix and Linux, CentOS, Solaris

 

 

CERTIFICATIONS

·         CCA 175 Cloudera Certified Spark Developer Associate.

 

 

EDUCATION SUMMARY

·         Bachelor of Technology in Electronics and communications Engineering, JNTU.

 

           

PROFESSIONAL EXPERIENCE

 

Regions Financial Corporation, Birmingham, AL                                                                   Jan 2017 to Present

Big Data/Hadoop Developer

 

Responsibilities: -

·         Involved in complete project life cycle starting from design discussion to production deployment.

·         Installed Hadoop, Map Reduce, HDFS, AWS and developed multiple MapReduce jobs in PIG and Hive for data cleaning and pre-processing.

·         Developed a job server (REST API, Spring boot, ORACLE DB) and job shell for job submission, job profile storage, job data (HDFS) query/monitoring.

·         Implemented solutions for ingesting data from various sources and processing the Data utilizing Big Data Technologies such as Hive, Pig, Sqoop, Hbase, Map reduce, etc.

·         Design and develop a daily process to do incremental import of raw data from DB2 into Hive tables using Sqoop.

·         Involved in debugging Map Reduce job using MR Unit framework and optimizing Map Reduce.

·         Extensively used Hive/HQL or Hive queries to query data in Hive Tables and loaded data into Hive tables.

·         Developed data pipeline using Flume, Sqoop, Pig and MapReduce to ingest data into HDFS for analysis.

·         Used Oozie and Zookeeper for workflow scheduling and monitoring.

·         Effectively used Sqoop to transfer data from databases (SQL, Oracle) to HDFS, Hive.

·         Integrated Apache Storm with Kafka to perform web analytics. Uploaded click stream data from Kafka to HDFS, Hbase and Hive by integrating with Storm.

·         Designed Hive external tables using shared meta-store instead of derby with dynamic partitioning &buckets.

·         Worked on Big Data Integration &Analytics based on Hadoop, SOLR, Spark, Kafka, Storm and web Methods.

·         Worked on migrating MapReduce programs into Spark transformations using Spark and Scala.

·         Design & implement ETL process using Talend to load data from Worked extensively with Sqoop for importing and exporting the data from HDFS to Relational Database systems/mainframe and vice-versa. Loading data into HDFS.

·         Developed PIG Latin scripts to extract the data from the web server output files to load into HDFS.

·         Created concurrent access for hive tables with shared/exclusive locks enabled by implementing Zookeeper in cluster.

·         Strongly recommended to bring in Elastic Search and was responsible for installing, configuring and administration.

·         Implemented using SCALA and SQL for faster testing and processing of data. Real time streaming the data using with KAFKA.

·         Used OOZIE Operational Services for batch processing and scheduling workflows dynamically.

·         worked on creating End-End data pipeline orchestration using Oozie.

·         Populated HDFS and Cassandra with massive amounts of data using Apache Kafka.

·         Involved in design and developed Kafka and Storm based data with the infrastructure team.

·         Worked on major components in Hadoop Ecosystem including Hive, PIG, HBase, HBase-Hive Integration, Scala, Sqoop and Flume.

·         Developed Hive Scripts, Pig scripts, Unix Shell scripts, programming for all ETL loading processes and converting the files into parquet in the Hadoop File System.

·         Worked with Oozie and Zookeeper to manage job workflow and job coordination in the cluster.

 

Environment: Hadoop, Hive, Impala, Oracle, Spark, Pig, Sqoop, Oozie, Map Reduce, SQL

 

 

Apple, Santa Clara, CA                                                                                                                        Sep 2015 to Dec 2016

Bigdata/Hadoop Developer

 

Responsibilities:

·         Experience of gathering business requirements from customers, analyze them and point out the impacts.

·         Created sessions, tasks and workflows using Alteryx.

·         Used Alteryxto perform end to end ETL process which included developing mappings, extracting data from sources (Flat files & Oracle Database 11g), cleansing and transforming the data to specific business requirements/rules, and loading the data into target.

·         Worked on big data such as Streamline analytics for building predictive models inside Machine Learning using Scala, Python and R.

·         Created Technical Documentation for all the developed mappings.

·         Used Apache Kafka for handling real-time user data feeds.

·         Used Kafka Direct Stream API to connect Kafka with Spark Streaming.

·         Developed Spark Streaming application in Java that connects to Kafka and fetch data in real-time.

·         Used Kafka to load data streaming in to HDFS.

·         Developed multiple Kafka Producers and Consumers as per the software requirement specifications.

·         Continuous monitoring and managing the Hadoop cluster using Cloudera Manager.

·         Used Hue for UI based PIG script execution, Tidal scheduling and creating tables in Hive.

·         Worked with BSA team in generating the reports on Tableau.

·         Experience working with spark machine learning and SPSS.

·         Worked on Sequence files, RC files, Map side joins, bucketing, partitioning for Hive performance enhancement and storage improvement.

·         Involving in Analysis, Design, Implementation and Bug Fixing Activities. Handling structured and unstructured data and applying ETL processes. Responsible for building scalable distributed data pipelines using Hadoop.

·         Worked on developing ETL processes to load data from oracle data source to HDFS Sqoop, perform structural

·         Worked on requirement gathering, analysis and translated them into technical design.

·         Used Jenkins scheduler to schedule the ETL workflows.

·         Extensively used UNIX for shell Scripting and pulling the Logs from the Server.

·         Conduct daily stand up with offshore team: updating them with applicable tasks & getting updates for the onshore team on a day-to-day basis.

·         Co-ordinate with offshore and onsite team to understand the requirements and prepare High level and Low-level design documents from the requirements specification.

 

Environment: HDFS, Spark, ETL, Sqoop, Jenkins, Collibra, Oracle Database, Cloudera Distribution for Hadoop3(cdh3u5), Linux, Kafka

 

 

Adobe, San Jose, CA                                                                                                                                 Jun 2014 to Aug 2015

Bigdata/Hadoop Developer

 

Responsibilities: 

·         Worked on Streamline analytics and data consolidation projects on the product Lucidworks Fusion.

·         Integrated Kafka, Spark, Scala and HBasefor streamline analytics for creating a predictive model and implemented Machine Learning protocol.

·         Developed Scala scripts, UDFFs using both Data frames in Spark for Data Aggregation, queries and writing data back into OLTP system through Sqoop.

·         Used Solr/Lucene for indexing and querying the JSON formatted data.

·         Handled cloud operations inside Rackspacefor persistence logic.

·         Monitored OOTB requests with ATG, Akamai and TIBCO.

·         Used REST services for handling unfinished jobs, knowing the status and creating a dataset inside a URL.

·         Worked on ingesting, reconciling, compacting and purging base table and incremental table data using Hive and HBaseand job scheduling through Oozie.

·         Designed and implemented streaming data on UI with Scala.js 

·         Utilized DevOps principle components to ensure operational excellence before deploying in production.

·         Operating the cluster on AWS by using EC2, Akka, EMR, S3 and cloudwatch.

·         Transported data to HBase using Flume.

·         Used Java UDFs for performance tuning in Hive and Pig by manually driving the MR part.

·         Used Java APIs such as machine learning library functions, graph algorithms for training and predicting the linear model in spark streaming.

·         Have implemented unit testing in Java for pig and hive applications.

·         Developed workflow in Oozie to automate the tasks of loading the data into HDFS and pre-processing with Pig.

·         Involved in source system analysis, data analysis, and data modeling to ETL (Extract, Transform and Load).

·         Written Spark programs to model data for extraction, transformation and aggregation from multiple file formats including XML, JSON, CSV& other compressed file formats.

·         Worked on loading the RDBMS data onto Hadoop/HDFS, using sqoop.

·         Developing the Pig scripts / UDFs to manipulate/transform the loaded data. 

·         Creating Hive tables, loading data into Hive using UDFs, partioning and bucketing principles.

·         Migration Hive data into Hbase. 

·         Involved in integrations among Pig, Hive and Hbase. 

·         Instrumental in debugging. Created Hive External tables to store the processed data from Map Reduce

 

Environment: Hadoop0.20, HDFS, Hive,Pig, Sqoop, Java, Cloudera Distribution for Hadoop3(cdh3u5), Linux.

 

 

Bed Bath& Beyond, Northern, NJ                                                                                            Apr 2013 to May 2014

Spark/Hadoop Developer                                   

           

Responsibilities:

·         Developed multiple Spark jobs in PySpark for data cleaning and Pre-processing.

·         Analyzed large data sets by running Hive queries and Pig scripts.

·         Involved in creating Hive tables and loading and analyzing data using Hive queries.

·         Developed simple/complex MapReduce jobs using Hive and Pig.

·         Migrated existing MapReduce programs to Spark using Scala and Python

·         Loaded and transformed large sets of structured, semi structured and unstructured data.

·         Involved in running Hadoop jobs for processing millions of records of text data.

·         Worked with application teams to install Operating Systems, Hadoop updates, patches, and version upgrades as required.

·         Used APIs such as machine learning functions, graph algorithms for training and predicting the linear model in spark streaming.

·         Wrote the PIG UDF in for converting Date format and time stamp formats from the unstructured files to required date formats and processed the same.

·         Created 30 buckets for each Hive table based on clustering by client Id for better performance (optimization) while updating the tables.

·         Written the Apache PIG scripts to process the HDFS data.

·         Created Hive tables to store the processed results in a tabular format.

·         Developed the Sqoop scripts in order to make the interaction between Pig and SQL Database.

·         Writing the script files for processing data and loading to HDFS

·         Writing CLI commands using HDFS.

·         Developed the UNIX shell scripts for creating the reports from Hive data.

·         Completely involved in the requirement analysis phase.

·         Moved all log/text files generated by various products into HDFS location

·         Created External Hive Table on top of parsed data

 

Environment: CDH4, MapReduce, HDFS, Hive, Pig, Sqoop, Linux, XML, PL/SQL, SQL connector

 

 

General Motors, Grand Blanc, MI                                                                                               Jan 2012 to Mar 2013

Hadoop Developer                                                   

 

Responsibilities:

·         Importing and exporting data in HDFS and Hive using Sqoop

·         Extracted files from MongoDB through Sqoop and placed in HDFS and processed

·         Written Hive UDFs to extract data from staging tables

·         Analyzed the web log data using the HiveQL and process through Flume

·         Responsible for loading the customer's data and event logs from Oracle database, MySQL, Teradata into HDFS using Sqoop

·         Implementing MapReduce programs to analyze large datasets in warehouse for business intelligence

·         End-to-end performance tuning of Hadoop clusters and Hadoop MapReduce routines against very large data sets.

·         Responsible for building scalable distributed data solutions using Hadoop.

·         Handled importing of data from various data sources, performed transformations using Hive, MapReduce.

·         Experience in optimization of MapReduce algorithm using combiners and partitions to deliver the best results and worked on Application performance optimization.

·         Written the Spouts and Bolts after collecting the real stream customer data from Kafka broker to process and store into HBASE.

·         Installed and configured Hive, Pig, Sqoop, Flume andOozie on the Hadoop cluster.

·         Developed simple to complex Map/Reduce jobs using Java, and scripts using Hive and Pig.

·         Analyzed the data by performing Hive queries (HiveQL) and running Pig scripts (Pig Latin) for data ingestion

·         Experienced on loading and transforming of large sets of structured and semi structured data.

·         Managing and Reviewing Hadoop Log Files, deploy and Maintaining Hadoop Cluster.

·         Involved in creating Hive tables, loading with data and writing Hive queries.

·         Created data-models for customer data using the Cassandra Query Language.

·         <span style=