Candidates About

Professional Summary

•     Overall 3+ years of experience with strong emphasis on Design, Development, Implementation, Testing and Deployment of Software Application.

·         2+ years in Big Data, Hadoop, HDFS, MapReduce, YARN, Hadoop Ecosystem

•   Highly capable for processing large sets of Structured, Semi-structured and Unstructured datasets and            supporting Big Data applications.

•     Hands on experience with Hadoop Ecosystem components like Map Reduce (Processing), HDFS (Storage), YARN, Sqoop, Pig, Hive, HBase, Oozie, ZooKeeper and Spark for data storage and analysis

•        Expertise in transferring data between a Hadoop ecosystem and structured data storage in a RDBMS such as MY SQL, Oracle, Teradata and DB2 using Sqoop.

•        Experience in developing MapReduce jobs in Java for data cleaning and preprocessing.

•        Expertise in writing Pig Latin, Hive Scripts and extended their functionality using User Defined Functions (UDF's).

•        Expertise in handling arrangement of data within certain limits (Data Layout's) using Partitions and Bucketing in Hive.

•     Experience in NoSQL databases like Mongo DB, HBase and Cassandra.

•     Performance tuning in Hive& Impala using multiple methods but not limited to dynamic partitioning, bucketing, indexing, file compressions, and cost based optimization, etc.

·         Hands on experience handling different file formats like Json, AVRO, ORC and Parquet.

·         Hands on experience with Spark using Java, Scala and Python.

·         Strong knowledge on DevOps tools and techniques like Jenkins, Docker, Puppet, Ansible.

·         Hands on experience in Spark architecture and its integrations like Spark SQL, Data Frames and Datasets API.

·         Hands on experience with Amazon Web Services(AWS) cloud services like EC2, EMR, S3, EBS, RDS and VPC.

·         Hands on experience spinning up different AWS instances including EC2-classic and EC2-VPC using cloud formation templates.

·         Hands on experience in application development using Java, RDMS and UNIX shell scripting.

·          Experience in Object Oriented Analysis Design (OOAD) and development.

·         Experience in Java, NodeJS, Servlets, Web Logic, Web Sphere, Hibernate, Spring, JBoss, JDBC, RMI, Java Script, Ajax, JQuery, XML, HTML and have the strong understanding of Data Structures & Algorithms

Work Experience:

Client:  Tista Tech, Rockville, Maryland

Role:  Big Data Engineer                                                                                                                      April 2018 - Present

Description: Tista is a proud and supporter of the health IT domain by providing leadership and innovation in area of electronic health records(EHRs), privacy and security, E-Health, health quality and convenience, and Health IT strategic planning. TISTA is provider and patient quality and resources, to building a Health IT strategic plan for an NIH center in its effort meet its constituents needs.

Responsibilities:

·         Responsible Worked on building hadoop cluster in AWS Cloud on multiple EC2 instances.

·         Used Amazon Simple Storage Service(S3) for storing and accessing data to hadoop cluster.

·         Migrating the needed data from Oracle, MySQL in to HDFS using Sqoop and importing various formats of flat files in to HDFS.

·         Working on a cluster of size 200 nodes and 2 petabyte capacity.

·         Extensively using YAML for configuration.

·         Writing new rules and modifying existing rules by using Java and Scala

·         Working with Spark using pyspark and scala.

·         Using Git as a repository for the application project folders and JIRA for trouble tickets and Confluence for our Knwledge base.

·         Upgraded the hadoop system from CDH4 to CDH5 for better performance.

·         Ingested transactional data from Oracle into HDFS using sqoop

·         Performed data profiling and quality validation using transient and staging tables in Hive.

·         After all the actions are done, data is loaded into the staging tables.

·         Developed custom Apache spark programs for data validation to filter unwanted data and cleanse the data.

·         Ingest traditional RDBMS data into the HDFS from the existing SQL server using Sqoop.

Environment: Hive, Sqoop, Linux, Cloudera CDH 5, Scala, Pig, Spark, Zookeeper, HBase, Tableau and SQL Server, AWS

Client: Apptium Technologies, MO, US                                                                             May 2017- March 2018

Role:  Big Data Engineer

Description: Apptium provides an intuitive, configurable platform that allows you to create experiences that         delight customers and improve the way you deliver services. Apptium simplifies and automates your operations so that you can reduce costs and improve business agility. The Big data platform includes drag & drop chart widgets insights from your data. You can tie them to data from your other systems via APIs to create powerful, custom dashboards.

Responsibilties:

•    Responsible for writing MapReduce jobs to perform operations like copying data on HDFS and defining job flows on EC2 server, load and transform large sets of structured, semi-structured and unstructured data.

·         Used Sqoop for importing and exporting data from Teradata into HDFS and Hive.

·         Imported Hive tables into Spark SQL context and converted into RDDs.

·         Used Data Frames and Datasets APIs for performing analysis on Hive tables.

·         Involved in creating Hive tables, loading with data and writing hive queries which will run internally in map reduce pattern.

·         Developed Spark SQL scripts using Scala to perform transformations and actions on RDD’s in spark for faster data Processing. 

·         Experienced in performance tuning of Spark Applications for setting right Batch Interval time, correct level of Parallelism and memory tuning.

·         Responsible for analyzing the performance Hive queries using Impala.

·         Developed Flume ETL job for handling data from HTTP source and sink as HDFS.

·         Experienced in using Kafka as a data pipeline between JMS (Producer) and Spark Streaming Application (Consumer).

·          Worked on building hadoop cluster in AWS Cloud on multiple EC2 instances.

·         Used Amazon Simple Storage Service(S3) for storing and accessing data to hadoop cluster.

·         Migrating the needed data from Oracle, MySQL in to HDFS using Sqoop and importing various formats of flat files in to HDFS.

•     Worked  with  NoSQL  databases like HBase in creating HBase tables  to  load  large  sets  of  semi  structured  data coming from various sources.

·         Involved in exploration of new technologies like AWS, Apache Flink, and Apache NIFI etc which can increase the business value.

Environment: , Yarn, MapReduce, Oracle, Teradata, Sqoop, Oozie, Hive, Impala, HBase, Flume, Spark Streaming, Spark SQL, Scala, Eclipse, Cloudera, AWS, pyspark, S3, EC2.

Client: Tata Consultancy Servies(TCS), Banglore, India                                                                June 2014-July 2016

Role: Jr. Hadoop /ETL Developer                                                                                                                             

Responsibilties:

•     Performed performance tuning and troubleshooting of MapReduce jobs by analyzing and reviewing Hadoop log files.

•     Involved Low level design for MR, Hive, Impala, Shell scripts to process data.

•     Involved in complete Big Data flow of the application starting from data ingestion upstream to HDFS, processing the data in HDFS and analyzing the data. 

•     Used Spark Streaming API with Kafka to build live dashboards; Worked on Transformations & actions in RDD, Spark Streaming, Pair RDD Operations, Check-pointing, and SBT.

•     Creating Hive tables to import large data sets from various relational databases using Sqoop and export the analyzed data back for visualization and report generation by the BI team.    

•     Developed a process for the Batch ingestion of CSV Files, Sqoop from different sources and also generating views on the data source using Shell Scripting and Python.

•     Implemented partitioning, dynamic partitions and buckets in HIVE.

•     Involved in the development of Spark Streaming application for one of the data source using Scala, Spark by applying the transformations.

•     Developed a script in Scala to read all the Parquet Tables in a Database and parse them as Json files, another script to parse them as structured tables in Hive.

·         Performance optimizations on Spark/Scala.

·          Diagnose and resolve performance issues.

·         Responsible for developing Python wrapper scripts which will extract specific date range using Sqoop by passing custom properties required for the workflow.

•     Developed a unit test script to read a Parquet file for testing Pypark on the cluster.

Environment:  Hadoop, HDFS, Map Reduce, Hive, HBase, Zookeeper, Impala,  Java(jdk1.6), Cloudera, Oracle, SQL Server, UNIX Shell Scripting, Flume, Oozie, Scala, Spark, Sqoop, , PySpark.

Client: TCS, Bangalore, India                                                                                                                      Jan 2014- June 2014

Role: Software Engineer Intern

Responsibilties:

·          Involved in complete software development life cycle management using UML.

·         Coding interfaces for Web Services

·         Application was developed using Spring MVC Web flow modules

·         Implemented spring framework for application transaction management

·         Managed UAT testing and developed test strategies, test plans, reviewed QA test plans for appropriate test coverage.

·         Performed functional, integration, system and validation testing.

Education:

Masters in Computer Science

University of Central Missouri

Bachelors in Computer Science

Jawaharlal Nehru Technological University

Certifications:

AWS certified solution Architect.

Microsoft Certified Database Fundamentals.

Technical Skills:

Big Data Technologies: HDFS, MapReduce, YARN, Hive, Pig, Sqoop, Impala, Oozie, Flume, Zookeeper, Kafka, HBASE                             

Spark Components:  Spark, Spark SQL, Spark Streaming, Pyspark.

AWS Cloud Services: S3, EBS, EC2, VPC, Redshift, EMR      

Programming Languages:  Java, Python, Scala.

Databases: Oracle, MySQL, SQL Server.

Scripting and Query Languages:  Unix Shell scripting, SQL and PL/SQL.

Web Technologies:  JSP, Servlets, JavaBeans, JDBC, XML, CSS, HTML, JavaScript, AJAX.

Operating Systems: Windows, UNIX, Linux distributions, Mac OS.

Other Tools: Maven, Eclipse, Tableau, GitHub, Jenkins.