Candidates About

Professional Summary:-

 

·         9years of professional IT experience in analyzing requirements, designing, building, highly distributed mission critical products and Applications.

·          4+ years of Data Analytics experience in Apache Hadoop Cloudera and Hortonworks Distributions

·         Expertise in core Hadoop and Hadoop technology stack which includes HDFS, Map Reduce, Oozie, Hive, Sqoop, Pig, Flume,Teradata,HBase, Spark, Storm, Kafka and Zookeeper.

·         Experience in AWS cloud environment and on s3 storage and ec2 instances and deploying in it.

·         In-depth knowledge of Statistics, Machine Learning, Data mining.

·         Developed schedulers that communicated the the cloud based services (aws) to retrieve the data.

·         Experienced in implementing complex algorithms on semi/unstructured data using Map reduce programs.

·         Expertise knowledge on Microsoft Azure.

·         Experienced in working with structured data using Hive QL, join operations, Hive UDFs, partitions, bucketing and internal/external tables.

·         Experienced in migrating ETL kind of operations using Pig transformations, operations and UDF's.

·         Good knowledge on Python.

·         Spark Streamingcollects this data from Kafka in near-real-time and performs necessary transformations and aggregation on the fly to build the common learner data model and persists the data in NoSQL store (Hbase).

·         Experienced in implementing POC's to migrate iterative map reduce programs into Spark transformations using Scala.

·         Specialization in Data Ingestion, Processing, Development from Various RDBMS data sources into a Hadoop Cluster using Map Reduce/Pig/Hive/Sqoop

·         Experienced in implementing unified data platform to get data from different data sources using Apache Kafka brokers, cluster, Java producers and Consumers.

·         Excellent Working Knowledge in Spark Core, Spark SQL, Spark Streaming.

·         Developed Spark jobs usingscala in test environment for faster data processing and used Spark SQL for querying.

·         Experienced in working with in-memory processing frame work like Spark transformations, SprakSQL and Spark streaming using scala.

·         Excellent understanding and knowledge of NOSQL databases like HBase, Cassandra, MongoDB, Teradata and on Data warehouse.

·         Implemented Frameworks using java and python to automate the ingestion flow.

·         Involved in NoSQL (Datastax Cassandra) database design, integration and implementation and written scripts and invoked them using CQLSH.

·         Involved in data modeling in Cassandra and Involved in implementing sharding and replication strategies in MongoDB.

·         Designed, developed, and monitored Oracle-NoSQL databases, Apache web and cloud server frameworks in LINUX for high performance, VMWare cloud storage for performance-query tuning, ETL processes, large file storage.

·         Experienced in implementing custom interceptors and sterilizers in flume for specific customer requirements.

·         Experienced with batch processing of data sources using Apache Spark, elastic search. 

·         Tool monitored log input from several datacenters, via Spark Stream , was analyzed in Apache Storm and data was parsed and saved into Cassandra.

·         Experience in importing and exporting data using Sqoop from HDFS to Relational Database Systems MYSQL, Oracle, Teradataand vice versa.

·         Excellent understanding / knowledge of Hadoop architecture and various components such as HDFS, Job Tracker, Task Tracker, Name Node, Data Node and MapReduce programming paradigm.

·         Good Exposure on Apache Hadoop Map Reduce programming, PIG Scripting and Distribute Application and HDFS.

·         Expertise knowledge on apache nifi.

·         Experience in managing Hadoop clusters using Cloudera Manager Tool.

·         Very good experience in complete project life cycle (design, development, testing and implementation) of Client Server and Web applications.

·         Worked on Cluster co-ordination services throughZookeeper. 

·         Actively involved in coding using Core Java and collection API's such as Lists, Sets and Maps.

·         Hands on experience in application development using Java, RDBMS, and Linux shell scripting.

·         Experience on different operating systems like UNIX, Linux and Windows.

·         Experience on Java Multi-Threading, Collection, Interfaces, Synchronization, and Exception Handling.

·         Involved in writing PL/SQL stored procedures, triggers and complex queries.

·         Worked in Agile environment with active scrum participation.

Technical Skills & Abilities:-

Hadoop/Big Data	HDFS, Map reduce, HBase, Pig, Hive, Sqoop, MongoDB, Cassandra, Flume, Oozie, Zookeeper, AWS, Spark, Kafka, Teradata, Storm, ETL, Informatica, solr, scala, Jenkins, Apache nifi, presto, Microsoft Azure.
Java & J2EE Technologies	Core Java, Servlets, JSP, JDBC, Java Beans, Maven, Gradle, JUnit, TestNG.
IDE’s	Eclipse, Net beans, Intellij Idea.
Frameworks	MVC, Struts, Hibernate, Spring.
Programming languages	C,C++, Java, Python, Ant scripts, Linux shell scripts
Databases	Oracle 11g/10g/9i, MYSQL, DB2, MS-SQL SERVER, teradata
Web Servers	Web Logic, Web Sphere, Apache Tomcat,
Web Technologies	HTML, XML, JavaScript, AJAX, SOAP, WSDL, JAX-RS, Restful, JAX-WS.
Network Protocols	TCP/IP, UDP, HTTP, DNS, DHCP
Version Controls	CVS, SVN, GIT.

Client: CISCO, SAN JOSE, CA                                                                                          FEB’ 18 to Till Date

Role: BigData/Spark Developer&MapR Platform support.

Project Overview:

·         Worked on analyzing Hadoop cluster and different big data analytical and processing tools including Sqoop, Pig, Hive, Spark, Kafka andPyspark.

·         Worked on MapR platform team for performance tuning of hive and spark jobs of all users.

·         Using Hive TEZ engine to increase the performance of the applications.

·         Working on incidents created by users for platform team on hive and spark issues by monitoring hive and spark logs and fixing it or else by raising MapR cases.

·         Analyzed large amounts of data sets to determine optimal way to aggregate and report on it.

·         Worked on Hadoop Data Lake for ingesting data from different sources such as oracle and Teradata through INFOWORKSingestion tool.

·         Worked on ARCADIA for creating analytical views on top of tables as if the batch is loading also no issue in reporting or table locks as it will point to arcadia view.

·         Scripts to automate permissions and storing the assigned permission in a table and if updated in table also the permission will be assigned to updated group.

·         Worked on Python API for converting assigned group level permissions to table level permission using MapR ace by creating a unique role and assigning through EDNA UI.

·         Migrating various Hive UDF's and queries into Spark SQL for faster requests. 

·         Configured to receive real time data from the ApacheKafka and store the stream data to HDFS using Kafka connect. 

·         Hands on experience in Spark using scala and pythoncreating RDD's, applying operations -Transformation and Actions. 

·         Extensively perform complex data transformations in Spark using Scala language.

·         Involved in converting Hive/SQL queries into Spark transformations using Scala.

·         UsedPyspark and scala languages to process the data.

·         Used Bitbuket and Git repositories.

·         Used text, AVRO, ORC and Parquet file formats for Hive tables.

·         Experienced Scheduling jobs using Crontab.

·         Developed and implemented hive custom UDFs involving date functions.

·         Used sqoop to import data from Oracle, Teradata to Hadoop.

·         Used TESScheduler engine to manage interdependent Hadoop jobs and to automate several types of Hadoop jobs such as Java map-reduce Hive, Pig, Spark, Kafka and Sqoop. 

·         Experienced in creating recursive and replicated joins in hive.

·         Experienced in developing scripts for doing transformations using Scala.

·         Involved in developing Shell scripts to orchestrate execution of all other scripts and move the data files within and outside of HDFS. 

·         Using Kafka on publish-subscribe messaging as a distributed commit log, have experienced in its fast, scalable and durability. 

·         Written Map Reduce code to process and parsing the data from various sources and storing parsed data into HBase and Hive using HBase-Hive Integration.

TIAA , Charlotte, NC                                                                                         May’ 16 to Dec’ 17

Role: Spark Developer

 

TIAA, formerly TIAA—CREF (Teachers Insurance and Annuity Association—College Retirement Equities Fund), is a Fortune 100 financial services organization that is the leading retirement provider of financial services for the academic, research, medical, cultural and governmental industries.

TIAA serves over 5 million active and retired employees participating at more than 15,000 institutions.

Much of TIAA operates on a not-for-profit basis, with surplus returned to participants.

Project Overview

The objective of the Confirms v2 project is to implement a future state confirms architecture for Pension (OMNI) to include simplified selection criteria, streamlined confirm templates, centralized rules application, and Output Management.

 

Responsibilities:

·         Worked on analyzing Hadoop cluster and different big data analytic tools including Pig, Hive,Sqoop,  spark, Scala, Impala, Python, Linux, Shell Scripts,Cloudera, Teradata, java transformation of data into xml.

·         Process involved extracting data through sqoop, Transforming Data using Pig,Hive,Pyspark, Scala and loading data into oracle/Teradata.

·         Involved in creating Hive tables, loading with data and writing hive queries which will run internally in MapReduce way.

·         Used Hive to analyze the partitioned and bucketed data and compute various metrics for reporting.

·         Worked on writing automaticsqoop scripts to import/export the data from external sources such as Oracle and Teradata.

·         Developed Scala scripts using both Data frames/SQL/Data sets and RDD/MapReduce in Spark for Data Aggregation, queries and writing data back into OLTP system through Sqoop.

·         Involved in converting Hive/SQL queries into Spark transformations using SparkRDD'S and Scala.

·         Developed Spark scripts by using Scala shell commands as per the requirement.

·         Exported the analyzed data to the relational databases using Sqoop for visualization and to generate reports for the tableau.

·         Extensively worked with Scala / Spark SQL for data cleansing and generating Data Frames to transform them into row DF’s to populate the aggregate tables in Hive.

·         Adept at developing generic Spark-Scala methods for transformations and designing schema for rows.

·         Adept at writing efficient Spark-Scala code to generate aggregation functions on Data Frames according to business logic.

·         Designed ETL pipelines of loading data from RDBMS(oracle, teradata) into hive datawarehouse.

·         Designed ETL pipelines of loading flatfiles into hive datawarehouse and doing it in automation by generic script.

·         Worked on supporting ETL datastage jobs in production environment.

·         Worked on exporting from Hadoop hive database or hadoop file from hdfs/AWS to external sources (Oracle).

·         Involved in creating Hive tables, loading with data and writing hive queries that will run internally in Map Reduce way.

·         Analyzed large data sets by running Hive queries and Pig scripts.

·         Worked on tuning the performance Pig querieson production jobs.

·         Knowledge on presto and analyzed large data sets by running queries.

·         Created Pig Latin scripts to sort, group, join and filter the enterprise wise data.

·         Worked on getting the data from oracle to HDFS/AWS as file in sqoop and then created a view on top of it for querying the data and then for automating it.

·         Worked on handling special characters in Data using hive and pig.

·         Managed and reviewed Hadoop log files.

·         Worked on automating batch using Jill scripts for autosys.

·         Worked on schedule jobs through autosysand migrating jobs to all Higher environments.

·         Worked on migrating the code to higher environments.

·         Supported ST and prod runs.

·         Implemented partitioning, bucketing and worked on Hive, using file formats and compressions techniques with optimizations.

·         Extensively worked on Text, ORC, Avro and Parquet file formats and compression techniques like Snappy, Gzip and Zlib.

·         Plan, design and launch solution for building Hadoop cluster on cloud by using EMRand EC2 of AWS

·         Strong experience in working with Elastic MapReduce and setting up environments on Amazon AWS EC2 instances.

·         Developed Spark scripts using Python, Spark SQL to access hive tables in spark for faster data processing.

·         Extensively used Spark SQL, Pyspark API's for querying and transformation of data residing in Hive.

·         Large data sets were analyzed using Pig scripts and Hive queries.

·         Worked on custom Pig loaders to work with a variety of data formats such as JSON, CSV etc.

·         Extensively used the Teradata utilities like BTEQ, DDL Commands and DML Commands (SQL).

·         Created a BTEQ script for pre population of the work tables prior to the main load process.

·         Performance Tuning of sources, Targets, mappings and SQL queries in transformations.

·         Worked exclusively with the Teradata SQL Assistant to interface with the Teradata

·         Developed Pig Latin scripts to extract data from the web server output files to load into HDFS.

·         Developed Shell Script to perform Data Profiling on the ingested data with the help of hive.

·         Experience in scripting for automation, and monitoring using Shell scripts.

Client:   Cardinal Health, Dublin, OH                                                                                         Jan’ 16to May’16

Role: Hadoop Developer

Cardinal Health, Inc. is a health care services company based in Dublin. The company specializes in distribution of pharmaceuticals and medical products, serving more than 100,000 locations. The company also manufactures medical and surgical products, including gloves, surgical apparel and fluid management products. In addition, it operates the nation’s largest network of radio pharmacies.  Cardinal Health provides medical products to over 75 percent of hospitals in the United States.

Responsibilities:

·         Worked on analyzing Hadoop cluster and different big data analytic tools including Pig, Hive,Hbase database and Sqoop, Impala, Flume, Cassandra, zookeeper, AWS, Cloudera.

·         Evaluated business requirements and prepared detailed specifications that follow project guidelines required to develop written programs.

·         Responsible for building scalable distributed data solutions using Hadoop.

·         Analyzed large amounts of data sets to determine optimal way to aggregate and report on it using Map Reduce programs.

·         Implemented Map reduce programs to retrieveresults from unstructured data set.

·         Optimized Map Reduce Jobs to use HDFS efficiently by using various compression mechanisms.

·         Handled importing of data from various data sources, performed transformations using Hive, MapReduce, loaded data into HDFS and Extracted the data from HDFStoMYSQL, Oracle, Teradatausing Sqoop.

·         Worked on reading multiple data formats on HDFS using Scala.

·         Involved in converting Hive/SQL queries into Spark transformations using Spark RDDs and Scala.

·         Developed multiple POCs using Scala and deployed on the Yarn cluster, compared the performance of Spark, with Hive and SQL/Teradata.

·         Worked on a POC to compare processing time of Impala with SparkSQL for efficiency batch processing. 

·         Developed Spark Applications for various business logics using Scala, Python.

·         Involved in moving the data between HDFS and AWS S3 by using apache distCp.

·         Involved in pulling the data from Amazon S3 data lake and built Hive tables using Hive Context in Spark

·         Involved in running hive queries and spark jobs on data stored in S3.

·         Run short term ad-hoc queries, jobs on the data stored on S3 using AWS EMR. hive

·         Analyzed the SQL scripts and designed the solution to implement using Scala.

·         Built data platforms, pipelines, storage systems using the Apache Kafka, Apache Storm and search technologies such as elastic search.

·         Experienced in implementing POC's to migrate iterative map reduce programs into Spark transformations using Scala.

·         Experienced with batch processing of data sources using Apache Spark, elastic search. 

·         Experience in AWS cloud environment and on s3 storage and ec2 instances

·         Good knowledge about Cassandra architecture, read, write paths and query.

·         Developed Spark jobs usingscala in test environment for faster data processing and used Spark SQL for querying.

·         ConfiguredSparkstreaming to receive real time data from theKafkaand store the stream data to HDFS.

·         Designed and implemented SOLR indexes for the metadata that enabled internal applications to reference Scopus content. 

·         Used Spark for Parallel data processing and better performances using Scala.

·         Extensively used Pig for data cleansing and extract the data from the web server output files to load into HDFS.

·         Developed a data pipeline usingKafka and Storm to store data into HDFS.

·         Implemented Kafka Java producers, create custom partitions, configured brokers and implemented High level consumers to implement data platform.

·         Managed and reviewed Hadoop log files.

·         Involved in creating Hive tables, loading with data and writing hive queries which will run internally in MapReduce way.

·         Used Hive to analyze the partitioned and bucketed data and compute various metrics for reporting.

·         Installed and configured Pig and also written Pig Latin scripts.

·         Used Maven as the build tool and is scheduled/triggered by Jenkins (build tool).

Environment: Hadoop, MapReduce, HDFS, Hive, Pig, Java, SQLSERVER, Sqoop, Java (jdk 1.6), Spark,kafka, AWS, MongoDB, Storm, Cassandra, ETL, Python, REST API, XML, JSON,  solr, cloudera, Oracle, Teradata, scala, GIT, Agile, Jenkins, Elastic Search.

Role: Hadoop Developer

Responsibilities:

·         Worked on writing Map Reduce jobs to discover trends in data usage by customers.

·         Worked on and designed Big Data analytics platform for processing customer interface preferences and comments using Java, Hadoop, Hive, Impala and Pig, Cloudera.

·         Importing and exporting data into HDFS and Hive using Sqoop from Oracle, Teradata and vice versa.

·         Exported the analyzed data to the relational databases using Sqoop for visualization and to generate reports for the BI team.

·         Migrating various hive UDF’s and queries into Spark SQL for faster requests as part of POC implementation.

·         <span style="font-size:9.5pt;font-family:Cambria, s