Candidates About

NARASIMHA CHEMBROLU

Sr. Hadoop Developer

 

·         8+ Years of experience in Information Technology Industry which includes 4+ years of experience as Hadoop/Spark developer using Big data technologies likeHadoop and Spark Ecosystems.

·         Experience in Hadoop Ecosystem components like MapReduce, Sqoop, Flume, Kafka, Pig, Hive, Spark, Storm, HBase, Oozie and Zookeeper.

·         Experience in Big data Hadoop, Hadoop Ecosystem components like MapReduce, Sqoop, Flume, Kafka, Pig, Hive, Spark, Storm, HBase, Oozie, and Zookeeper.

·         Excellent knowledge on Hadoop ecosystems such as HDFS, Job Tracker, Task Tracker, Name Node, Data Node and Map Reduce programming paradigm.

·         Good Knowledge in writing Spark Applications in Scala and Java.

·         Hands on Experience in designing and developing applications in Spark using Scala to compare the performance of Spark with Hive and SQL/Oracle.

·         Experienced working with Spark Streaming, SparkSQL and Kafka for real-time data processing.

·         Hands on experience with Spark Data frames, Spark-SQL and RDD API of Spark for performing various data transformations and dataset building.

·         Imported the data from different sources like AWS S3, Local file system into Spark RDD.

·         Involved in converting Hive/SQL queries into Spark transformations using Spark Data frames and Scala.

·         Used Spark Data Frames API over Cloudera platform to perform analytics on Hive data.

·         Used Spark Data Frame Operations to perform required Validations in the data.

·         Replaced existing map-reduce jobs and Hive scripts with Spark Data-Frame transformation and actions. Good knowledge on Spark architecture and real-time streaming using Spark.

·         Worked on loading CSV/TXT/AVRO/PARQUET files using Scala/Java language in Spark Framework and process the data by creating Spark Data frame and RDD and save the file in parquet format in HDFS.

·         Experience in importing and exporting data using Sqoop from HDFS/Hive to Relational Database Systems and vice - versa.

·         Uploaded and processed terabytes of data from various structured and unstructured sources into HDFS (AWS cloud) using Sqoop. 

·         Experienced and well versed in writing and using UDFs in both Hive and PIG using Java.

·         Extensive experience in collecting and storing stream data like log data in HDFS using Apache Flume.

·         Hands on experience knowledge in NoSQL databases like HBase, Cassandra, Mongo db.

·         Good knowledge in querying data from Cassandra for searching grouping and sorting.

·         Experience in Data Modeling and working with Cassandra Query Language (CQL).

·         Experience in implementing spark solution to enable real time reports from Cassandra data.

·         Hands on expertise in working and designing of Row keys & Schema Design with NOSQL databases like Mongo DB.

·         Experience in extracting files from MongoDB through Sqoop and placed in HDFS and processed.

·         Experience with NoSQL database by using Indexing, Replication and Sharding in MongoDB. Sorted the data by using indexing.

·         Experienced with performing CRUD operations using HBase Java Client API.

·         Created dataflow between SQL Server and Hadoop clusters using Apache Nifi.

·         Involved in developing Impala scripts to do Adhoc queries.

·         Experienced in running query using Impala and used BI tools to run ad-hoc queries directly on Hadoop.

·         Worked with different file formats like CSV, Text files, Sequence files, XML, JSON, Avro files.

·         Working knowledge of Amazon'sElasticCloudCompute(EC2) infrastructure for computational tasks and Simple Storage Service (S3) as Storage mechanism.

·         Working experience on Hortonworks distribution and Cloudera Hadoop distribution versions CDH4 and CDH5 for executing the respective scripts.

·         Experienced in moving data from different sources using Kafka producers, consumers and preprocess data using Storm topologies.

·         Experienced in writing live Real-time Processing using Spark Streaming with Kafka.

·         Implemented Kafka Custom encoders for custom input format to load data into Kafka Partitions.

·         Experience in developing and scheduling ETL workflows in Hadoop using Oozie with the help of deployment and managing Hadoop cluster using Cloudera and Horton works.

·         Experience in cluster coordination using Zookeeper.

·         Used Oozie and Zookeeper operational services for coordinating cluster and scheduling workflows.

·         Used AWS S3 and Local Hard Disk as underlying File System (HDFS) for Hadoop.

·         Supported various reporting teams and experience with data visualization tool Tableau.

·         Designing & Creating ETL Jobs through Talend to load huge volumes of data into Cassandra, Hadoop Ecosystem and relational databases.

·         In depth understanding/knowledge of Hadoop Architecture and various components such as HDFS, MapReduce Programming Paradigm, High Availability and YARN architecture.

·         Expertise in developing Hive Generic UDF's to implement complex business logic to incorporate into Hive QL.

·         Worked with both MapReduce 1 (Job Tracker) and MapReduce 2 (YARN).

·         Expertise in implementing Ad-hoc queries using Hive QL and good knowledge in creating Hive tables and loading and analyzing data using hive queries.

·         Experience in importing data from a Relational database management system (RDBMS) such as MySQL and Oracle into HDFS, Hive and exported the processed data back into RDBMS using Sqoop.

·         Knowledge on YARN (MapReduce 2.0) architecture and components such as Resource Manager, Node Manager, Container and Application Master and execution of a MapReduce job.

·         Good knowledge in creating PL/SQL Stored Procedures, Packages, Functions, Triggers, Cursors with Oracle (9i, 10g, 11g).

·         Excellent communication, interpersonal, analytical skills, and strong ability to perform as part of team.

·         Experience with Agile Development process tools like Jira.

·         Having strong technical skills in Core Java with working knowledge.

·         Experience in various faces of Software Development including analysis, design, development and deployment of applications using Servlets, JSP, Spring Framework, JDBC.

·         Experience with version control tools like Git and SVN.

 

Technical Summary

Operating System:	Windows, Linux distributions like Ubuntu, CentOS
Hadoop Distribution:	Cloudera (CDH 3, CDH4, CDH5), Horton Works
Languages:	Java,Scala, Python
Data stores:	MySQL, SQL Server
Big data:	MapReduce, HDFS, Flume, Hive, Pig, Oozie, HBase, Sqoop, Spark, NiFi and Kafka
Amazon Stacks:	AWS EMR, S3, EC2, Lambda, Route 53, EBS, CloudFront
RDBMS	Teradata, Oracle 9i,10g,11i, MS SQL Server, MySQL and DB2
ETL:	Talend and Informatica
Web Design Tools	HTML, DHTML, AJAX, JavaScript, jQuery and CSS, AngularJs, and JSON
Development/Build tools	Eclipse, Ant, Maven, IntelliJ, JUNIT and log4J.
No SQL Database	Cassandra, Mongo DB, H Base
Java Technologies	Servlets, JavaBeans, JSP, JDBC, and Spring MVC

 

Work Experience

Client: Rx Savings Solutions, Kansas City, KSApr’17-Present

Role: Sr. Hadoop Developer

Description:Rx Savings Solutions is a web-based healthcare consumerism software providing proven results to save employers, employees, and health plans money on prescription medications.The company offers a unique and personalized experience for each member, eliminating complexity during their decision-making process and providing them with the information they need to save money.

·         Developed Spark Applications by using Scala, Java, Implemented Apache Spark data processing project to handle data from various RDBMS and Streaming sources.

·         Used Spark SQL on data frames to access hive tables into spark for faster processing of data.

·         Implemented Partitioning, Dynamic partitioning and Bucketing in Hive using internal and external table for more efficient data.

·         Working knowledge of Spark RDD, Data Frame API, Data set API, Data Source API, Spark SQL and Spark Streaming.

·         Used Spark Data Frame API to process Structured and SemiStructured files and load them into S3 Bucket.

·         Used Spark Data Frames Operations to perform required Validations in the data and to perform analytics on the Hive data.

·         Used Different Spark Modules like Spark core, Spark SQL, Spark Streaming, Spark Data sets and Data frames.

·         Worked as a support for Spark machine learning (MLlib) team. Used regression algorithms like Random Forest, ANN and SVM.

·         Classify users as buyers/non-buyers via Naïve Bayes classifiers, Decision trees, Random forests, Linear/ Logistic regression and SVMs using R, Mahout and MLLib.

·         Managing multiple AWS instances, assigning the security groups, Elastic Load Balancer and AMIs.

·         Created detailed AWS Security groups which behaved as virtual firewalls that controlled the traffic allowed reaching one or more AWS EC2 instances.

·         Imported data from AWS S3 into Spark RDD, Performed transformations and actions on RDD's.

·         Designed Columnar families in Cassandra and Ingested data from RDBMS, performed data transformations, and then exported the transformed data to Cassandra as per the business requirement.

·         Tested the cluster Performance using Cassandra-stress tool to measure and improve the Read/Writes.

·         Used Impala to read, write and query the Hadoop data in HDFS from Cassandra and configured Kafka to read and write messages from external programs.

·         Experience in creating Impala views on hive tables for fast access to data.

·         Experienced in running query using Impala and used BI tools to run ad-hoc queries directly on Hadoop.

·         Queried and analyzed data from Cassandra for quick searching, sorting and grouping through CQL.

·         Responsible for developing multiple Kafka Producers and Consumers from scratch as per the software requirement specifications.

·         Extract Real time feed using Kafka and Spark Streaming and convert it to RDD and process data in the form of Data Frame and save the data as Parquet format in HDFS.

·         Creating end to end Spark-Solr applications using Scala to perform various data cleansing, Validation, transformation according to the requirement.

·         Worked on apache Solr for indexing and load balanced querying to search for specific data in larger datasets­­­

·         Involved in converting Hive/SQL queries into Spark transformations using Spark RDDs, and Scala.

·         Developed Oozie workflow engine to run multiple Hive, Pig, Sqoop and Spark jobs.

·         Real time streaming the data using Spark with Kafka.

·         Involved in creating data-lake by extracting customer's data from various data sources to HDFS which include data from Excel, databases, and log data from servers.

·         Enabled speedy reviews and first mover advantages by using Oozie to automate data loading into the Hadoop Distributed File System and Pig to pre-process the data

·         Successfully migrated the data from AWS S3 source to the HDFS sink using Kafka

·         Created several jobs in Talend ETL tool to perform transformation on source files.

·         Used Spark transformations for Data Wrangling and ingesting the real-time data of various file formats.

·         Used HUE for running Hive queries. Created partitions according to day using Hive to improve performance. 

·         Exported the analyzed data to the relational databases using Sqoop for visualization and to generate reports for the BI team.

Environment:Hadoop, Map Reduce, HDFS, Hive, Cassandra, Sqoop, Oozie, SQL, Kafka, Spark, Scala, Java, AWS, GitHub, Talend Big Data Integration,Solr,Impala.

 

Client:Deliver Logic,Tampa, FLFeb’16 – Feb’17

Role: Hadoop Developer

Description:Deliver Logic is a national network of restaurant delivery services. We provide software and support. Whether you’re a local independent cafe or national restaurant chain, DeliverLogic helps in boosting profits by offering customers online ordering & delivery.

 

·         Worked with Hortonworks distribution of Hadoop for setting up the cluster and monitored it using Ambari.

·         Created ODBC connection through Sqoop between Hortonworks and SQL Server.

·         Implemented Spark Scripts using Scala, Spark SQL to access hive tables into spark for faster processing of data.

·         Loaded the data into Spark RDD and do in memory data Computation to generate the Output response.

·         Great hands on experience with Pyspark for using Spark libraries by using python scripting for data analysis.

·         Extensively worked with Avro and Parquet files and converted the data from either format Parsed Semi Structured JSON data and converted to Parquet using Data Frames in PySpark.

·         Extensively worked on Hive, Pig, Map Reduce, Sqoop, Oozie in an optimized way of distributed processing.

·         Documented the requirements including the available code which should be implemented using Spark, Hive, HDFS.

·         Worked extensively on Hive to create, alter and drop tables and involved in writing hive queries.

·         Developed Scala scripts, UDFs using both Data frames/SQL/Data sets and RDD in Spark 1.3 for Data Aggregation. 

·         Worked extensively with importing metadata into Hive and migrated existing tables and applications to work on Hive and AWS cloud.

·         Successfully migrated the data from AWS S3 source to the HDFS sink using Flume.

·         Involved in moving log files generated from various sources to HDFS for further processing through Flume.

·         Used Pig as ETL tool to do transformations, event joins, filter both traffic and some pre-aggregations before storing the data onto HDFS.

·         Wrote Hive and Pig scripts for joining the raw data with the lookup data and for some aggregative operations as per the business requirement.

·         Good Knowledge in using NiFi to automate the data movement between different Hadoop systems. Collected data using Spark Streaming from AWS S3 bucket in near-real-time and performs necessary Transformations and Aggregations.

·         Implemented Apache Nifi flow topologies to perform cleansing operations before moving data into HDFS.

·         Reading the log files using Elastic search Logstash and alerting users on the issue and also saving the alert details to MongoDB for analyzations.

·         Worked on extracting files from MongoDB through Sqoop and placed in HDFS and processed.

·         Wrote queries in MongoDB to generate reports to display in the dash board.

·         Worked on MongoDB database concepts such as locking, transactions, indexes, sharding, replication and schema design.

·         Used Sqoop to import data from RDBMS to HDFS cluster using custom scripts.

·         Strong experience in working with ELASTIC MAPREDUCE(EMR) and setting up environments on Amazon AWS EC2 instances.

·         Tested Apache Tez, an extensible framework for building high performance batch and interactive data processing applications, on Pig and Hive jobs

·         Developed Tableau workbooks from multiple data sources using Data Blending.

·         Developed Tableau visualization and dashboards using Tableau Desktop.

Environment: Hadoop, Map Reduce, Yarn, Hive, Pig, Flume, Sqoop, AWS, Tableau, Core Java,Spark, Scala, MongoDB, Horton Works, Elastic Search 5.x, Eclipse.

Client: The Bank of Edwardsville, Edwardsville, ILDec’14-Jan’16

Role: Hadoop Developer

Description:TheBANK of Edwardsville is a leading, locally-owned bank in the St. Louis metro area with20 locations in Illinois and Missouri.This project involved extraction of data from multiple sources and transforms the data as per business logic and is loaded into the data warehouse which is used for reporting.

 

·         Importing and exporting data into HDFS, Pig, Hive and HBase using SQOOP.

·         Responsible to manage data coming from different sources.

·         Successfully migrated Legacy application to Big Data application using Hive/Pig/HBase in Production level

·         Load and transform large sets of structured, semi structured, and unstructured data that includes Avro, sequence files and XMLfiles.

·         Responsible to manage data coming from different data sources.

·         Involved in gathering the requirements, designing, development and testing.

·         Utilized Apache Hadoop environment by Cloudera.

·         Implemented helper classes that access HBase directly from java using Java API to perform CRUD operations.

·         Handled different time series data using HBase to perform store data and perform analytics based on time to improve queries retrieval time.

·         Integrated Map Reduce with HBase to import bulk amount of data into HBase using Map Reduce Programs.

·         Developed simple and complex MapReduce programs in Java for DataAnalysis.

·         Load data from various data sources into HDFS using Flume.

·         Developed the PigUDF'S to pre-process the data for analysis.

·         Worked on Hue interface for querying the data.

·         Created Hive tables to store the processed results in a tabular format.

·         Developed Hive Scripts for implementing dynamic Partitions.

·         Developed Pig scripts for data analysis and extended its functionality by developing custom UDF's.

·         Extensive knowledge on PIG scripts using bags and tuples.

·         Developed workflow in Oozie to automate the tasks of loading the data into HDFS and pre-processing with Pig.

·         Created concurrent access for hive tables with shared/exclusive locks enabled by implementing Zookeeper in cluster

·         Transferred data from various OLTP data sources, such as Oracle, MS Access, MS Excel, Flat files, CSV files into SQL Server.

Environment: Java, Hadoop, HDFS, Hive, HBase, Pig, SQOOP, Oozie, MySQL, MapReduce, Linux, Eclipse, Zookeeper, Cloudera.

<span style="font-size:12pt;