Candidates About

Anirudh Jaiswal

Senior Hadoop Developer

 

Professional Summary:

·         8+ years of overall experience in building and developing Hadoop Map Reduce solutions.

·         Strong experience with Big Data and Hadoop technologies with excellent knowledge of Hadoop ecosystem: Hive, Spark, Sqoop, Impala, Pig, HBase, Kafka, Flume, Storm, Zookeeper, Oozie.

·         Experience with various tools and frameworks that enable capabilities within the data ecosystem (Hadoop, Kafka, NIFI, Hive, YARN, HBase, NoSQL).

·         Experienced in transferring data from different data sources into HDFS systems using Kafka producers, consumers and Kafka brokers.

·         Expert in creating and designing data ingest pipelines using technologies such as Apache Storm-Kafka.

·         Good Knowledge in creating processing data pipelines using Kafka and Spark Streaming.

·         Experience in Apache Flume for efficiently collecting, aggregating, and moving large amounts of log data.

·         Experience in using job monitoring and scheduling tools like Zookeeper, OOZIE and NiFi.

·         Experience in Importing and Exporting the data using Sqoop from HDFS to Relational Database systems/mainframe and vice-versa.

·         Experience with Apache Spark ecosystem using Spark-SQL, Data Frames, RDD's and knowledge on Spark MLib.

·         Worked with Spark engine to process large scale data and experience to create Spark RDD.

·         Knowledge on developing Spark Streaming jobs by using RDDs and leverage Spark-Shell.

·         Expertise in Talend Big data tool with involved in architectural designing and development of ingestion and extraction job in Big Data and Spark Streaming.

·         Having experience on RDD architecture and implementing Spark operations on RDD and also optimizing transformations and actions in Spark.

·         Hands on Apache Spark jobs using Scala in test environment for faster data processing and used Spark SQL for querying.

·         Knowledge of Spark code and SparkSQL for testing and processing of data using Scala.

·         Knowledge on cloud services Amazon web services(AWS).

·         Good in analyzing data using HiveQL, PigLatin and custom MapReduce program in Java.

·         Expertise in MapReduce programs in HIVE and PIG to validate and cleanse the data in HDFS, obtained from heterogeneous data sources, to make it suitable for analysis.

·         Good in Hive and Impala queries to load and processing data in Hadoop File system (HFS).

·         Good understanding of NoSQL Data bases and hands on work experience in writing applications on NoSQL databases like Cassandra and MongoDB.

·         Hands on Datalake cluster with Hortonworks Ambari on AWS using EC2 and S3.

·         Strong knowledge in MongoDB concepts includes CRUD operations and aggregation framework and in document Schema design.

·         Experience in maintenance/bug-fixing of web based applications in various platforms.

·         Experience in managing life cycle of MongoDB including automation, monitoring and tuning.

·         Implemented Proofs of Concept on Hadoop Stack and different big data analytic tools, migration from different databases (i.e Teradata, Oracle,MYSQL ) to Hadoop.

·         Experience in storing, processing unstructured data using NoSQL databases like HBase.

·         Good in developing web-services using REST, HBase Native API Client to query data from HBase.

·         Experience in working with MapReduce programs using Hadoop for working on Big Data.

·         Knowledge of ETL methods for data extraction, transformation and loading in corporate-wide ETL Solutions and Data Warehouse tools for reporting and data analysis.

·         Experienced in job workflow scheduling, monitoring tools like Oozie and Zookeeper.

·         Experience with Oozie Workflow Engine in running workflow jobs with actions that run Java MapReduce and Pig jobs.

·         Experience in managing and reviewing Hadoop Log files generated through YARN.

·         Experience in using Apache Solr for search applications.

·         Experienced in Java, Spring Boot, Apache Tomcat, Maven, Gradle, Hibernate and open source frameworks/ software's.

·         Preparation of Dashboards using Tableau.

·         Experience with all stages of the SDLC and Agile Development model right from the requirement gathering to Deployment and production support.

·         Proficient on Test driven development (TDD), Agile to produce high quality deliverables.

·         Hands on Agile (Scrum), Waterfall model along with automation and enterprise tools like Jenkins, Chef, JIRA, Confluence to develop projects and version control, Git.

Technical Skills:

Big Data	Hadoop HDFS, MapReduce, HIVE, PIG, HBase, Zookeeper, Sqoop, Oozie, Cassandra, Spark, Scala, Storm, Flume, Kafka and Avro.
Web Technologies	Core Java, J2EE, Servlets, JSP, JDBC, JBOSS, JSF, XML, AJAX, SOAP, WSDL.
Methodologies	Agile, SDLC, Waterfall model, UML, Design Patterns, Scrum.
Frameworks	ASP.NET, Java EE, MVC, Struts 2/1, Hibernate 3, Spring 3/2.5/2.
Programming Languages	Core Java, C, C++, SQL, Python, Scala, XML, Unix Shell scripting, HTML, CSS, JavaScript.
Data Bases	Oracle 11g/10g, IBM DB2, MongoDB, Microsoft SQL Server, MySQL, MS-Access.
Web Services	Apache Tomcat, JSON-RPC, Web Logic, Web Sphere.
NOSQL	Cassandra, MongoDB, HBase.
Monitoring, Reporting tools	Ganglia, Nagios, Custom Shell Scripts.

Professional Experience:

Unified Trust, Lexington, KY.                                                                          Jan’17 – Current

Hadoop Developer

Description: At Unified Trust we focus on our clients' long-term retirement, wealth management, and financial planning goals. Our financial experts create innovative solutions to meet those goals.

Roles and Responsibilities:

·         Responsible for design and development of analytic models, applications and supporting tools, which enable Developers to create algorithms/models in a big data ecosystem.

·         Ensures appropriate data testing is completed and meets test plan requirements.

·         Engage with business stakeholders to design and own end-to-end solutions to empower data driven decision making.

·         Involved in developing code to write canonical model JSON records from numerous input sources to Kafka Queues.

·         Developed UDF's using Scala Scripts, which used in Data frames/SQL and RDD in Spark for Data Aggregation, queries and writing data back into RDBMS through Sqoop.

·         Experienced on Scala programming as a part of Spark batches and Streaming Data pipelines development.

·         Loaded the data into Spark RDD and do in memory data Computation to generate the Output response.

·         Involved in converting Hive/SQL queries into Spark transformations using Spark RDD’s, and Scala.

·         Implementated spark using scala and sparkSQL, Kafka SQL for faster testing and processing.

·         Exposed to Spark Structured Streaming and worked as a part of Agile spike to check the functionality that suits the business requirement.

·         Improving performance and optimization of the existing algorithms in Hadoop using Spark Context, Spark-SQL, Data Frame and Pair RDD's.

·         Hands on experience in Apache Spark creating RDD's and Data Frames applying operations like Transformation and Actions and converting RDD's to Data Frames.

·         Used Spark API over Hadoop YARN to perform analytics on data in Hive and Spark SQL.

·         Implemented ELK (Elastic Search, Log stash, Kibana) stack to collect and analyze the logs produced by the spark cluster.

·         Collected the logs data from web servers and integrated in to HDFS using Flume.

·         Involved in setting QA environment by implementing Pig and Sqoop Scripts.

·         Developed Pig Latin Scripts to do operations of sorting, joining and filtering enterprise data.

·         Experience in writing storm topology to accept events from Kafka producer and emit to Cassandra.

·         Developed Oozie workflow for scheduling ETL process and Hive Scripts.

·         Hands on experience in installing, configuring Cloudera.

·         Integrated Oozie with the rest of the Hadoop stack supporting several types of Hadoop jobs out of the box (like MapReduce, Pig, Hive, Sqoop) as well as system specific jobs.

·         Involved in writing queries in SparkSQL using Scala.

·         Used Spark API over Hortonworks Hadoop YARN to perform analytics on data in Hive.

·         Have good experience with NOSQL database like Cassandra.

·         Analyze Cassandra database and compare it with other open-source NoSQL databases to find which one of them better suites the current requirement.

·         Develop, Maintain and support Continuous Integration(CI) framework based on Jenkins.

·         Designed Implemented the prototype for the functional comparison using Cassandra vs Solr, and then knowledge sharing with other team members on Cassandra.

·         Configured and used FTP by the Informatica Server to access source files.

·         Implemented on Hadoop stack and different big data analytic tools, migration from different databases SQL Server2008 R2, Oracle, MYSQL to Hadoop.

·         Worked on Hadoop cluster and data querying tools like Hive to store and retrieve data.

·         Performed advanced procedures like text analytics and processing, using the in-memory computing capabilities of Spark using Scala.

·         Development of company´s internal CI system, providing a comprehensive API for CI/CD

·         Loading the sstable files to Scylla using the Scylla sstableloader tool + Data validation.

·         Loading data from Linux file system (LFS) to HDFS and vice-versa.

·         POC for enabling member and suspect search using Solr.

·         Knowledge of ETL methods for data extraction, transformation and loading in corporate-wide ETL Solutions and Data warehouse tools for reporting and data analysis.

·         Developed an autonomous continuous integration system by using Git, Gerrit, Jenkins, MySQL and custom tools developed in Python and Bash.

·         Developed Merge jobs in Python in order to extract and load data into MySQL database and used Test driven approach for developing applications.

·         Designed, developed, implemented, and maintained solutions for using Docker, Jenkins, Git, and Puppet for microservices and continuous deployment.

·         Used Erwin for reverse engineering to connect to existing database and ODS to create graphical representation in the form of Entity Relationships and elicit more information

·         Enhanced microservice performance by using task manages and webpack with serverless framework.

·         Used ERwin for effective model management of sharing, dividing and reusing model information and design for productivity improvement.

·         Developing and deploying distributed computing Big Data applications using Open Source frameworks like Apache Spark, Apex, Flink, Storm, NIFI and Kafka.

·         Using CSVExcelStorage to parse with different delimiters in PIG.

·         Modified reports and Talend ETL jobs based on the feedback from QA testers and Users in development and staging environments.

·         Created and implemented ER models and dimensional models.

·         Provide solutions for data engineers such as NIFI processors and workflows.

·         Created different Groovy Scripts to extract and process the content of the FlowFile in Apache NiFi and generated the Hive Tables automatically.

·         Implemented Test Scripts to support test driven development and integration.

·         Developed multiple MapReduce jobs in Java to clean datasets.

·         Involved in loading data from Linux file systems, servers, java web services using Kafka producers and consumers.

·         Prepared Avro schema files for generating Hive tables.

·         Experienced in working with applications team in installing Hadoop updates, upgrades based on requirement.

Environment: Hortonworks, ETL, YARN, HDFS, PIG, Hive, Kafka, Talend, Cassandra, Eclipse, Java, Sqoop, Nifi, Avro, Spark, Spark API, SparkSQl, Spark Streaming, Scala, Solr, Talend, Linux file system, Linux Shell Scripting, Oozie, Agile.

 

Gogo Inc.,Chiacgo,IL                                                                                            Jan'16 – Dec’16

Big Data Developer

Description: Gogo Inc. is a provider of in-flight broadband Internet service and other connectivity services for commercial and business aircraft. Gogo’s worldwide inflight Wi-Fi services have made internet and video entertainment a regular part of flying.

Roles and Responsibilities:

·         Collected and aggregated large amounts of weblog data from different sources such as webservers, mobile and network devices using Apache Flume and stored the data into HDFS for analysis.

·         Experience in setting up Fan-out workflow in Flume to design V-shaped architecture to take data from many sources and ingest into thesingle sink.

·         Reviewing and managing Hadoop log files from multiple machines using Flume.

·         Real time processing of raw data stored in Kafka and storing processed data in Hadoop using Spark Streaming(DStreams).

·         In pre-processing phase used SparkRDD transformations to remove all the missing data and to create new features.

·         Developed SparkSQL queries for generating statistical summary and filtering operations for specific use cases working with SparkRDD's on distributed cluster running ApacheSpark.

·         Worked with Spark to create structured data from the pool of unstructured data received.

·         Involved in working with Spark on top of YARN for interactive and batch analysis.

·         Importing and exporting data into HDFS and Hive using Sqoop.

·         Installed and configured Hive and also written Hive UDFs and UDAFs.

·         Worked on analyzing Hadoop Cluster and different big data analytic tools including Pig, Hive and MongoDB.

·         Involved in creating Hive tables, loading with data and writing hive queries which will run internally in map reduce way.

·         Worked on Impala for creating views for business use-case requirements on top of Hive tables.

·         Created Hive tables by using Apache NiFi and loaded the data into tables using Hive Query Language.

·         Created AngularJS directives, factories and services for developing single page web applications.

·         Processed the Web server logs by developing Multi-Hop Flume agents by using Avro Sink and loaded into MongoDB for further analysis, Extracted files from MongoDB through Flume and Involved in loading data from UNIX file system to HDFS.

·         Extracted and restructured the data into MongoDB using import and export command line utility tool.

·         Maintained stored definitions, transformation rules and targets definitions using Informatica repository Manager and Infoworks.

·         Used MongoDB as part of POC and migrated few of the stored procedures in SQL to MongoDB.

·         Worked with NOSQL databases (MongoDB) and Hybrid implementations.

·         Monitoring of Document growth and estimating storage size for large MongoDB clusters.

·         Experienced in working with Spark eco system using SparkSQL and Scala queries on different formats loke Text file, CSV file.

·         Expertized in implementing Spark using Scala and SparkSQL for faster testing and processing of data responsible to manage data from different sources.

·         Developed interactive Dashboards using Tableau connecting to Impala.

·         Involved in setting QA environment by implementing Pig and Sqoop scripts.

Environment: Linux, MongoDB, Hadoop, Spark, Sqoop, Nifi, Pig, Impala, Hive, Kafka, Flume, Cloudera, Design Patterns, Apache Tomcat, My SQL Server 2008.

 

Wellcare Health Plans, Tampa, FL.                                                                     Nov’14- Dec’15

Hadoop Developer   

Description: WellCare focuses exclusively on providing government-sponsored managed care services, primarily through Medicaid, Medicare Advantage and Medicare.

Roles and Responsibilities:

·         Experience in creating integration between Hive and HBase for effective usage and performed MR Unit testing for the Map Reduce jobs.

·         Created BI reports(Tableau) and dashboards from HDFS data using Hive.

·         Experience in importing and exporting the data from Relational Database Systems to HDFS by using Sqoop.

·         Developed a common framework to import the data from Teradata to HDFS and to export to Teradata using Sqoop.

·         Imported the log data from different servers into HDFS using Flume and developed MapReduce programs for analyzing the data.

·         Used Flume to handle the real time infoworks log processing for attribution reports.

·         Worked on tuning the performance of Pig queries.

·         Developed Python code to gather the data from HBase and designs the solution to implement using PySpark.

·         Great hands on experience with Pyspark for using Spark libiries by using python scripting for data analysis.

·         Involved in building database Model, APIs and Views utilizing Python, in order to build an iterative web based solution.

·         Performed troubleshooting, fixed and deployed many Python bug fixes for two main applications that were main source of data for customers and internal customer service team.

·         Involved in loading data from UNIX file system to HDFS.

·         Performed operation using Partitioning pattern in MapReduce to move records into different categories.

·         Implemented Fair schedulers on the Job tracker to share the resources of the Cluster for the MapReduce jobs given by the users.

·         Created HBase tables to load large sets of data coming from UNIX and NoSQL.

·         Implemented the Web Service client for the login authentication, credit reports and applicant information using Apache Axis 2 Web Service.

·         Design, develop, test, implement and support of Data Warehousing ETL using Talend and Hadoop Technologies.

·         Built and deployed Java applications into multiple Unix based environments and produced both unit and functional test results along with release notes

Environment: WebSphere 6.1, HTML, XML, ANT 1.6, MapReduce, Pyspark, Sqoop, UNIX, NoSQL, Java, JavaScript, MR Unit, Teradata, Node.js, JUnit 3.8, ETL, Talend, HDFS, Hive, HBase.

 

Grand Bank of Texas,Dallas,TX                                                                          Nov’13- Oct’14

Hadoop/Java Developer

Description: Grand Bank of Texas is a locally-owned and operated independent community bank to serve the financial needs of the community and area.

Roles and Responsibilities:  

·         Experience with the Hadoop ecosystem (MapReduce, Pig, Hive, HBase) and NoSQL.

·         Analyze and determine the relationship of input keys to output keys in terms of both type and number, identify the number, type, and value of emitted keys and values from the Mappers, Reducers and the number and contents of the output files.

·         Developed MapReduce pipeline jobs to process the data and create necessary HFiles and loading the HFiles into HBase for faster access without taking performance hit.

·         Designed & developed UI Screens with Spring (MVC), HTML5, CSS, JavaScript, AngularJS to provide interactive screens to display data.

·         Written multiple MapReduce programs in java for data extraction, transformation and aggregation from multiple file formats including XML, JSON, CSV and other file formats.

·         Analyze to determine the correct InputFormat, OutputFormat on order of MapReduce job requirements.

·         Fetching the records for SOAP and Restful requests from Oracle DB using SOLR search.

·         Built AngularJS modules, controllers, pop up modals, and file uploaders.

·         Involved in Enhancement of existing application utilizing AngularJS, created HTML navigation menu.

·         Created database tables and wrote T-SQL Queries and stored procedures to create complex join tables and to perform CRUD operations.

·         By using AWS, MapReduce job processed the data stored in the AWS.

·         Using Automated Build and continuous integration systems Jenkins and test-driven Unit Testing frame