Candidates About

Anirudh J

Senior Hadoop Developer

 

Professional Summary:

·         8+ years of overall experience in building and developing Hadoop Map Reduce solutions.

·         Strong experience with Big Data and Hadoop technologies with excellent knowledge of Hadoop ecosystem: Hive, Spark, Sqoop, Impala, Pig, HBase, Kafka, Flume, Storm, Zookeeper, Oozie.

·         Experienced in transferring data from different data sources into HDFS systems using Kafka producers, consumers and Kafka brokers.

·         Expert in creating and designing data ingest pipelines using technologies such as Apache Storm-Kafka.

·         Good Knowledge in creating processing data pipelines using Kafka and Spark Streaming.

·         Experienced in data ingestion using Sqoop, Storm, Kafka and Apache Flume.

·         Experience in Apache Flume for efficiently collecting, aggregating, and moving large amounts of log data.

·         Knowledge in Streaming the Data to HDFS using Flume.

·         Hands on Flume to handle the real time log processing for attribution reports.

·         Experience in Importing and Exporting the data using Sqoop from HDFS to Relational Database systems/mainframe and vice-versa.

·         Experience with Apache Spark ecosystem using Spark-SQL, Data Frames, RDD's and knowledge on Spark MLib.

·         Experience in ingestion, storage, querying, processing and analysis of Big Data with hands on experience in Big Data including Apache Spark, Spark SQL, Spark Streaming.

·         Worked with Spark engine to process large scale data and experience to create Spark RDD.

·         Knowledge on developing Spark Streaming jobs by using RDDs and leverage Spark-Shell.

·         Expertise in Talend Big data tool with involved in architectural designing and development of ingestion and extraction job in Big Data and Spark Streaming.

·         Having experience on RDD architecture and implementing Spark operations on RDD and also optimizing transformations and actions in Spark.

·         Hands on Apache Spark jobs using Scala in test environment for faster data processing and used Spark SQL for querying.

·         Knowledge of Spark code and SparkSQL for testing and processing of data using Scala.

·         Knowledge on cloud services Amazon web services(AWS).

·         Good in analyzing data using HiveQL, PigLatin and custom MapReduce program in Java.

·         Expertise in MapReduce programs in HIVE and PIG to validate and cleanse the data in HDFS, obtained from heterogeneous data sources, to make it suitable for analysis.

·         Good in Hive and Impala queries to load and processing data in Hadoop File system (HFS).

·         Good understanding of NoSQL Data bases and hands on work experience in writing applications on NoSQL databases like Cassandra and MongoDB.

·         Hands on Datalake cluster with Hortonworks Ambari on AWS using EC2 and S3.

·         Strong knowledge in MongoDB concepts includes CRUD operations and aggregation framework and in document Schema design.

·         Experience in maintenance/bug-fixing of web based applications in various platforms.

·         Experience in managing life cycle of MongoDB including sizing, automation, monitoring and tuning.

·         Experience in storing, processing unstructured data using NoSQL databases like HBase.

·         Good in developing web-services using REST, HBase Native API Client to query data from HBase.

·         Experience in working with MapReduce programs using Hadoop for working on Big Data.

·         Knowledge of ETL methods for data extraction, transformation and loading in corporate-wide ETL Solutions and Data Warehouse tools for reporting and data analysis.

·         Experienced in job workflow scheduling, monitoring tools like Oozie and Zookeeper.

·         Experience with Oozie Workflow Engine in running workflow jobs with actions that run Java MapReduce and Pig jobs.

·         Good experience with both Job Tracker (Map reduce 1) and YARN (Map reduce 2).

·         Experience in managing and reviewing Hadoop Log files generated through YARN.

·         Experience in using Apache Solr for search applications.

·         Knowledge of Business Intelligence and Reporting.

·         Experienced in Java, Spring Boot, Apache Tomcat, Maven, Gradle, Hibernate and open source frameworks/ software's.

·         Preparation of Dashboards using Tableau.

·         Experience with all stages of the SDLC and Agile Development model right from the requirement gathering to Deployment and production support.

·         Proficient on Test driven development (TDD), Agile to produce high quality deliverables.

·         Hands on Agile (Scrum), Waterfall model along with automation and enterprise tools like Jenkins, Chef, JIRA, Confluence to develop projects and version control, Git.

Technical Skills:

Big Data	Hadoop HDFS, MapReduce, HIVE, PIG, HBase, Zookeeper, Sqoop, Oozie, Cassandra, Spark, Scala, Storm, Flume, Kafka and Avro.
Web Technologies	Core Java, J2EE, Servlets, JSP, JDBC, JBOSS, JSF, XML, AJAX, SOAP, WSDL.
Methodologies	Agile, SDLC, Waterfall model, UML, Design Patterns, Scrum.
Frameworks	ASP.NET, Java EE, MVC, Struts 2/1, Hibernate 3, Spring 3/2.5/2.
Programming Languages	Core Java, C, C++, SQL, Python, Scala, XML, Unix Shell scripting, HTML, CSS, JavaScript.
Data Bases	Oracle 11g/10g, IBM DB2, MongoDB, Microsoft SQL Server, MySQL, MS-Access.
Web Services	Apache Tomcat, JSON-RPC, Web Logic, Web Sphere.
NOSQL	Cassandra, MongoDB, HBase.
Monitoring, Reporting tools	Ganglia, Nagios, Custom Shell Scripts.

Professional Experience:

Unified Trust, Lexington, KY.                                                                          Jan’17 – Current

Hadoop Developer

Description: At Unified Trust we focus on our clients' long-term retirement, wealth management, and financial planning goals. Our financial experts create innovative solutions to meet those goals.

Roles and Responsibilities:

·         Responsible for design and development of analytic models, applications and supporting tools, which enable Developers to create algorithms/models in a big data ecosystem.

·         Ensures appropriate data testing is completed and meets test plan requirements.

·         Engage with business stakeholders to design and own end-to-end solutions to empower data driven decision making.

·         Involved in developing code to write canonical model JSON records from numerous input sources to Kafka Queues.

·         Developed UDF's using Scala Scripts, which used in Data frames/SQL and RDD in Spark for Data Aggregation, queries and writing data back into RDBMS through Sqoop.

·         Experienced on Scala programming as a part of Spark batches and Streaming Data pipelines development.

·         Loaded the data into Spark RDD and do in memory data Computation to generate the Output response.

·         Involved in converting Hive/SQL queries into Spark transformations using Spark RDD’s, and Scala.

·         Exposed to Spark Structured Streaming and worked as a part of Agile spike to check the functionality that suits the business requirement.

·         Improving performance and optimization of the existing algorithms in Hadoop using Spark Context, Spark-SQL, Data Frame and Pair RDD's.

·         Hands on experience in Apache Spark creating RDD's and Data Frames applying operations like Transformation and Actions and converting RDD's to Data Frames.

·         Used Spark API over Hadoop YARN to perform analytics on data in Hive and Spark SQL.

·         Collected the logs data from web servers and integrated in to HDFS using Flume.

·         Involved in setting QA environment by implementing Pig and Sqoop Scripts.

·         Developed Pig Latin Scripts to do operations of sorting, joining and filtering enterprise data.

·         Experience in writing storm topology to accept events from Kafka producer and emit to Cassandra.

·         Developed Oozie workflow for scheduling ETL process and Hive Scripts.

·         Hands on experience in installing, configuring Cloudera.

·         Involved in writing queries in SparkSQL using Scala.

·         Used Spark API over Hortonworks Hadoop YARN to perform analytics on data in Hive.

·         Have good experience with NOSQL database like Cassandra.

·         Analyze Cassandra database and compare it with other open-source NoSQL databases to find which one of them better suites the current requirement.

·         Integrated Cassandra as a distributed persistent metadata store to provide metadata resolution for network entities on the network.

·         Worked on Hadoop cluster and data querying tools like Hive to store and retrieve data.

·         Performed advanced procedures like text analytics and processing, using the in-memory computing capabilities of Spark using Scala.

·         Loading data from Linux file system (LFS) to HDFS and vice-versa.

·         POC for enabling member and suspect search using Solr.

·         Knowledge of ETL methods for data extraction, transformation and loading in corporate-wide ETL Solutions and Data warehouse tools for reporting and data analysis.

·         Using CSVExcelStorage to parse with different delimiters in PIG.

·         Modified reports and Talend ETL jobs based on the feedback from QA testers and Users in development and staging environments.

·         Worked extensively with Sqoop for importing and exporting the data from HDFS to Relational Database systems/mainframe and vice-versa.

·         Implemented Test Scripts to support test driven development and integration.

·         Developed multiple MapReduce jobs in Java to clean datasets.

·         Involved in loading data from Linux file systems, servers, java web services using Kafka producers and consumers.

·         Prepared Avro schema files for generating Hive tables.

·         Experienced in working with applications team in installing Hadoop updates, upgrades based on requirement.

Environment: Hortonworks, ETL, YARN, HDFS, PIG, Hive, Kafka, Talend, Cassandra, Eclipse, Java, Sqoop, Avro, Spark, Spark API, SparkSQl, Spark Streaming, Scala, Solr, Talend, Linux file system, Linux Shell Scripting, Oozie, Agile.

 

Gogo Inc.,Chiacgo,IL                                                                                            Jan'16 – Dec’16

Big Data Developer

Description: Gogo Inc. is a provider of in-flight broadband Internet service and other connectivity services for commercial and business aircraft. Gogo’s worldwide inflight Wi-Fi services have made internet and video entertainment a regular part of flying.

Roles and Responsibilities:

·         Collected and aggregated large amounts of weblog data from different sources such as webservers, mobile and network devices using Apache Flume and stored the data into HDFS for analysis.

·         Experience in setting up Fan-out workflow in Flume to design V-shaped architecture to take data from many sources and ingest into thesingle sink.

·         Reviewing and managing Hadoop log files from multiple machines using Flume.

·         Real time processing of raw data stored in Kafka and storing processed data in Hadoop using Spark Streaming(DStreams).

·         In pre-processing phase used SparkRDD transformations to remove all the missing data and to create new features.

·         Developed SparkSQL queries for generating statistical summary and filtering operations for specific use cases working with SparkRDD's on distributed cluster running ApacheSpark.

·         Involved in converting SQL queries into Apache Spark transformations using ApacheSparkDataFrames.

·         Worked with Spark to create structured data from the pool of unstructured data received.

·         Involved in working with Spark on top of YARN for interactive and batch analysis.

·         Importing and exporting data into HDFS and Hive using Sqoop.

·         Installed and configured Hive and also written Hive UDFs and UDAFs.

·         Worked on analyzing Hadoop Cluster and different big data analytic tools including Pig, Hive and MongoDB.

·         Involved in creating Hive tables, loading with data and writing hive queries which will run internally in map reduce way.

·         Worked on Impala for creating views for business use-case requirements on top of Hive tables.

·         Experienced in managing and reviewing Hadoop log files.

·         Processed the Web server logs by developing Multi-Hop Flume agents by using Avro Sink and loaded into MongoDB for further analysis, Extracted files from MongoDB through Flume and Involved in loading data from UNIX file system to HDFS.

·         Extracted and restructured the data into MongoDB using import and export command line utility tool.

·         Used MongoDB as part of POC and migrated few of the stored procedures in SQL to MongoDB.

·         Worked with NOSQL databases (MongoDB) and Hybrid implementations.

·         Monitoring of Document growth and estimating storage size for large MongoDB clusters.

·         Experienced in working with Spark eco system using SparkSQL and Scala queries on different formats loke Text file, CSV file.

·         Expertized in implementing Spark using Scala and SparkSQL for faster testing and processing of data responsible to manage data from different sources.

·         Developed interactive Dashboards using Tableau connecting to Impala.

·         Supported Map Reduce Programs those are running on the cluster.

·         Involved in setting QA environment by implementing Pig and Sqoop scripts.

Environment: Linux, MongoDB, Hadoop, Spark HBase, Sqoop, Pig, Impala, Hive, Kafka, Flume, Cloudera, Design Patterns, Apache Tomcat, My SQL Server 2008.

 

Wellcare Health Plans, Tampa, FL.                                                                     Nov’14- Dec’15

Hadoop Developer   

Description: WellCare focuses exclusively on providing government-sponsored managed care services, primarily through Medicaid, Medicare Advantage and Medicare Prescription Drug Plans, to families, children, seniors and individuals with complex medical needs.

Roles and Responsibilities:

·         Experience in creating integration between Hive and HBase for effective usage and performed MR Unit testing for the Map Reduce jobs.

·         Created BI reports(Tableau) and dashboards from HDFS data using Hive.

·         Experience in importing and exporting the data from Relational Database Systems to HDFS by using Sqoop.

·         Developed a common framework to import the data from Teradata to HDFS and to export to Teradata using Sqoop.

·         Imported the log data from different servers into HDFS using Flume and developed MapReduce programs for analyzing the data.

·         Used Flume to handle the real time log processing for attribution reports.

·         Worked on tuning the performance of Pig queries.

·         Involved in loading data from UNIX file system to HDFS.

·         Performed operation using Partitioning pattern in MapReduce to move records into different categories.

·         Implemented Fair schedulers on the Job tracker to share the resources of the Cluster for the MapReduce jobs given by the users.

·         Involved in templates and screens in HTML and JavaScript.

·         Created HBase tables to load large sets of data coming from UNIX and NoSQL.

·         Implemented the Web Service client for the login authentication, credit reports and applicant information using Apache Axis 2 Web Service.

·         Design, develop, test, implement and support of Data Warehousing ETL using Talend and Hadoop Technologies.

·         Built and deployed Java applications into multiple Unix based environments and produced both unit and functional test results along with release notes

Environment: WebSphere 6.1, HTML, XML, ANT 1.6, MapReduce, Sqoop, UNIX, NoSQL, Java, JavaScript, MR Unit, Teradata, Node.js, JUnit 3.8, ETL, Talend, HDFS, Hive, HBase.

 

Grand Bank of Texas,Dallas,TX                                                                          Nov’13- Oct’14

Hadoop/Java Developer

Description: Grand Bank of Texas is a locally-owned and operated independent community bank to serve the financial needs of the community and area.

Roles and Responsibilities:  

·         Experience with the Hadoop ecosystem (MapReduce, Pig, Hive, HBase) and NoSQL.

·         Analyze and determine the relationship of input keys to output keys in terms of both type and number, identify the number, type, and value of emitted keys and values from the Mappers, Reducers and the number and contents of the output files.

·         Developed MapReduce pipeline jobs to process the data and create necessary HFiles and loading the HFiles into HBase for faster access without taking performance hit.

·         Designed & developed UI Screens with Spring (MVC), HTML5, CSS, JavaScript, AngularJS to provide interactive screens to display data.

·         Written multiple MapReduce programs in java for data extraction, transformation and aggregation from multiple file formats including XML, JSON, CSV and other file formats.

·         Analyze to determine the correct InputFormat, OutputFormat on order of MapReduce job requirements.

·         Created database tables and wrote T-SQL Queries and stored procedures to create complex join tables and to perform CRUD operations.

·         By using AWS, MapReduce job processed the data stored in the AWS.

·