Candidates About

                                                       

 

Saiguru P.V                                                                                                                             

 

SUMMARY:

 

·         6 years of professional experience in designing, developing, debugging Web-based as well as Enterprise applications using OOA, OOD, OOPS and JAVA/J2EE technologies.

·         Over 4 years of experience in the Hadoop ecosystem components like Hadoop MapReduce, Cloudera, Hortonworks, HBase, Oozie, Hive, Sqoop, Pig, Flume, Yarn, Impala, and Cassandra.

·         Experience in Hadoop architecture and various components such as HDFS, Job Tracker, Task Tracker, Name Node, Data Node and MapReduce programming paradigm.

·         Experience in converting MapReduce applications to Spark.

·         Experience in data processing like collecting, aggregating, moving from various sources using Apache Flume and Kafka.

·         Extensively worked on Spark streaming and Apache Kafka to fetch live stream data.

·         Extensive experience in developing PIG Latin Scripts and using Hive Query Language for data analytics.

·         Good working experience using Sqoop to import data into HDFS from various RDBMS and vice versa.

·         Experience in analyzing data using HIVEQL and Pig Latin and custom MapReduce programs in Java

·         Working knowledge of Software Design Patterns, Big Data Technologies (Hadoop, Hortonworks Sandbox) and Cloud Technologies & design

·         Familiarity with Hadoop architecture and its components like HDFS, MapReduce, Job Tracker, Task Tracker, Name Node and Data Node

·         Knowledge on handling Hive queries using spark SQL that integrate with spark environment implemented in Scala.

·         Experience in setting up Hadoop clusters on cloud platforms like AWS.

·         Knowledge in Hadoop And Sub-Modules; HDFS, MapReduce, Apache Pig, Hive, HBase And Sqoop

·         Experience with NoSQL databases like CouchDB, MongoDB.

·         Good experience in job scheduling and monitoring tools like Oozie and ZooKeeper.

·         Experience in Hadoop administration activities such as installation and configuration of clusters using Apache and Cloudera.

·         Experienced in using Agile software methodology (Scrum).

·         Experience in developing distributed Enterprise and Web applications using UML, Java/J2EE, Web technologies that include EJB, JSP, Servlets, Struts, JMS, JDBC, JPA HTML, XML, XSL, XSLT, Java Script, Spring and Hibernate.

·         Expertise in using J2EE Application Servers like Web Logic 8.1/9.2, IBM Web Sphere 7.x/6.x and Web Servers like Tomcat 5.x/6.x

·         Designed Use Case diagrams, Class diagrams, Activity diagram, Sequence diagrams, Flow Charts, and deployment diagrams using Rational Rose Tool.

·         Proficient in writing and handling SQL Queries, Stored Procedures, and triggers.

·         Knowledge of different operating systems including Linux, Windows and UNIX Shell Script.

 

 

 

 

SKILL SET:

 

Big Data Eco System                       HDFS, MapReduce, Hadoop Streaming, ZooKeeper, kafka, Oozie, Sqoop,Hive, Pig, HBase, Spark, Cloudera CDH 3/4/5, Airflow, Flume.

NoSQL                                                  HBase, Cassandra, MongoDB

Languages                                           Java/ J2EE, SQL, Shell Scripting, Python

Web Technologies                          HTML, JavaScript, CSS, XML, Servlets, SOAP.

Web/ Application Server             Apache Tomcat Server, LDAP, JBOSS, IIS

Operating system                            Windows, Linux and Unix

Frameworks                                       Springs, MVC, Hibernate, Swings

DBMS / RDBMS/ETL                        Oracle 11g/10g/9i, SQL Server 2012/2008, MySQL, DataStage 7.5.2, SSIS

Reporting Tools                                                Tableau Server 10.1,   

Version Control                                SVN, CVS and Rational Clear Case Remote Client

 

 

PROFESSIONAL EXPERIENCE:

 

Client: Huawei, Bellevue, WA                                                                                                  

Role: Sr Hadoop Developer                                                                                                                        Mar 2017 – Present

 

Responsibilities:

·         Installed, Configured and Maintained Hadoop clusters for application development and Hadoop tools like Hive, Pig, Hbase, Zookeeper and Sqoop. 

·         Created complex SQL stored procedures and developed reports using Microsoft SQL Server 2012. 

·          Designed SSIS (ETL) Packages to extract data from various heterogeneous data sources such as Access database, Excel spreadsheet and flat files into SQL Server. 

·          Created the packages in SSIS (ETL) with the help of Control Flow Containers, Tasks and Data Flow Transformations. 

Loading the data from the different Data sources like (Teradata and DB2) into HDFS using sqoop and load into Hive tables, which are partitioned.

·          Worked on creating Hive tables and written Hive queries for data analysis to meet business requirements and experienced in Sqoop to import and export the data from Oracle & MySQL 

·         Imported and Exported the analyzed data to the relational databases using Sqoop for visualization and to generate reports for the BI team.

·         Worked on importing and exporting data into HDFS and Hive using Sqoop.

·         Used Flume to handle streaming data and loaded the data into Hadoop cluster

·         Developed and executed hive queries for de-normalizing the data.

·         Developed the Apache Storm, Kafka, and HDFS integration project to do a real-time data analyses.

·         Responsible for executing hive queries using Hive Command Line, Web GUI HUE and Impala to read, write and query the data into HBase.

·         Moved data from HDFS to Cassandra using MapReduce and Bulk Output Format class

·         Developed bash scripts to bring the T-log files from ftp server and then processing it to load into hive tables.

·         Developed Oozie Workflows for daily incremental loads, which gets data from Teradata and then imported into hive tables.

·         Worked on analyzing data with Hive and Pig.

·         Designed Apache Airflow entity resolution module for data ingestion into Microsoft SQL Server.

·         Developed batch processing pipeline to process data using python and airflow. Scheduled spark jobs using airflow.

·         Created a new airflow DAG to find popular items in redshift and ingest in the main Postgres DB via a web service call.

·         Worked on migrating MapReduce programs into Spark transformations using Spark and Scala.

·         Involved in writing, testing, and running MapReduce pipelines using Apache Crunch.

·         Managed, reviewed Hadoop log file and also worked in analyzing SQL scripts and designed the solution for the process using Spark.

·         Integrated Apache Storm with Kafka to perform web analytics and to perform click stream data from Kafka to HDFS.

·         Integrated Kafka-Spark streaming for high efficiency throughput and reliability.

·         Worked with Kafka for the proof of concept for carrying out log processing on a distributed system.

·         All the bash scripts are scheduled using Resource Manager Scheduler.

·         Developed pig scripts to transform the data into structured format and it are automated through Oozie coordinators.

·         Worked on MongoDB, HBase (NoSQL) databases which differ from classic relational databases.

·         Improving the performance and optimization of existing algorithms in Hadoop using Spark context, Spark-SQL and Spark YARN.

·         Developed MapReduce programs for applying business rules on the data.

·         Implemented the workflows using Apache Oozie framework to automate tasks.

·         Involved in joining and data aggregation using Apache Crunch.

·         Developed Spark Applications by using Scala, Java and Implemented Apache Spark data processing project to handle data from various RDBMS and Streaming sources. 

·         Developed Kafka consumer's API in Scala for consuming data from Kafka topics

·         Written multiple MapReduce programs in Java for data extraction, transformation and aggregation from multiple file formats including XML, JSON, CSV and other compressed file formats

·         Worked with NoSQL databases like Hbase in creating Hbase tables to load large sets of semi structured data coming from various sources.

·         Manage the day-to-day operations of the cluster for backup and support

·         Defined best practices for creating Tableau dashboards by matching requirements to the charts to be chosen, color patterns as per user's needs, standardizing dashboard's size, look and feel etc. 

·          Developed visualizations using sets, Parameters, Calculated Fields, Actions, sorting, Filtering, Parameter driven analysis. 

·         Experience in setting up Hadoop clusters on cloud platforms like AWS.

·         Responsible for building scalable distributed data solutions using Hadoop cluster environment with Hortonworks distribution

 

Environment: Apache Hadoop, HBase, Hive, Pig, Sqoop, ZooKeeper, Hortonworks, NoSQL, HBase, Storm, Microsoft SQL Server 2012,  ETL, YARN, Apache Airflow Dag, MapReduce , Tableau Server 10.1 HDFS, Scala, Impala, Flume, MySQL, JDK1.6, J2ee, JDBC, Servlets, JSP, Struts 2.0, Spring 2.0, Hibernate, Python, WebLogic, SOAP, MongoDB, Spark.

 

Client: Western Digital Corporation, San Jose CA

Role: Sr Hadoop Developer                                                                                                                        Jan 2016 – Feb 2017

 

Responsibilities:

·         Worked with NoSQL databases like Hbase in creating Hbase tables to load large sets of semi structured data coming from various sources. Importing and exporting data into HDFS and Hive using SQOOP.

·         Written Hive jobs to parse the logs and structure them in tabular format to facilitate effective querying on the log data

·         Developed the SQL Server Integration Services (SSIS) packages as well as created stored procedures used in SSIS to load/transform data to the database. 

·         Personalized the dashboards to reflect user-specific alerts and metrics to track Key Performance Indicators (KPI). 

·         Created Stored Procedures to transform the Data and worked extensively in T-SQL for various needs of the transformations while loading the data. 

·         Worked as a developer in creating complex T-SQL, stored procedures, cursors, tables, and views and other SQL joins and statements for applications. 

·         Involved in creating Hive Tables, loading data and writing Hive queries.

·         Utilized Apache Hadoop environment by Cloudera.

·         Created Data model for Hive tables.

·         Coordinated with business customers to gather business requirements. And also interact with other technical peers to derive Technical requirements and delivered the BRD and TDD documents.

·         Worked on analyzing Hadoop cluster and different Big Data analytic tools including Pig, Hive HBase database and SQOOP.

·         Written Spark applications using Scala to interact with the MySQL database using Spark SQL Context and accessed Hive tables using Hive Context. 

·         Designed and Modified Database tables and used HBASE Queries to insert and fetch data from tables.

·         Involved in creating Hive tables, loading with data and writing hive queries that will run internally in MapReduce way. Implemented six nodes CDH4 Hadoop Cluster on CentOS

·         Installed Hadoop, MapReduce, HDFS, and Developed multiple MapReduce jobs in PIG and Hive for data cleaning and pre-processing. Extensively involved in Design phase and delivered Design documents.

·         Involved in Testing and coordination with business in User testing.

·         Responsible to manage data coming from different sources.

·         Involved in creating Hive Tables, loading data and writing Hive queries.

·         Utilized Apache Hadoop environment by Cloudera.

·         Created Data model for Hive tables.

·         Consumed XML messages using Kafka and processed the xml file using Spark Streaming to capture UI updates.

·         Developed real time ingestion of System and free form remarks/messages using Kafka and Spark Streaming to make sure the events are available in customer's activity timeline view in real-time 

·         Populated HDFS and Cassandra with huge amounts of data using Apache Kafka.

·         Uploaded streaming data from Kafka to HDFS, HBase and Hive by integrating with storm.

·         Proficient in designing Row keys and Schema Design for NoSQL Database, Hbase and knowledge of other NOSQL database Cassandra.

·         Used Spark API over Cloudera Hadoop YARN to perform analytics on data in Hive.

·         Responsible for importing log files from various sources into HDFS using Flume.

·         Monitored and managed Hadoop cluster using the Cloudera Manager web- interface.

·         Used Hive to analyze the partitioned and bucketed data and compute various metrics for reporting.

·         Experienced in managing and reviewing the Hadoop log files.

·         Designing ETL Data Pipeline flow to ingest the data from RDBMS source to Hadoop using shell script, SQOOP, package and MYSQL.

·         Worked with NoSQL database HBase in getting real time data analytics using Apache Spark with Scala.

·         Used Impala to read, write and query the Hadoop data in HDFS from HBase or Cassandra and configured Kafka to read and write messages from external programs.

·         Hands on experience in performing real time analytics on big data using HBase and Cassandra in Kubernetes & Hadoop clusters.

·         Used Pig as ETL tool to do Transformations, even joins and some pre-aggregations before storing the data onto HDFS.

·         Continuously monitored and managed the Hadoop Cluster using Cloudera Manager.

·         Experienced in using Zookeeper and OOZIE Operational Services for coordinating the cluster and scheduling workflows.  Involved in Unit testing and delivered Unit test plans and results documents.

·         Developed Hive queries for Analysis across different banners.

·         Developed Spark scripts by using Scala shell commands as per the requirement.

·         Developed Hive UDF’s to bring all the customers email-id into a structured format.

·         Exported data from HDFS environment into RDBMS using Sqoop for report generation and visualization purpose. Worked on Oozie workflow engine for job scheduling.

·         Experience in using Pentaho Data Integration tool for data integration, OLAP analysis and ETL process

·         Responsible to manage data coming from different sources.

 

Environment: CDH4 with Hadoop, HDFS, Pig, Cloudera, Hive, Hbase, kafka, Zookeeper, MapReduce, Java, Sqoop, Oozie, Hortonworks, Microsoft SQL Server 2008 R2, MS Windows Server 2008, Storm, Impala, ETL, CSS, Ambari, NoSQL, AWS (Amazon Web services), Linux, UNIX Shell Scripting and Big Data.

 

Client: State Of Illinois, Springfield, IL

Role: Hadoop Developer                                                                                                                   Mar 2015 – Jan2016

 

Responsibilities:

·         Responsible for building scalable distributed data solutions using Hadoop.

·         Responsible for running Hadoop streaming jobs to process terabytes of xml's data.

·         Load and transform large sets of structured, semi structured and unstructured data using Hadoop/Big Data concepts.

·         Handled importing data from various data sources, performed transformations using Hive, MapReduce, and loaded data into HDFS.

·         Analyzed the existing ETL process and came up with an ETL design document that listed the jobs to load, the logic to load and the frequency of load of all the tables.

·          Analyzed, designed, developed, implemented and maintained Parallel jobs using Enterprise Edition of DataStage. 

·          Developed complex jobs using various stages like Lookup, Join, Merge, Sort, Transformer, Dataset, Row Generator, Column Generator, Sequential File and Aggregator Stages. 

·         Developed multiple POCs using Pyspark and deployed on the YARN cluster, compared the performance of Spark, with Hive and SQL and Involved in End-to-End implementation of ETL logic

·         Extracted the data from Teradata into HDFS using the Sqoop.

·         Exported the patterns analyzed back to Teradata using Sqoop.

·         Uploaded streaming data from Kafka to HDFS, HBase and Hive by integrating with storm.

·         Involved in designing the row key in HBase to store Text and JSON as key values in HBase table and designed row key in such a way to get/scan it in a sorted order.

·         Developed syllabus/Curriculum data pipelines from Syllabus/Curriculum Web Services to HBASE and Hive tables.

·         Integrated Oozie with the rest of the Hadoop stack supporting several types of Hadoop jobs out of the box (such as MapReduce, Pig, Hive, and Sqoop) as well as system specific jobs (such as Java programs and shell scripts).

·         Involved in loading data from UNIX file system to HDFS.

·         Installed Oozie workflow engine to run multiple Hive and Pig jobs which run independently with time and data availability

·         Responsible for Cluster maintenance, adding and removing cluster nodes, Cluster Monitoring and Troubleshooting, Manage and review data backups and log files.

·         Analyzed data using Hadoop components Hive and Pig.

·         Responsible for creating Hive tables, loading data and writing hive queries.

·         Analyzed the web log data using the HiveQL to extract number of unique visitors per day, page views, visit duration, most purchased product on website.            

·         Experienced in developing applications in Hadoop Impala, Hive, Sqoop, Oozie, Java MapReduce, SparkSQL, HDFS, Pig and TEZ.

·         Exported the analyzed

 

Environment: Hadoop Cluster, IBM Info Sphere Data stage, Quality stage & Information Analyzer 8.0.1,HDFS, Hive, Pig, Sqoop, Linux, Hadoop MapReduce, HBase, Shell Scripting. Linux, UNIX Shell Scripting and Big Data.

 

Client: INDMAX, India

Role:  Hadoop Developer                                                                                                                            June 2012 – Oct 2014

 

·         Converting the existing relational database model to Hadoop ecosystem. 

·          Generate datasets and load to HADOOP Ecosystem. 

·          Worked with Linux systems and RDBMS database on a regular basis to ingest data using Sqoop. 

·          Continuous monitoring and managing the Hadoop cluster through Cloudera Manager. 

·          Involved in review of functional and non-functional requirements. 

·          Implemented Frame works using Java and python to automate the ingestion flow. 

·          Responsible to manage data coming from different sources. 

·          Loaded the CDRs from relational DB using Sqoop and other sources to Hadoop cluster by using Flume. 

·          Experience in processing large volume of data and skills in parallel execution of process using Talend functionality. 

·<span style="font-style:normal;font-