Candidates About

 

Yaswanth

 

·         8+ years of experience in IT experience in software design, development, implementation and support of business applications for Telecom, health and Insurance industries

·         Experience in Big data Hadoop, Hadoop Ecosystem components like MapReduce, Sqoop, Flume, Kafka, Pig, Hive, Spark, Storm, HBase, Oozie, and Zookeeper

·         Worked extensively on installing and configuring Hadoop ecosystem components Hive, SQOOP, PIG, HBase, Zookeeper and Flume

·         Good Knowledge in writing Spark Applictions in Scala Python(Pyspark)

·         Hands of experience on build tools like Maven, Log4j, Junit and Ant

·         Working with the data extraction, transformation and load using Hive, Pig and HBase

·         Hands on Experience in designing and developing applications in Spark using Scala to compare the performance of Spark with Hive and SQL/Oracle.

·         Experience in understanding the security requirements for Hadoop and integrate with Kerberos authentication and authorization infrastructure.

·         Good understanding in Cassandra &MongoDB implementation. 

·         Implemented ETL operations using Big Data platform

·         Hands on experience on Streaming data ingestion and Processing

·         Experienced in designing different time driven and data driven automated workflows using Oozie.

·         Conviva and MLlib from Spark are used for predictive intelligence, customer segmentation and for smooth maintenance in spark streaming.

·         Expertise in writing the Real-time processing application Using spout and bolt in Storm.

·         Experience in configuring various topologies in storm to ingest and process data on the fly from multiple sources and aggregate into central repository Hadoop.

·         Highly Acumen in choosing an efficient ecosystem in Hadoop and providing the best solutions to Big Data problems.

·         Well versed with Design and Architecture principles to implement Big Data Systems.

·         Experience in configuring the Zookeeper to coordinate the servers in clusters and to maintain the data consistency

·         Acumen on Data Migration from Relational Database to Hadoop Platform using SQOOP.

·         Experienced in migrating ETL transformations using Pig Latin Scripts, transformations, join operations.

·         Good understanding of MPP databases such as HP Vertica and Impala.

·         Hands on experience in configuring and working with Flume to load the data from multiple sources directly into HDFS

·          Experience in using design pattern, Java, JSP, Servlets, JavaScript, HTML, JQuery, Angular JS, Mobile JQuery, JBOSS 4.2.3, XML, Web Logic, SQL, PL/SQL, JUnit, and Apache-Tomcat, Linux. 

·         Expertise in relational databases like Oracle, My SQL and SQL Server. 

·         Experienced in implementing projects in Agile and Waterfall methodologies.

·         Well versed with Sprint ceremonies that are practiced in Agile methodology.

·         Strong Experience on Data Warehousing ETL concepts using Informatica Power Center, OLAP, OLTP and AutoSys.

·         Highly involved in all phases of SLDC with Analysis, Design, Development, Integration, Implementation, Debugging, and Testing of Software Applications in a client server environment, Object Oriented Technology and Web based applications.

·         Strong analytical and problem solving skills, highly motivated, good team player with very Good communication & interpersonal skill

 

 

Technical skills:

 

Big Data Ecosystem	Hadoop, MapReduce, Pig, Hive, YARN, Kafka, Flume, Sqoop, Impala, Oozie, Zookeeper, Spark, Ambari, Mahout, MongoDB, Cassandra, Avro, Storm, Parquet and Snappy.
Hadoop Distributions	Cloudera (CDH3, CDH4, and CDH5), Hortonworks, MapR and Apache
Languages	Java, Python, Jruby, SQL, HTML, DHTML, Scala, JavaScript, XML and C/C++
No SQL Databases	Cassandra, MongoDB and HBase
Java Technologies	Servlets, JavaBeans, JSP, JDBC, JNDI, EJB and struts
XML Technologies	XML, XSD, DTD, JAXP (SAX, DOM), JAXB
Methodology	Agile, waterfall
Web Design Tools	HTML, DHTML, AJAX, JavaScript, JQuery and CSS, AngularJs, ExtJS and JSON
Development / Build Tools	Eclipse, Ant, Maven, IntelliJ, JUNIT and log4J.
Frameworks	Struts, spring and Hibernate
App/Web servers	WebSphere, WebLogic, JBoss and Tomcat
DB Languages	MySQL, PL/SQL, PostgreSQL and Oracle
RDBMS	Teradata, Oracle 9i,10g,11i, MS SQL Server, MySQL and DB2
Operating systems	UNIX, LINUX, Mac os and Windows Variants
Data analytical tools	R and MATLAB
ETL Tools	Talend, Informatica, Pentaho

 

 

 

 

Work Experience:

Sr. Hadoop/Spark Developer

Windstream, North Carolina                                                                     February 2016 to Present

Responsibilities:

·         Expertise in designing and deployment of Hadoop cluster and different Big Data analytic tools including Pig, Hive, HBase, Oozie, Sqoop, Flume, Spark, Impala.

·         Ingested the data from Relational Databases to HDFS using SQOOP

·         Designed the Column families in Cassandra.

·         Implemented advanced procedures like text analytics and processing using the in-memory computing capabilities like Apache Spark written in Scala

·         Implemented Spark using Scala and Spark SQL for faster testing and processing of data.

·         Involved in converting Hive/SQL queries into Spark transformations using Spark RDDs, Scala.

·         Worked with Spark to create structured data from the pool of unstructured data received.

·         Implemented intermediate functionalities like events or records count from the flume sinks or

Kafka topics by writing Spark programs in java and Scala.

·         Written Storm topology to accept the events from Kafka producer and emit into Cassandra DB

·         Understanding of Kerberos authentication in Oozie workflow for hive and Cassandra

·         Documented the requirements including the available code which should be implemented using Spark, Hive, HDFS and Elastic Search.

·         Extract Real time feed using Kafka and Spark Streaming and convert it to RDD and process data in the form of Data Frame and save the data as Parquet format in HDFS.

·         Worked towards creating real time data streaming solutions using Apache Spark/Spark Streaming, Kafka.

·         Worked with Kafka, Flume for building robust and fault tolerant data Ingestion pipeline for transporting streaming data into HDFS.

·         Experienced in transferring Streaming data, data from different data sources into HDFS, No SQL databases

·         Worked on Talend ETL tool and used features like context variable and database components like input to oracle, output to oracle, tFile compare, tFile copy, to oracle close ETL components

·         Created ETL Mapping with Talend Integration Suite to pull data from Source, apply transformations, and load data into target database.

·         Prepare a Teradata ETL design with Teradata Utilities and Informatica..

·         Well versed with the Database and Data Warehouse concepts like OLTP, OLAP, Star Schema

·         Developed code in Java which creates mapping in Elastic Search even before data is indexed into

·         Maintained ELK (Elastic Search, Logstash, Kibana) and Wrote Spark scripts using Scala shell

·         AWS provides a secure global infrastructure, plus a range of features that use to secure the data in the cloud

·         Good experience of AWS Elastic Block Storage (EBS), different volume types and use of various types of EBS volumes based on requirement.

·         Worked and learned a great deal from Amazon Web Services (AWS) Cloud services like EC2, S3, EBS, RDS and VPC.

·         Implemented AWS provides a variety of computing and networking services to meet the needs of applications

·         Ingested data from RDBMS and performed data transformations, and then export the transformed data to Cassandra as per the business requirement.

·         Developed multiple Kafka Producers and Consumers from scratch to as per the software

requirement specifications.

·         Performed advanced procedures like text analytics and processing, using the in-memory computing capabilities of Spark using Scala.

·         Worked with Apache Spark which provides fast and general engine for large data processing integrated with functional programming language Scala.

·         Implemented Spark using Scala and utilizing Data frames and Spark SQL API for faster processing of data.

·         Working as Cassandra architect, Expertise in Data stack Enterprise & Apache Cassandra, Cassandra Data Modeling, Cassandra installation & multi data center cluster set up with solr & spark enabled, automation using Ansible.

·         Exposure to Mesos (DC/OS), Docker for Automation

·         Utilized Container technology like Docker along with Mesos and aurora to manage whole cluster of hosts

·         Implemented Docker containers and created clients respective Docker images and leveraged Apache Mesos to manage Cluster hosts for Applications

·         Implemented custom apps in Docker on Mesos, managed by Marathon

·         Experience in using  Solr and Zookeeper technologies.

·         Expert in benchmarking and load testing a Cassandra cluster using a Java-based stress testing utility called Cassandra-stress tool.

·         Real time streaming the data using Spark with Kafka.

·         In detail understanding of Cassandra cluster topology, Virtual and Manual nodes.

·         Cassandra Cluster planning which includes Data sizing estimation, and identify hardware requirements based on the estimated data size and transaction volume

·         Prepare solr queries & cql queries to support Micro Services integration with Cassandra cluster.

·         Good Knowledge in using NiFi to automate the data movement between different Hadoop systems.

·         Designed and implemented custom Nifi processors that reacted, processed for the data pipeline

·         Experience designing and executing time driven and data driven Oozie workflows.

·         Experience processing Avro data files using Avro tools and MapReduce programs.

·         Worked on Cassandra in creating Cassandra tables to load large sets of semi structured data coming from various sources.

·         Worked on different file formats (ORCFILE, TEXTFILE) and different Compression Codecs (GZIP, SNAPPY, LZO).

·         Processed Multiple Data sources input to same Reducer using Generic Writable and MultiInput format.

·          Involved in performance tuning of spark jobs using Cache and using complete advantage of cluster environment.

Environment:  Hadoop, Hive, Flume, Map Reduce, Sqoop, Kafka, Spark, Yarn, Pig, Cassandra,  Oozie, shell Scripting, Scala, Maven, Java, JUnit, agile methodologies, MySQL

 

 

 

Hadoop Developer

Onset Computer Corporation, Texas                                              September 2014 to January 2016

Responsibilities:

·         Worked extensively on Hadoop Components such as HDFS, Job Tracker, Task Tracker, Name Node, Data Node, YARN, Spark and Map Reduce programming

·         Converting the existing relational database model to Hadoop ecosystem. 

·         Worked with Linux systems and RDBMS database on a regular basis in order to ingest data using Sqoop.

·         Developed Schedulers that communicated with the Cloud based services (AWS) to retrieve the data.

·         Strong experience in working with ELASTIC MAPREDUCE and setting up environments on Amazon AWS EC2 instances.

·         Ability to spin up different AWS instances including EC2-classic and EC2-VPC using cloud formation templates.

·         Collected data using Spark Streaming from AWS S3 bucket in near-real-time and performs necessary Transformations and Aggregations to build the data model and persists the data in HDFS

·         Imported the data from different sources like AWS S3, LFS into Spark RDD.

·         Experienced in working with Amazon Web Services (AWS) EC2 and S3 in Spark RDD

·         Managed and reviewed Hadoop and HBase log files.

·         Worked extensively with importing metadata into Hive and migrated existing tables and applications to work on Hive and AWS cloud.

·         Designed and implemented HIVE queries and functions for evaluation, filtering, loading and storing of data.

·         Successfully deployed 1-hadoop cluster and it was integrated with Teradata using Teradata-Hadoop query grid as PoC.

·         Loading data from large data files into Hive tables and retrieving data from both Teradata and Hive, by running queries from Bteq.

·         Analyze table data and implement compression techniques like Teradata Multivalued compression

·         Involved in ETL process from design, development, testing and migration to production environments.

·         Involved in writing the ETL test scripts and guided the testing team in executing the test scripts.

·         Involved in performance tuning of the ETL process by addressing various performance issues at the extraction and transformation stages.

·         Writing Hadoop MapReduce jobs to run on Amazon EMR clusters and creating workflows for running jobs

·         Generating analytics reporting on probe data by writing EMR (elastic map reduce) jobs to run on Amazon VPC cluster and using Amazon data pipelines for automation.

·         Worked with Elastic MapReduce (EMR) on Amazon Web Services (AWS).

·         Have good understanding of Teradata MPP architecture such as Partitioning, Primary Indexes,

·         Good knowledge in Teradata Unity, Teradata Data Mover, OS PDE Kernel internals, Backup and Recovery

·         Worked on huge datasets from Hive/Presto to understand and visualize the data for sales analysis of Top Executives within the Company.

·         Created HBase tables to store variable data formats of data coming from different portfolios.

·         Created Partitions, Buckets based on State to further process using Bucket based Hive joins.

·         Involved in transforming data from Mainframe tables to HDFS, and HBase tables using Sqoop

·         Creating Hive tables and working on them using HiveQL.

·         Creating and truncating HBase tables in hue and taking backup of submitter ID 

·         Developed data pipeline using Kafka to store data into HDFS.

·         Used Spark API over Hadoop YARN as execution engine for data analytics using Hive. 

·         Continuous monitoring and managing the Hadoop cluster through Cloudera Manager.

·         Involved in review of functional and non-functional requirements.

·         Developed ETL Process using HIVE and HBASE.

·         Worked as an ETL Architect/ETL Technical Lead and provided the ETL framework Solution for the Delta process, Hierarchy Build and XML generation.

·         Prepared the Technical Specification document for the ETL job development.

·         Impact analysis to the existing ETL jobs as part of the FDW enhancements.

·         Performance testing for the enhancements and SLA improvement of ETL jobs.

·         Experience in configuring the Storm in loading the data from MYSQL to HBASE using jms

·         Responsible to manage data coming from different sources. 

·         Loaded the CDRs from relational DB using Sqoop and other sources to Hadoop cluster by using Flume. 

·         Experience in processing large volume of data and skills in parallel execution of process using Talend functionality.

·         Written column families in HBase

·         Involved in loading data from UNIX file system and FTP to HDFS. 

·         Developed Hive queries to analyze the output data. 

·         Developed workflow in Oozie to automate the tasks of loading the data into HDFS and pre-processing with Pig. 

·         Designed Cluster co-ordination services through Zookeeper. 

·         Collected the logs data from web servers and integrated in to HDFS using Flume. 

·<span style="font:7pt 'Ti