Candidates About

 

Nikhil V

Sr. Hadoop/Spark Developer

·         An accomplished Hadoop/Spark developer experienced in ingestion, storage, querying, processing and analysis of big data.

·         Extensive experience in developing applications that perform Data Processing tasks using Teradata, Oracle, SQL Server and MySQL database.

·         Hands on expertise in working and designing of Row keys & Schema Design with NOSQL databases like Mongo DB 3.0.1, HBase, Cassandra and DynamoDB (AWS).

·         Extensively worked on Spark using Scala on cluster for computational (analytics), installed it on top of Hadoop performed advanced analytical application by making use of Spark with Hive and SQL/Oracle.

·         Excellent Programming skills at a higher level of abstraction using Scala, Java and Python.

·         Extensive experience in working with various distributions of Hadoop like enterprise versions of Cloudera(CDH4/CDH5), Hortonworks and good knowledge on MAPR distribution, IBM Big Insights and Amazon’s EMR (Elastic MapReduce).

·         Working knowledge of Amazon’s Elastic Cloud Compute(EC2) infrastructure for computational tasks and Simple Storage Service (S3) as Storage mechanism.

·         Experienced in implementing scheduler using Oozie, Airflow, Crontab and Shell scripts.

·         Preparing JIL scripts for scheduling the workflows using Autosys and automated jobs with Oozie. 

·         Good working experience in importing data using Sqoop, SFTP from various sources like RDMS, Teradata, Mainframes, Oracle, Netezza to HDFS and performed transformations on it using Hive, Pig and Spark.

·         Extensive experience in importing and exporting streaming data into HDFS using stream processing platforms like Flume and Kafka messaging system.

·         Strong experience and knowledge of real time data analytics using Spark Streaming, Kafka and Flume.

·         Exposure to Data Lake Implementation using Apache Spark and developed Data pipe lines and applied business logics using Spark.

·         Hands-on-Experience in Analytical tools like SAS, R, RStudio, Python, NumPy, Sci-Kit learn, Spark MLLib, Neo4J and GraphDB 

·         Extensively worked on Spark streaming and Apache Kafka to fetch live stream data.

·         Used Scala and Python to convert Hive/SQL queries into RDD transformations in Apache Spark.

·         Expertise in writing Spark RDD transformations, Actions, Data Frames, Case classes for the required input data and performed the data transformations using Spark-Core.

·         Experience in integrating Hive queries into Spark environment using Spark SQL.

·         Expertise in performing real time analytics on big data using HBase and Cassandra.

·         Experience in developing data pipeline using Pig, Sqoop, and Flume to extract the data from weblogs and store in HDFS. Accomplished developing Pig Latin Scripts and using Hive Query Language for data analytics.

• Developed customized UDFs and UDAFs in java to extend Pig and Hive core functionality.

·         Experience in validating and cleansing the data using Pig Latin operations and UDFs. Hands-on experience in developing Pig MACROS.

·         Great familiarity with creating Hive tables, Hive joins & HQL for querying the databases eventually leading to complex Hive UDFs.

·         Worked on GUI Based Hive Interaction tools like Hue, Karmasphere for querying the data.

·         Proficient in NoSQL databases including HBase, Cassandra, MongoDB and its integration with Hadoop cluster.

·         Working knowledge in installing and maintaining Cassandra by configuring the cassandra.yaml file as per the business requirement and performed reads/writes using Java JDBC connectivity.

·         Experience in writing Complex SQL queries, PL/SQL, Views, Stored procedure, triggers, etc.

·         Involved in maintaining the Big Data servers using Ganglia and Nagios.

·         Written multiple MapReduce Jobs using Java API, Pig and Hive for data extraction, transformation and aggregation from multiple file formats including Parquet, Avro, XML, JSON, CSV, ORCFILE and other compressed file formats Codecs like gZip, Snappy, Lzo.

·         Good experience in optimizing MapReduce algorithms using Mappers, Reducers, combiners and partitioners to deliver the best results for the large datasets.

·         Expert in Coding Teradata SQL, Teradata Stored Procedures, Macros and Triggers. 

·         Installed, Configured and Administered PostgreSQL Databases in the Dev, Staging and Prod environment.

·         Extracted data from various data source including OLE DB, Excel, Flat files and XML.

·         Experienced in using build tools like Ant, SBT, Log4j, Maven to build and deploy applications into the server.

·         Experienced in migrating data from different sources using PUB-SUB model in Redis, and Kafka producers, consumers and preprocess data using Storm topologies.

·         Experienced in writing Ad Hoc queries using Cloudera Impala, also used Impala analytical functions. Good understanding of MPP databases such as HP Vertica.

·         Had competency in using Chef, Puppet and Ansible configuration and automation tools. Configured and administered CI tools like Jenkins, Hudson Bambino for automated builds.

·         Knowledge in installation, configuration, supporting and managing Hadoop Clusters using Apache, Cloudera (CDH3, CDH4) distributions and on Amazon web services (AWS).

·         Built AWS secured solutions by creating VPC with public and private subnets.

·         Proficient in developing, deploying and managing the SOLR from development to production. 

·         Experience in Enterprise search using SOLR to implement full text search with advanced text analysis, faceted search, filtering using advanced features like dismax, extended dismax and grouping. 

·         Worked on data warehousing and ETL tools like Informatica, Talend, and Pentaho.

·         Designed ETL workflows on Tableau, Deployed data from various sources to HDFS.

·         Working experience on Test Data Management tools HP Quality Center, HPALM, Load Runner, QTP and Selenium.

·         Worked on ELK stack like Elastic search, Logstash, Kibana for log management.

·         Experience in managing and reviewing Hadoop log files.

·         Hands-on knowledge in Core Java concepts like Exceptions, Collections, Data-structures, I/O. ­­Multi-threading, Serialization and deserialization of streaming applications.

·         Worked on various programming languages using IDEs like Eclipse, NetBeans, and Intellij.

·         Experience in Software Design, Development and Implementation of Client/Server Web based Applications using JSTL, jQuery, JavaScript, Java Beans, JDBC, Struts, PL/SQL, SQL, HTML, CSS, PHP, XML, AJAX and had a bird’s eye view on React Java Script Library.

·         Used various Project Management services like JIRA for tracking issues, bugs related to code and GitHub for various code reviews and Worked on various version control tools like CVS, GIT, PVCS, SVN.

·         Proficiency with the application servers like WebSphere, WebLogic, JBOSS and Tomcat

·         Experience with best practices of Web services development and Integration (both REST and SOAP). 

·         Experianced working in Test Driven and Behavioural Driven Development.

·         Experience in automated scripts using Unix shell scripting to perform database activities.

·         Experience in complete Software Development Life Cycle (SDLC) in both Waterfall and Agile methodologies.

·         Working experience with Linux lineup like Redhat and CentOS.

·         Good understanding of all aspects of Testing such as Unit, Regression, Agile, White-box, Black-box.

·         Good analytical, communication, problem solving skills and adore learning new technical, functional skills.

 

 

Professional Experience:

Sr. Hadoop/Spark Developer   

 

Responsibilities:

·         Involved in analyzing business requirements and prepared detailed specifications that follow project guidelines required for project development.

·         Provide technical expertise in delivering backend orchestration for executing bank processes.

·         Responsible for building scalable distributed data solutions using Apache Hadoop and Spark. 

·         Used Kafka functionalities like distribution, partition, replicated commit log service for messaging systems by maintaining feeds.

·         Experienced in writing live Real-time Processing and core jobs using Spark Streaming with Kafka as a data pipe-line system.

·         Involved in loading data from rest endpoints to Kafka Producers and transferring the data to Kafka Brokers.

·         Involved in writing customized Java Springboot framework to read and parse incoming messaging from Kafka messaging topics for both batch and realtime data on the System of Records(SOR) cluster.

·         Configuring and developing synchronous and asynchronous Kafka producers, High level and low-level consumer API’s, topics, brokers using java.

·         Developed a NiFi Workflow to pick up the data from mainframes, SFTP servers, IBM CDC replication engine and send that to Kafka broker.

·         Used Apache Spark with ELK cluster for obta­­­ining some specific visualization which require more complex data processing/querying.

·         Developed Spark Applications by using Scala, Java and Implemented Apache Spark data processing project to handle data from various RDBMS and Streaming sources.

·         Worked with the Spark for improving performance and optimization of the existing algorithms in Hadoop using Spark Context, Spark-SQL, Spark MLlib, Data Frame, Pair RDD's, Spark YARN.

·         Implemented Spark SQL for faster processing of data and handle Skew Data for real time analysis in Spark.

·         Migrated Flume with Spark for real time data and Developed the Spark Streaming Application with java to consume the data from Kafka and push them into Hive.

·         Evaluated Hortonworks NiFi (HDF 2.0) and recommended solution to inject data from multiple data sources to HDFS & Hive using NiFi and importing data using Nifi tool from Linux servers.

·         Performed ad-hoc queries on structured data using Hive QL and used Partitioning, bucketing techniques and joins with Hive for faster data access.

·         Used Dbeaver as SQL to see oversee the sample data and oversee the structure of data in the hive data base.

·         Worked with ELK Stack cluster for importing logs into Logstash, sending them to Elasticsearch nodes and creating visualizations in Kibana.

·         Designed Columnar families in Cassandra and Ingested data from RDBMS, performed data transformations, and then exported the transformed data to Cassandra as per the business requirement.

·         Tested the cluster Performance using Cassandra-stress tool to measure and improve the Read/Writes.

·         Diagnosed Cassandra problems by setting Log4J Debug mode for detailed tracing and analyzing Cassandra deferred reads and writes.

·         Configured internode communication between Cassandra nodes and client using SSL encryption.

·         Worked on tuning Bloom filters and configured compaction strategy based on the use case.

·         Closely associated with Cassandra DBA in implementing Cassandra data model in application environment to ensure solution is not affecting existing business as usual

·         Implemented Unit/Load testing using ScalaTest, JUnit/JMeter/Blazemeter tools.

·         Involved in reading uncompressed data formats like Gzip, Avro, Parquet and compressed the same according to the business logic by writing generic code.

·         Co-ordinated with different teams for implementing App logging in Splunk and creating dashboards.

·         App monitoring on Dynatrace and WilyIntroscope for our microservices running on custom cloud (GAIA).

·         Experianced in GAIA custom cloud in JPMorgan environment to create kafka services by parsing them in a json string and developed other modules such as lightswitch and Blue/Green deployment in cloud.

·         Worked in provisioning and managing multi-tenant Hadoop clusters on public cloud environment - Amazon Web Services (AWS)/GAIA and on private cloud infrastructure - Open stack cloud platform.

·         Hands on experience to deploy code in production server by following organization standards (Devops model - Jenkins, Jules, Bitbucket, Changeman tools).

·         Performing code analysis through tool sets such as SONAR, Fortify(SSAP), Blackduck in addition to risks and security initiatives such as Pentests.

·         Involved in designing batches to automate execution in CA Uniter, CTRL-M, Oozie and Autosys.

·         Following complete Agile methodology starting from product backlog, sprint backlog, spring planning, sprint retrospectives, user stories, assigning storypoints etc.

·         Conduct code reviews for team members to ensure proper test coverage and consistent code standards.

Environment: Apache Kafka, Spark, Spark-Streaming, Spark SQL, AWS EMR, CDH, HDFS, Hive, Java (JDK SE 6, 7), Scala, Shell scripting, Maven, IBM Infosphere, Jules/Jenkins, Groovy, Lombok, IntelliJ, Splunk, Dynatrace, Oracle, BitBucket, Jira, Nifi, MySQL, Soap, Cassandra and Agile Methodologies.

 

Sr. Big Data Engineer   

 

Responsibilities:

·         Involved in analyzing business requirements and prepared detailed specifications that follow project guidelines required for project development.

·         Responsible for building scalable distributed data solutions using Apache Hadoop and Spark. 

·         Deployed Scalable Hadoop cluster on AWS using S3 as underlying file system for Hadoop. 

·         Worked in the cluster disaster recovery plan for the Hadoop cluster by implementing the cluster data backup in Amazon S3 buckets. 

·         Developed Spark scripts by using Scala IDE as per the bussiness requirement. 

·         Collected the JSON data from HTTP Source and developed Spark APIs that helps to do inserts and updates in Hive tables. 

·         Developed Spark scripts to import large files from Amazon S3 buckets and Imported the data from different sources like HDFS/HBase into SparkRDD.

·         Used Spark-Streaming APIs to perform necessary transformations and actions on the fly for building the common learner data model which gets the data from Kinesis in near real time and Persists into Cassandra. 

·         Worked with xml's extracting tag information using xpaths and Scala XML libraries from compressed blob datatypes. 

·         Written several Restful API's in scala functional language to implement the functionality defined. 

·         Involved in making changes in spark API when migrating from spark 1.6 to spark 2.2.0.

·         Used Spark API over Cloudera Hadoop YARN to perform analytics on data in Hive. 

·         Worked on the integration of Kafka messaging service for near live stream processing.

·         Partitioning data streams using Kafka, designed and configured Kafka cluster to accommodate heavy throughput of 1 million messages per second. Used Kafka producer API's to produce messages. 

·         Created Hive queries that helped market analysts spot emerging trends by comparing fresh data with EDW reference tables and historical metrics. 

·         Handled importing data from different data sources into HDFS using Sqoop and performing transformations using Hive and then loading data into HDFS. 

·         Built and maintained scalable data pipelines using the Hadoop ecosystem and other open source components like Hive and HBase. 

·         Performed a POC on HBase observer to improve the performance of persisting tables in HBase.

·         Captured data from existing databases that provide SQL interfaces using Sqoop.

·         Implemented Sqooping from Oracle to Hadoop and load back in parquet format.

·         Enabled speedy reviews and first mover advantages by using Oozie to automate data loading into the Hadoop Distributed File System(HDFS).

·         Designed Columnar families in Cassandra and Ingested data from RDBMS, performed data transformations, and then exported the transformed data to Cassandra as per the business requirement.

·         Used DataStax Spark-Cassandra connector to load data into Cassandra and used CQL to analyze data from Cassandra tables for quick searching, sorting and grouping. 

·         <spa