Candidates About

 

Kanhu Badtia

Lead Big Data Architect

Professional Summary

¨  Hadoop/Big-Data Architect for HA Clusters of thousands of nodes and hundreds of petabytes scale.

¨  Architect & design applications to claim optimum performance in over-subscribed multitenant environment.

¨  Expertise in Cloudera(CDH) & Hortonworks(HDP) platforms of very large-scale implementations.

¨  Expertise in designing and performance tuning of Big-Data applications such as Spark/Map-Reduce/Hive for extracting best performance especially during shuffle & sort phases, eliminate GC pauses, on-heap/off-heap memory management, avoiding data skew, maximize parallelism reducing I/O time, better utilization of vCores & other specialized techniques.

¨  Experience in planning and establishing Hadoop technology standards & usage frameworks (e.g. Name & Space Quota, ACL management, Yarn fair/capacity scheduler management thru efficient queueing techniques, Hadoop security & monitoring)

¨  Developed custom tools to determine system faults in advance and apply preventive measures. dynamic resource management & cluster load management.

 

¨  Experience in AWS & GCP cloud platforms.

¨  Expertise in large-scale NO-SQL implementation of Cassandra, HBase & MongoDB.

¨  Expertise in Migration and/or integration of existing Data Warehouse with Hadoop.

¨  Brings ongoing innovation, initiative, resourcefulness, and above all – a proven record of delivering unmatched results.

Key Skills

 

Big Data / Hadoop eco system	Hadoop, Spark, Zeppelin, Hive, HBase, Sqoop, Flume, Atlas, Solr, Pig, Falcon, Oozie, Hue, Tez, Mapreduce, Apache Nifi, Kafka. Ignite.
Cloud	AWS, GCP
No-SQL	Cassandra, HBase, Mongo DB
Monitoring & Orchestration	Puppet, Zabbix, Cloudera Manager, Ambari, Splunk, Dr. Elephant & self-developed custom tools­
Languages	Python, Scala, Pig, Bash, R, Java
DBMS	Teradata, SQL Server, Oracle, DB2
BI tools	Microstrategy, Tableau

 

 

 

 

Experience

Description: Esteemed Customer, Apple Inc. required to consolidate all Hadoop infrastructure which were earlier managed separately by individual teams and support day-to-day operations on the clusters. The team is responsible to ensure stable environment, manage configurations, propose and create new sophisticated tools and implement any performance tuning opportunities

Contribution:

·         Architect for Apple's massive Hadoop infrastructure and operations

o    Plan and establish Hadoop technology standards and usage frameworks (e.g. HDFS Quota, Yarn resource allocation, Hadoop Security and access control)

o    Define Cluster configurations & thresholds for admin team. 

o    Work in concert with team of pipeline (ETL) developers to ensure efficient and accurate data transfer using Cloudera CDH Big Data Platform. 

o    Troubleshoot Hadoop infrastructure problems and maintain operational stability. 

o    Design & integrate various batch & streaming applications involving MAPREDUCE, SPARK, HIVE, SOLR, CASSANDRA, KAFKA etc.

o    Advise engineering teams during design phase on maximizing performance.

o    Lead for maintaining SPLUNK monitoring across all clusters including engineering collector scripts, creating alerts & dashboards.

·         Develop next generation Big Data Technology products to address the ever-growing data processing requirements.

o   Develop and maintain custom tools for operational effectiveness.

o    Develop proofs-of-concept and prototypes to help illustrate approaches to technology and business problems. 

o    Test and implement performance improvement techniques.

o    Build and optimize information models, physical data layouts, configuration, optimization and monitoring of Hadoop environments and improve overall processing efficiencies to support the needs of the business.

·         Develop highly scalable and fault-tolerant distributed systems using techniques such as garbage collection internals, Concurrency models, Native and Async IO, Off-heap memory management, etc. 

o    Analyze and performance-tune map-reduce/Spark/Cassandra/Hive application code to extract best performance especially during shuffle and sort phases, eliminate GC pauses, ensure better on-heap/off-heap memory management, reduce executor time spent in I/O, maximize parallelism and better utilization CPU VCores etc.

o    Use various RM, NN and WEBHDFS REST APIs to develop monitoring and administration scripts.

o    Work with business teams and technical analysts to understand business requirements. Determine how to leverage big-data technology to create solutions that satisfy the business requirements. 

 

 

 

 

 

Description: Lowe’s stood up Hadoop clusters in May 2015 & our team operated out of Discovery cluster primarily used for Enterprise Analytics. As a team lead & Project manager, was responsible for execution of projects covering diverse business areas such as USHI, Allied Businesses, Finance, Forecasting & so on.

 

Contribution:

·         Data development & Analytics Projects. Major projects are outlined below.

o   Consolidation of Allied BU Sales, Inventory, customer, GL & other data – Developed data Ingestion pipeline using SQOOP & Nifi. Developed scripts using Bash, Spark, Hive, Pig. Data Visualization using MSTR VI.

o   Single View Of Product – Developed scripts using SQOOP, SCP & Hive to consolidate PCM & PSSA attributes of all product sold at Lowe’s. Oozie coordinator is used for scheduling.

o   IRIS – Implemented Alteryx workflow to load IRIS data from Cassandra DB in MS Azure to our analytics cluster & used SPARK scripts for data clean up & formatting. Built Hive scripts to prepare data for analytics.

o   Sales Forecast & USHI metrics – Implemented Hive scripts for implementing sales metrics for reporting & forecasting. Oozie is used for scheduling. 

o   Brand View – Developed Shell & Hive, Spark-Scala scripts for consolidating BrandView pricing data. Oozie is used for scheduling.

o   Zillow – As part of this innovation project, used Jupyter notebook for identifying correlation between Lowe’s sales & US home price trend.

o   Adobe Sitecat: Implemented Spark Streaming for data landing in our analytics cluster. PySpark & Spark SQL is used for analytical transformations.     

·         Working with BU partners to understand requirements & translating to technical design.

·         Responsible for coordinating with Data Governance team to capture & publish Metadata & documentation – There’s a strong need to capture data lineage & governance around all the data that is stored in Hadoop. As a lead, was responsible for building Analytical Metadata Platform in SQL server based Maestro & partnered with security & governance team to maintain the system. 

·         Validation framework: Implemented automated Hive scripts for ongoing validation to ensure correctness of the data so that Analysts can have complete confidence on the data.

·         Architect & SME for Allied business applications – Architected Orchard, Canada, Mexico & Alacrity metrics & GL consolidation applications.

·         Pivotal role of developing partnership with Lowes.com, Web Analytics & Hadoop admin team.

·         Champion for Innovation & Big data adoption.

·         Attending conferences to learn & adopt new technologies & tools – Attended Hadoop Summit in 2015 & 2016 to keep updated with industry trends & implement innovative tools & techniques. 

·         Participate in leadership meetings to provide project updates.

 

 

Project Lead

Project Description: Syntel’s strategic offering group is responsible for promoting big data projects & implemented various Big data solutions for several of its customers. For Belk & other retail, logistics & telecom (RLT) vertical clients, trained experts like me are responsible for business development.   

Contribution:

•       Planning, Architecture and implementation of scalable big data clusters – Assisted multiple clients in setting up their POC clusters using HDP & CDH platforms.

•       Design & application development using Pig, Hive, Sqoop, Flume. 

•       Hadoop Cluster monitoring, configuration & administration.

•       Implement monitoring, alerts, and disaster recovery.

•       Champion best practices for big data solutions and environments.

•       Market basket analysis using associative algorithms in R. 

•       Implementation experience of HBase & Cassandra Clusters.

•       ETL process formation for Hadoop data ingestion using Sqoop & Flume.

Environment: 

Cloudera Manager, CDH3/4, HDP 2.1/2.3, HBase 0.98, R 3.0.10,  Cassandra

 

Project Lead

Project Description: Esteemed Customer: Belk needed a dedicated database expert to manage & administer its mainframe environments that is shared by as much as 30 different applications. Mainframe environment at Belk consists of 2 Dev regions, 2 heavily shared QA regions & one production region. Team is responsible for managing & administering QA environments & coordinate with IBM for production environment related efforts.

CPU optimization is one key objective & team is expert in tuning application & SQLs. In 2012, we were able to save a million dollar for Belk.

Contribution:

•       MF IT Governance, configuration management & Change management.

•       MF CPU optimization. Application/Database tuning.

•       Mainframe product upgrades & maintenance.

•       IT compliance & audit reporting.

•       Database modeling as per application requirements.

•       Conflict management for data sharing.

•       Administration of QA database.

•       Monitoring database performance & index maintenance.

•       Quality assurance.

•       Mass-refreshes of QA databases.

•       Organization wide communication of environmental proceedings. Reporting to senior management including IT directors.

Environment: 

DB2 9, 10.1, Data studio, Omegamon XE, CA Platinum R 16

 

Technical Lead

Project Description: Esteemed Customer: Franklin Templeton Investments is a recognized investment management Organization headquartered in San Mateo, California.  On October 3, 2008, the Emergency Economic Stabilization Act of 2008 was signed into law. Under the new law, mutual fund companies such as Franklin Templeton will become subject to new reporting requirements to both shareholders and the IRS.

The project aims at preparing new Cost-basis engine to comply with the reporting requirements due to change in regulation.     

Contribution:

·         Coordination and interacting with teams from multiple geographies.  

·         Analysis & translation of business requirements.

·         Development of brand new IMSDB and DB2 programs & JCLs.

·         Preparing test cases, Code review & release.

·         Project estimates & release management.

·         Management reporting.

Environment: 

Cobol, Db2, IMS DB/DC 

 

Technical Lead

Project Description: Esteemed Customer: Farmers Insurance Group Inc. is the third largest insurance carrier in the United States and is a wholly owned subsidiary of ZFS, or Zurich Financial Services, based out of Zurich, Switzerland.  It is a personal lines property and casualty insurance group providing Homeowners Insurance, Auto Insurance, Life Insurance and Financial Services.

MIS system is a downstream reporting application of claims/ policy processing data which receives premium data from respective Policy processing systems (APPS, FPPS, TIE, MCNA & WCS) & losses data from FACTS & NON-FACTS & formats the data to prepare must use files that are used for Internal/External reporting to various regulatory establishments.  

Contribution:

·         Release coordination & Incident management.

·         Business requirement analysis, HLD/LLD preparation.

·         Development of major/minor enhancements.

·         SME for MIS Internal projects & production support. 

·         Peer review & SPOC for performance improvement.

·         Quality assurance of project deliverables.

Team management activities.

Environment: 

Cobol, JCL, DB2, EZT, SYNCSORT 

 

Software engineer

Project Description: Esteemed Customer: GEFA-PMG markets and provides service for consumer clubs and insurance products to customers.  GEFA-PMG products include auto clubs, legal, dental, accident, and life and health insurance plans. The Personalized Automated Life/Liability Management (PALLM) system is a comprehensive software package designed to meet the needs of life and health insurance companies. It consists of a number of programs with the facility to store, manipulate, retrieve and assimilate information, which a life and Health Insurance Company needs to process its book of business.

Contribution:

·         Analyzing the specifications provided for the work request.

·         Preparing the understanding document for the specifications.

·         Especially accountable for premium calculation process and claim administration/ interest calculation processes.

·         Acting as primary / secondary support for the production batch cycle.

·         Working on major development work requests that required analysis of the current system and further designing, coding, preparation of test cases, unit testing and system testing of the new programs and online screens for new functionalities added to the PALLM system as required by the customer in all the major areas of the PALLM system described above.

·         Leading and implementing various tuning exercises in the project such as, reducing the frequency of potential processes, optimum utilization of important master files, periodic removal of unused reports and breaking of FTP steps to reduce number of abends.

Environment: 

Cobol, JCL, DB2, EZT, SYNCSORT 

 

 

professional experience

 

 

Company Name	Designation	Start date	End Date
Cyberthink Inc.	Hadoop Architect	15 MAY 2017	TILL DATE
Lowe’s Companies Inc.	Lead Analyst	12 MAY 2015	22 APR 2017
Syntel Inc.	Project Lead	25 APR 2011	11 MAY 2015
Ness Technologies	Sr. software engineer	23 DEC 2010	22APR 2011
CSC	Sr. Engineer	22 JAN 2007	23 NOV 2010
Satyam Computer Services Ltd.	Software Engineer	21 JUN 2004	18 JAN 2007

 

 

Education

 

·         Bachelor of Technology (B.Tech) at National Institute of Technology (NIT) Rourkela.