Candidates About

 

Praveen K Depur

Professional Synopsis

q  10 years of IT industry experience with consistently increasing responsibilities in Business Analysis, Design, Development and Testing.

q  Profound knowledge in System Analysis, Design and Development in the fields of databases, data warehouse and client server technologies.

q  Over 3 years’ experience in installing, configuring, testing Hadoop V2 ecosystem components.

q  Expertise in developing Hadoop ecosystem components like MapReduce,HDFS, Pig, Hive, Sqoop, Oozie, Impala, Mahout, Hbase, Yarn, Flume,Spark,Scala,Python.

q  Capable of processing large sets of structured, semi-structured and unstructured data and supporting systems application architecture.

q  Able to assess business rules, collaborate with stakeholders and perform source-to-target data mapping, design and review.

q  Familiar with data architecture including data ingestion pipeline design, Hadoop information architecture, data modelling and data mining, machine learning and advanced data processing. Experience optimizing ETL workflows.

q  In depth understanding/knowledge of Hadoop Architecture and various components such as HDFS, Job Tracker, Task Tracker, Name Node, Data Node.

q  Intense hands on experience in writing complex Map reduce jobs, Pig Scripts and Hive data modelling.

q  Experience in converting MapReduce applications to Spark.

q  Experience in converting HiveQL to SparkQL

q  Good working experience using Sqoop to import data into HDFS from RDBMS and vice-versa.

q  Good knowledge in using Job scheduling and workflow designing tools like Oozie.

q  Good Knowledge on Hadoop Cluster administration, monitoring and managing Hadoop clusters using Cloudera Manager.

q  Have good experience creating real time data streaming solutions using Apache Spark/Spark Streaming/Apache Storm, Kafka and Flume.

q  Developed a scalable, cost effective, and fault tolerant data ware house system on Amazon EC2 Cloud.

q  Experience in monitoring and controlling large-scale cloud (AWS) infrastructure

q  Extending Hive and Pig core functionality by writing custom UDFs.

q  Good understanding of Data Mining and Machine Learning techniques.

q  Experience in handling messaging services using Apache Kafka.

q  Developed Python scripts to format and create daily transmission files. 

q  Strong Experience in Java, J2EE, Spring, Struts and Hibernate

q  Extensive experience in creation of ETL Jobs and transformations using Datastage Designer to move data from multiple sources into target area.

q  Worked extensively with complex Jobs using Joins, Lookups, Remove duplicates, aggregator etc.

q  Expertise in implementing complex Business rules by creating robust jobs, reusable shared containers, Routines and shell scripts.

q  Experienced in performance tuning of Datastage jobs.

q  Worked on various databases like Oracle 9i, IBM DB2, MS Sql server etc.

q  Very strong knowledge in relational databases (RDBMS), data modeling and in building Data Warehouse, Data Marts using Star Schema and Snowflake Schema.

q  Extensive knowledge in pulling the data from different source systems.

q  Consolidate data from various departments into the data warehouse by identify the Common data and solve the discrepancy on data values and created various data marts.

q  Good knowledge of OLAP reporting tools like Cognos, Business objects.

q  Excellent working knowledge of UNIX Shell Scripting and automation of ETL processes using Autosys on platforms such as Unix.

q  Experience in Project Planning, Schedule & Resources management, Communication Management, Business Continuity Planning,  Training & Development and People Management

q  Possess excellent interpersonal, communication and organizational skills with proven abilities in training & development, customer relationship management and planning

q  Proficient in managing & leading teams for running successful process operations & experience of developing procedures, service standards for business excellence

 

Key Specialties

 

 

 

 

 

q  Hadoop Eco Systems, Hive, Pig, Sqoop, Spark, Flume, Python, Oozie, MapReduce, Kafka.

q  Java, J2EE, Springs, Struts & Hibernate

q  ETL Datastage

q  UNIX Shell Scripting

q  Scheduling Tools: Autosys,CA7,Ctrl-M,UC4.

 

 

Organizational Summary

 

 

 

 

 

Software Engineer at Comtec Consultants from March 2017 to till date

Application Architect-BigData at IBM Global Business Services from January 2014 to March 2017

Data Specialist – ETL Datastage at IBM Global Business Services from April 2010 to December 2013.

Escalation Lead – ETL Datastage with CSS Corp Pvt Ltd from August 2008– April 2010.

 

Experience

 

Client: Comcast Cable Corporation

Company: Comtec Info

Project: Comcast’s Residential Marketing (CRM)

Role: Application Developer-Bigdata

October 2017 – Till Date

Description:  Comcast’s Residential Marketing will transition from product-focused to a prospect and customer focused management across channels and devices. In the long term, Comcast will drive connects, upsells, and cross-sells while streamlining marketing processes using a fully orchestrated stack of marketing technologies.

 

Responsibilities:

Ø  Created generic framework to load the data from Teradata tables on HDFS using TPT( Teradata Parallel Transporter )

Ø  Most of the data is already loaded on to JRNL schema in HDFS , we are creating dimension tables in Semantic Schema with joining teradata tables and JRNL schema by creating external tables

Ø  Write HQLs to load the data on to HDFS

Ø  Developed Scripts and automated data management from end to end and sync up between all the clusters.

Ø  Extensively used the Hue browser for interacting with Hadoop components.

Ø  Documented the systems processes and procedures for future references.

Ø  Actively participated in software development lifecycle (scope, design, implement, deploy, test), including design and code reviews, test development, test automation.

Ø  Worked extensively on UC4 workflows to schedule the Hadoop jobs

Ø  Worked on creating Data Integration Framework Automation to ingest data from databases, files.

Ø  Created multiple generic scripts to load sftp to inbound and outbound files to reduce manual intervention

Ø  Worked closely with business to understand the requirements and deliver the deliverables with 100% Customer Satisfaction and in specified amount of time.

Ø  Not only worked with team and also lead the team of 10 to deliver defect free code to customers which yielded great applause from everyone.

 

 

Client: BarclayCardUS

Company: Comtec Info

Project: Real Time Fraud Data Re-engineering

Role: Application Developer-Bigdata

March 2017 – September 2017

Description:  BarclaycardUS has taken an initiative to find out real time fraud on their credit card transactions. All the transactions are captured by TSYS and sent as an event file to Barclays. This used to be once in a day process. This project is to re-engineer the entire model to get the event as soon as it happens and load in to the data mart, so that fraud strategy team will run the rules on the data to identify the fraud.

 

Responsibilities:

Ø  TSYS sents out events using IBM Message Queue, we have kafka receiver reads the data and sends it using flume and there is kafka consumer reads the data depending on the message topic

Ø  Kafka Consumer and receiver are written in scala

Ø  Once data the received it will be written in HDFS file system and a text file and with replication factor of 3

Ø  Then the data which is there is in HDFS is agnostified as per Barclays and loaded on to data mart for Fraud Strategy team to run their rules on it , all this happens real time.

Ø  Developed a data pipeline using Kafka and Spark to store data into HDFS and performed the real-time analytics on the incoming data.

Ø  Configured Spark streaming to receive real time data from the Kafka and store the stream data to HDFS using Scala.

Ø  Developed Spark Programs for Batch and Real time processing.

Ø  Implementing Spark Streaming using Scala and Sparksql for faster testing and processing of data of real time.

Ø  Using Spark streaming consumes topics from distributed messaging source Kafka and periodically pushes batch of data to Spark for real time processing

Ø  Responsible for reading text files in hadoop cluster, cleansing, transforming and writing data in avro, parquet format with Apache Spark on Scala

Ø  Performed Data Transformations from regular RDBMS databases to NoSQL Databases

Ø  Worked Extensively on NoSQL DataMarts

q    Co-Ordination with Offshore to ensure defect free delivery and in time.

q    Well Versed with Agile methodologies.

 

 

Client: SunTrust Banks

Company: IBM

Project: Atlas Data Lake

Role: Application Architect- Big Data

January 2013 – March 2017

Description: To challenge the regulatory pressure, cost efficiency, and speed to market with offerings based on client data insights, SunTrust came up with new concept called Data Lake on Hadoop environment. Data Lake is a single, trusted, controlled, read only and accessible source of data for all consumption across SunTrust. This assignment involves pulling all sources data from different LOB’s of SunTrust into HDFS (Hadoop Data File System). It involves analysis of source systems, capturing source types, connections, and entities details. In the process of lake build we will be cleansing the source data, applying the TDQ, maintenance of history through CDC and capturing of lineage/profile results at MHUB

 

Responsibilities:

q    Responsible to

Ø  Pull data from different DB (Oracle, DB2, SQL Server...) and Mainframe source systems.

Ø  Push data from flat file sources (can be external and internal) to data lake (Hadoop V2 environment).

Ø  Collect and aggregate large amounts of log data using Apache Flume and staging data in HDFS for further analysis

Ø  Developed OOZIE workflows for the Application execution

Ø  Implementing Spark Streaming using Scala and Sparksql for faster testing and processing of data of real time.

Ø  Using Spark streaming consumes topics from distributed messaging source Kafka and periodically pushes batch of data to Spark for real time processing

Ø  Received streams will be stored in memory using RDD’s.

Ø  Used DataFrame API in Scala for converting the distributed collection of data organized into named columns.

Ø  Performed complex joins of DataFrames (Inner, Outer, Left Outer , Semi- Join ) using spark SQL.

Ø  Wrote complex Hive queries and UDFs.

Ø  Writing Pig scripts for data processing

Ø  Worked on reading multiple data formats on HDFS using PySpark

Ø  Involved in converting Hive/SQL queries into Spark transformations using Spark RDDs, Python and Scala.

Ø  Developed multiple POCs using PySpark and deployed on the Yarn cluster, compared the performance of Spark, with Hive and SQL/Teradata.

Ø  Analysed the SQL scripts and designed the solution to implement using PySpark

Ø  Optimizing Hadoop Map Reduce code, Hive/Pig scripts for better scalability, reliability and performance.

Ø  Execution of Hadoop ecosystem and Applications through Apache HUE

Ø  Handled Full CDC, Delta processing, Incremental updates using hive and processed the data in hive tables

Ø  Developed PIG Latin scripts to extract data from source system.

Ø  Developed java Map reduce XML PARSER programs to process XML files using XSD's and XSLT's as per the clients requirement and used to process the data into Hive tables.

Ø  Implemented Hive tables and HQL Queries for the reports. Written and used complex data type in Hive. Storing and retrieved data using HQL in Hive. Developed Hive queries to analyse reducer output data.

Ø  Developed Scripts and automated data management from end to end and sync up between all the clusters.

Ø  Extensively used the Hue browser for interacting with Hadoop components.

Ø  Documented the systems processes and procedures for future references.

Ø  Actively participated in software development lifecycle (scope, design, implement, deploy, test), including design and code reviews, test development, test automation.

Ø  Cluster coordination services through Zoo Keeper

 

q    Responsible for importing data from various RDBMS and Mainframe sources in to  Hadoop environment with the tool using Sqoop import in full/Incremental.

 

q    Responsible for creation of Hive tables once data is landed on Hadoop, it involves

Ø  Creation of HIVE databases

Ø  Preparation of create hive table SQL by referring Hadoop file schema

Ø  Creation of external/internal Hive tables and portioning as of run date

Ø  Moving the data from external hive tables to Internal hive tables

Ø  Verifying the hive table's data with source systems data to make sure data is landing correctly on lake, in this unit testing process we have used extensively Hadoop tools Hive/Beeline sql's and Pig through Hue portal.

Ø  Responsible for reading text files in hadoop cluster, cleansing, transforming and writing data in avro, parquet format with Apache Spark on Scala

q    Co-Ordination with Offshore to ensure defect free delivery and in time.

q    Working with BIAS Team to address admin related issues.

q    Well Versed with Agile methodologies.

 

           

           

Client: Teachers Insurance and Annuity Association – College Retirement Equities Fund (TIAA – CREF)          

Company: IBM                                                                          

Project: Batch Production Support

Role: ETL Architect- Datastage

January 2012 – December 2012

 

 

TIAA has established a centralized Data Warehouse Environment to support its various reporting initiatives (e.g. ERISA), leveraging common, consistent information.In order to support the Data Warehouse and extend the development capability for new functionalities in a more cost effective fashion, TIAA is looking to create an offshore capability to meet these needs

 

Responsibilities:

q  Created Design and mapping documents and explained to Development team to create the Datastage Jobs

q  Responsible for working with various teams to deliver the Marketing, Sales, Opportunity, Revenue, segmentation, Transactional data to business teams on time.

q  To provide daily, weekly, monthly data before business hours to ensure high quality of customer system reporting.

q  Data analysis and fixes related to data inconsistency.

q  Performance analysis and performance tuning related to ETL process and Database queries.

q  Implementing the shell scripting as per the ETL flow requirements

q  Responsible for Processing Ad-Hoc data request from business within agreed time lines.

q  Responsible for knowledge transfer to team members on Datastage, Teradata and other products that are in use for this project.

q  Responsible for effective communication between the project team and the customer. Provide day to day direction to the project team and regular project status to the customer.

q  Establish Quality Procedure for the team and continuously monitor and audit to ensure team           meets quality goals.

q  Scheduling the datastage jobs using autosys calendars

q  Supporting the batch runs through autosys and ensure loads are completed within in the SLA’s.

q  Monitoring the production jobs and notifying the concern team in case of failures.

q  Implementing the production fixes and deploying it in the prod environment.

q  Continuous monitoring of file arrival and informing the SOR team in case of file arrival getting delayed.

q  Sending the complete report of the run stats every day after the load completion.

q  Coordinating with different teams such as SOR,DBA etc as part of production support.

q  Understand the business needs and the requirements and design the jobs accordingly.

q  Preparation of Autosys jils for scheduling the ETL jobs.

 

 

Environment: Datastage 8.5 /11.3 version, Ms SQL server, Teradata , Autosys, CA7 ,Ctrl M, Informatica 9.1, WINSCP, Unix Shell scripting, Oracle , Star Team.

 

 

 

 

 

 

 

 

 

 

Client: Bank of America                                                                                                

Project: IIS MDE

Company: IBM

Role: Technical Lead-ETL.Datastage

March 2011 – December 2012

 

 

Bank of America is an American multinational banking and financial services corporation headquartered in Charlotte. They are planning to deploy their software and hardware into a master data hub and employed Datastage as ETL tool for this purpose and had DB2 as the RDBMS which pulls data from Sql server. There are two types of loads being done 1) full File compare 2) change data capture. For full file compare we have a cdc process so only the inserted or updated records will be loaded into target table and deletes are being captured in the delete table. For the change data capture the load will be based on the previous max modified date.

 

Responsibilities:

q  Understand the business needs and the requirements and design the jobs accordingly.

q  Analyse the data in determining the cleansing requirements and effective implementation of Data Cleansing through Qualitystage.

q  Extensively used ETL to load data from MS SQL server to target IBM DB2 tables.

q  Involved in different reviews like Internal and external code review, weekly status calls, issue resolution meetings and onsite code acceptance meeting

q  Developed ETL jobs using various stages like ODBC Connector,Lookup,Join,Aggregator,

Transformer, Sort, Remove Duplicate, Dataset etc.

q  Performed Data profiling through Information Analyzer client..

q  Performance tuning of the ETL jobs and also problem analysis and Issue resolution

q  Got involved in issue logging/tracking, risk identifying/maintaining.

q  Preparation of Autosys JILs for scheduling the ETL jobs.

q  Responsible for the Autosys migration to different environments.

 

Environment: Datastage 8.5 version, Ms SQL server, IBM DB2

 V2R6, HP-UNIX 8000/9000, TOAD 8.1.5, Autosys, Cognos 8.1.

 

Achievements

q  Got the Orion Award for the Year 2012   

 

 

 

 

 

 

 

 

 

 

 

Client: Bank of America

Company: IBM                                                                                     

Project: IAF Transformations

Role: Technical Lead- ETL.Datastage

              April 2010 – Feb 2011

 

Bank of America is migrating current data warehouse model to IBM’s Banking Data Warehouse Model (BDW), which is the baseline logical and physical data model customized for Bank of America (BANA). Data is loaded to the newly created warehouse tables using Datastage ETL. The bank is also si