Employer Name: QUALITEST GROUP

Designation: Senior Data Engineer

Duration: April 2021-July2022

Client: WALMART LABS, USA

• Collaborated with cross-functional teams to gather, analyze, and translate complex data sets into actionable insights, enabling evidence-based product enhancements and strategic decisions.

• Led end-to-end data pipeline optimization for Walmart, utilizing PySpark and Airflow to streamline data

• extraction, transformation, and loading processes. This resulted in a 30% reduction in data processing time, positively impacting the product's real-time data availability.

• Orchestrated Airflow Directed Acyclic Graphs (DAGs) with comprehensive error handling, retry mechanisms,

• and data quality checks. Additionally, implemented an automated email alert notification system to promptly notify stakeholders about potential issues, ensuring seamless and error-free data flow.

• Contributed to the development of backend REST APIs using MuleSoft, facilitating seamless data integration of enterprise systems. These APIs were deployed on AWS, enhancing the product's connectivity and interoperability.

• Devised and implemented a batch processing ETL pipeline API that boosted the efficiency of ingesting 1 million records per day into a SQL database by an impressive 40%, thus improving the product's data processing capabilities.

• Successfully designed and executed deployment pipelines using Jenkins and Git, resulting in a 50% reduction in time to deploy APIs. This efficient deployment process contributed to faster feature releases and improved overall product agility.