Lead Data Engineer
- Bangalore, Karnataka
- Permanent
- Full-time
Delta Air Lines (NYSE: DAL) is the U.S. global airline leader in safety, innovation, reliability and customer experience. Powered by our employees around the world, Delta has for a decade led the airline industry in operational excellence while maintaining our reputation for award-winning customer service. With our mission of connecting the people and cultures of the globe, Delta strives to foster understanding across a diverse world and serve as a force for social good. Delta has fast emerged as a customer-oriented, innovation-led, technology-driven business. The Delta Technology Hub will contribute directly to these objectives. It will sustain our long-term aspirations of delivering niche, IP-intensive, high-value, and innovative solutions. It supports various teams and functions across Delta and is an integral part of our transformation agenda, working seamlessly with a global team to create memorable experiences for customers.Responsibilities:Data Pipeline Development and Maintenance:
- Design, build, and optimize scalable ETL/ELT pipelines using AWS Serverless services (Lambda, Glue, s3, Airflow) to ingest data from diverse sources such as APIs, cloud platforms, and databases.
- Ensure pipelines are robust, efficient, and capable of handling large volumes of data.
- Aggregate and normalize marketing performance data from multiple platforms (e.g., Adobe Analytics, Salesforce Marketing Cloud, Teradata data warehouse, ad platforms).
- Implement data transformation and enrichment processes to support analytics and reporting needs.
- Develop and implement data validation and monitoring frameworks to ensure data accuracy and consistency.
- Troubleshoot and resolve issues related to data quality, latency, or performance.
- Partner with marketing teams, analysts, and data scientists to understand data requirements and translate them into technical solutions.
- Provide technical support and guidance on data-related issues or projects.
- Leverage cloud-based solutions and frameworks (e.g., AWS serverless services - Lambda, Glue, s3, Airflow) to streamline processes and enhance automation.
- Maintain and optimize existing workflows while continuously identifying opportunities for improvement.
- Document pipeline architecture, data workflows, and processes for both technical and non-technical audiences.
- Follow industry best practices for version control, security, and data governance.
- Stay current with industry trends, tools, and technologies in data engineering and marketing analytics.
- Recommend and implement innovative solutions to improve the scalability and efficiency of data systems.
- Lead and mentor other junior members within the team
- Bachelor of Science degree in Computer Science or equivalent
- At least 6+ years of post-degree professional experience as a data engineer developing and maintaining data pipelines
- Extensive experience with databases and data platforms (Teradata and AWS serverless - (Lambda, Glue, s3, Airflow)
- 5+ Hands-on experience in designing, implementing, managing large scale data and ETL solutions utilizing AWS Compute, Storage and database services (S3, Lambda, Redshift, Glue, Athena)
- 3+ years of experience as a tech lead with experience in mentorship and guidance, task planning and delegation, collaboration across teams, and technical communication
- Should have experience in architecture and design decisions, code quality and reviews, hands-on development and tech stack and tools management
- Should have good working knowledge on sprint planning and execution, risk management, performance and stability focus
- Proficiency in Python, SQL, PySpark, Glue, Lambda, S3 and other AWS tool sets
- Good understanding of data warehouses, ETL, AWS serverless
- Strong analytical and programming skills with the ability to solve data-related challenges efficiently
- At least 7+ years of post-degree professional experience as a data engineer developing and maintaining data pipelines
- Collaborate with product owners to understand requirements and translate them into technical specs
- Perform regular code reviews and advocate for best practices (testing, CI/CD, documentation)
- Proven ability to learn new data models quickly and apply them effectively in a fast-paced environment.
- Excellent communication skills with the ability to present complex data findings to both technical and non-technical audiences.
- Desired Airline industry experience
- Desired experience working with marketing and media data
- Experience working with SAS (Statistical Analysis System) to develop data pipelines
- AWS certifications: Solution Architect (SAA / SAP) or Data Analytics Specialty (DAS)
- Experience migrating data pipelines and systems to modern cloud-based solutions
- Familiarity with marketing data platforms