Lead Data Engineer
- Bangalore, Karnataka
- Permanent
- Full-time
- Architect and design scalable, efficient, and reliable data pipelines and platforms and Lead the adoption and best practices for Date Engineering tools.
- Ensure high standards for data quality, reliability, governance, and security.
- Drive the team’s technical roadmap, identifying areas for innovation and optimization
- Lead and mentor a team of data engineers, fostering growth, ownership, and excellence.
- Manage sprint planning, code reviews, and technical delivery in an agile environment.
- Collaborate with Product, Analytics, and Data Science teams to align on data needs and priorities.
- Own end-to-end delivery of projects, ensuring scalability, performance, and sustainability.
- 8–10 years of experience in data engineering or backend engineering with significant data pipeline responsibilities.
- Expertise in Python for data engineering tasks, including data transformation and orchestration.
- Hands-on experience with Kafka for real-time streaming and event-driven architectures.
- Proven experience with Airflow or equivalent scheduling/orchestration tools.
- Deep knowledge of Databricks and Apache Spark for distributed data processing.
- Proficient in SQL, with strong understanding of relational and distributed databases (e.g., Snowflake, Redshift, BigQuery).
- Experience in designing data architectures on cloud platforms (AWS, Azure, or GCP).
- Solid understanding of data modeling, data warehousing, and data governance.
- Experience in managing cost optimization and performance in cloud-based data platforms.
- Familiarity with MLOps or data products serving ML pipelines.