Senior Pipeline Engineer
- India
- Permanent
- Full-time
Min Experience: 6 years
Location: India
JobType: full-timeRequirements:We are looking for an experienced and motivated Senior Pipeline Engineer to join our engineering team and take ownership of designing, building, and managing high-performance data pipelines. This role demands strong expertise in orchestration frameworks such as Celery and Temporal, combined with advanced skills in database design and query optimization. The ideal candidate will have a passion for solving complex technical challenges and delivering reliable, scalable solutions that power data-driven decision-making across the organization.As a senior team member, you will play a critical role in shaping the future of our data infrastructure, working closely with cross-functional teams including data scientists, backend engineers, and product managers. You will also mentor junior engineers and drive best practices in orchestration, database efficiency, and scalable pipeline design.Key Responsibilities
- Pipeline Development & Management
- Design, develop, and maintain robust and scalable data pipelines for batch and real-time processing.
- Automate workflows for data ingestion, transformation, validation, and delivery.
- Ensure pipelines are fault-tolerant, performant, and cost-efficient.
- Workflow Orchestration
- Build and manage orchestration workflows using Celery, Temporal, or similar tools.
- Monitor, schedule, and optimize pipeline execution to reduce latency and improve reliability.
- Implement scalable solutions for dependency management, retries, and error handling.
- Database Design & Query Optimization
- Create efficient database schemas and models to support large-scale data operations.
- Optimize complex queries, indexing, and partitioning strategies to ensure high performance.
- Collaborate with engineering teams to enforce database best practices and scalability standards.
- Collaboration & Leadership
- Partner with product managers, analysts, and business stakeholders to understand requirements and translate them into technical solutions.
- Mentor and guide junior engineers in pipeline development and orchestration practices.
- Contribute to architecture reviews, design discussions, and technical documentation.
- Continuous Improvement & Compliance
- Proactively identify opportunities to improve pipeline performance and reliability.
- Stay updated on industry trends, orchestration frameworks, and database technologies.
- Ensure adherence to data security, governance, and compliance policies.
- 6+ years of experience in pipeline engineering, data engineering, or related fields.
- Strong hands-on expertise in Celery, Temporal, or other orchestration frameworks.
- Proficiency in database design, query optimization, and performance tuning.
- Solid programming background in Python, Java, or similar languages.
- Deep understanding of distributed systems, messaging queues, and event-driven architectures.
- Experience with cloud platforms (AWS, Azure, GCP) and containerized environments (Docker, Kubernetes).
- Strong problem-solving, analytical, and debugging skills.
- Excellent communication and collaboration abilities.
- Familiarity with real-time data streaming platforms like Kafka or Pulsar.
- Experience with data warehousing tools (Snowflake, BigQuery, Redshift).
- Knowledge of CI/CD pipelines, DevOps practices, and infrastructure automation.