
Assistant Manager - Data Engineering
- Chennai, Tamil Nadu
- Permanent
- Full-time
- Lead and execute data engineering projects from inception to completion, ensuring timely delivery and high quality.
- Build and optimize data architectures for operational and analytical purposes.
- Collaborate with cross-functional teams to gather and define data requirements.
- Implement data quality, data governance, and data security practices.
- Manage and optimize cloud-based data platforms (Azure,AWS).
- Develop and maintain Python/PySpark libraries for data ingestion, Processing and integration with both internal and external data sources.
- Design and optimize scalable data pipelines using Azure data factory and Spark(Databricks)
- Work with stakeholders, including the Executive, Product, Data and Design teams to assist with data-related technical issues and support their data infrastructure needs.
- Develop frameworks for data ingestion, transformation, and validation.
- Mentor junior data engineers and guide best practices in data engineering.
- Evaluate and integrate new technologies and tools to improve data infrastructure.
- Ensure compliance with data privacy regulations (HIPAA, etc.).
- Monitor performance and troubleshoot issues across the data ecosystem
- Bachelors or master’s degree in computer science, Information Systems, Statistics, Math, Engineering, or related discipline.
- Minimum 10 + years of solid hands-on experience in data engineering and cloud services.
- Experience in leading and mentoring Team members.
- Good Experience in Azure data factory (ADF), Databricks , Python and PySpark.
- Good experience in modern data storage concepts data lake, lake house.
- Experience in other cloud services (AWS) and data processing technologies will be added advantage.
- Ability to enhance , develop and resolve defects in ETL process using cloud services.
- Experience handling large volumes (multiple terabytes) of incoming data from clients and 3rd party sources in various formats such as text, csv, EDI X12 files and access database.
- Experience with one or more programming languages such as Java, .Net in an application development environment is highly preferred.
- Experience with software development methodologies (Agile, Waterfall) and version control tools
- Highly motivated, strong problem solver, self-starter, and fast learner with demonstrated analytic and quantitative skills.
- Good communication skill.
- Experience in different cloud providers
- Experience in Programming
- Experience in DevOps