
Senior Data Engineering Consultant
- Bangalore, Karnataka
- Permanent
- Full-time
- Champion Data Engineering principles and foster a culture of data quality, scalability, performance, reliability, reusability, security, compliance, and observability
- Design and build scalable and reliable data jobs to process data in Hyperscale and other operational data platforms
- Support and maintain security and compliance controls at the database and data level in terms of HIPAA, PHI, PII and other governance requirements
- Automate infrastructure provisioning and configuration using Infrastructure as Code (IaC) tools for the database tech stack
- Manage and prioritize the security backlog, ensuring proactive identification and remediation of vulnerabilities for the Data infrastructure and pipelines
- Support and maintain systems during the migration from IBMi/AS400 and DB2 to a modern Azure-based stack, including Azure Hyperscale SQL, Azure Kubernetes Service (AKS), Azure App Services, Azure Data Factory (ADF) pipelines, Azure Databricks, Change Data Capture (CDC) and Kafka for real-time data synchronization
- Continuously evaluate and improve delivery processes, tooling, and team capabilities
- Develop and maintain data pipelines and workflows using Azure Data Factory (ADF) and Databricks
- Implement data integration and ETL processes to support analytics and reporting needs
- Ensure data quality, consistency, and reliability across the data engineering infrastructure
- Works with less structured, more complex issues
- Serves as a resource to others
- Comply with the terms and conditions of the employment contract, company policies and procedures, and any and all directives (such as, but not limited to, transfer and/or re-assignment to different work locations, change in teams and/or work shifts, policies in regards to flexibility of work benefits and/or work environment, alternative work arrangements, and other decisions that may arise due to the changing business environment). The Company may adopt, vary or rescind these policies and directives in its absolute discretion and without any limitation (implied or otherwise) on its ability to do so
- Undergraduate degree or equivalent experience
- 6+ years of experience in Data Engineering, or related engineering role
- 4+ years of experience querying and building data pipelines around high availability database systems (such as Hyperscale, Aurora, Mongodb Atlas)
- 3+ years of experience with Azure
- 3+ years of experience in scripting languages (e.g., Python, Bash)
- Hands-on experience with CI/CD tools such as Jenkins, Azure DevOps, or GitHub Actions
- Experience with Linux and/or Windows environments
- Experience working in distributed, cross-functional teams
- Familiarity with ITSM practices and automation
- Proficient in SQL, Python, PySpark, Scala or some data query language
- Proven solid communication, collaboration, and problem-solving skills