
Data Engineer
- Telangana
- Permanent
- Full-time
- Lead the migration and modernization of data platforms, moving applications and pipelines to Google Cloud-based solutions.
- Architect and maintain cloud-based data infrastructure leveraging AWS or GCP services.
- Ensure data security and governance, enforcing compliance with industry standards and regulations.
- Develop and promote best practices for data modeling, processing, and analytics. Mentor and guide a team of data engineers, fostering a culture of innovation and technical excellence.
- Manage and scale data pipelines from internal and external data sources to support new product launches and ensure high data quality.
- Develop automation and monitoring frameworks to capture key metrics and operational KPIs for pipeline performance.
- Collaborate with internal teams, including data science and product teams, to drive solutioning and proof-of-concept (PoC) discussions.
- Develop and optimize procedures to transition data into production.
- Define and manage SLAs for data products and operational processes.
- Research and apply state-of-the-art methodologies in data and Platform engineering.
- Create and maintain technical documentation for sharing knowledge.
- Develop reusable packages and libraries to enhance development efficiency.
- Lead and drive the development and optimization of scalable data architectures and pipelines.
- Develop real-time and batch data processing solutions, integrating structured and unstructured data sources.
- We are looking for a candidate with 5-8 years of experience in Data Engineering and Application development. They must have a graduate degree in Computer Science or a related field of study. They must have experience with programming languages such as Python, Java & DS&Algo, Spark, and Scala. Expertise in Python and Spark is a must.
- 2 + years of AWS and Cloud technologies. Experience in data platform engineering, with a focus on cloud transformation and modernization.
- Hands-on experience building large, scaled data pipelines in cloud environments and handling of data in PBs.
- Experience with CI/CD pipeline management in GCP DevOps.
- Understanding of data governance, security, and compliance best practices.
- Experience working in an Agile development environment.
- Prior experience in migrating applications from legacy platforms to the cloud.
- Knowledge of Terraform or Infrastructure-as-Code (IaC) for cloud resource management.
- Familiarity with Kafka, Event Hubs, or other real-time data streaming solutions.
- Experience with legacy RDBMS (Oracle, DB2, Teradata) & DataStage/Talend
- Having background supporting data science models in production.