Senior Data Engineer
- Chennai, Tamil Nadu
- Permanent
- Full-time
- Work in collaborative environment including pairing and mobbing with other cross-functional engineers
- Work on a small agile team to deliver working, tested software
- Work effectively with fellow data engineers, product owners, data champions and other technical experts
- Demonstrate technical knowledge/leadership skills and advocate for technical excellence
- Develop exceptional Analytics data products using streaming, batch ingestion patterns in the Google Cloud Platform with solid Datawarehouse principles
- Be the Subject Matter Expert in Data Engineering and GCP tool technologies
- Experience in working in an implementation team from concept to operations, providing deep technical subject matter expertise for successful deployment. Implement methods for automation of all parts of the pipeline to minimize labor in development and production
- Experience in analyzing complex data, organizing raw data and integrating massive datasets from multiple data sources to build subject areas and reusable data products
- Experience in working with architects to evaluate and productionalize appropriate GCP tools for data ingestion, integration, presentation, and reporting
- Experience in working with all stakeholders to formulate business problems as technical data requirement, identify and implement technical solutions while ensuring key business drivers are captured in collaboration with product management
- Proficient in Machine Learning model architecture, data pipeline interaction and metrics interpretation.
- In-depth understanding of Google's product technology (or other cloud platform) and underlying architectures
- 4+ years of analytics application development experience required
- 4+ years of SQL development experience
- 3+ years of Cloud experience (GCP preferred) with solution designed and implemented at production scale
- Experience working in GCP based Big Data deployments (Batch/Real-Time) leveraging Terraform, Big Query, Big Table, Google Cloud Storage, PubSub, Data Fusion, Dataflow, Dataproc, Cloud Build, Airflow, Cloud Composer etc.
- 2 + years professional development experience in Java or Python, and Apache Beam
- Experience developing with micro service architecture from container orchestration framework
- Extracting, Loading, Transforming, cleaning, and validating data
- Designing pipelines and architectures for data processing
- 1+ year of designing and building Tekton pipelines