Data Engineer

• Productionize pipelines for large, complex data sets which meet technical and business requirements.
• Partner with data asset managers, architects, and development leads to ensure a sound technical solution.
• Follow, and contribute to, coding standards and best practices to ensure pipelines and components are efficient, robust, cost effective and reusable.
• Identify, design, and implement internal process improvements.
• Optimize Spark jobs for performance and cost. Tune configurations, minimize shuffles, and leverage advanced techniques like broadcast joins, caching, and partitioning.
• Ensure data quality, reliability, and performance by implementing best practices for data validation, monitoring, and optimization.
• Monitor and troubleshoot data pipelines and workflows to ensure seamless operation.
• Stay updated on the latest Databricks features, tools, and industry trends to continuously improve data engineering practices.
• Strong understanding of distributed computing concepts and big data processing.
• Excellent problem-solving skills and the ability to work collaboratively in a team environment.

• Strong skills with Python, SQL, Delta Lake, Databricks, Spark/Pyspark, Github and Azure.
• You will be expected to Attain and/or maintain technical certifications related to the role (Databricks, Azure)
• Ability to use and implement CI/CD and associated tools such as Github Actions, SonarQube, Snyk
• Familiarity or experience in one or more modern application development framework methods and tools (e.g. Disciplined Agile, Scrum).
• Familiarity or experience with a range of data engineering best practices for development including query optimization, version control, code reviews, and documentation
• The ability to build relationships and work in diverse, multidisciplinary teams
• Excellent communication skills with business intuition and ability to understand business systems, versatility, and willingness to learn new technologies on the job

Procter & Gamble