Azure Data Engg

• Design, build, and maintain scalable and efficient data pipelines using Databricks Notebooks, Workflows, and Delta Live Tables (DLT).
• Leverage Databricks SQL, PySpark, and Python for data transformation, enrichment, and analytics.
• Implement and manage Delta Lake architectures to support real-time and batch data processing.
• Use Unity Catalog for data governance, access control, and metadata management.
• Orchestrate complex workflows using Databricks Workflows and integrate with external schedulers if needed.
• Optimize performance of Spark jobs and manage cluster configurations for cost and efficiency.
• Collaborate with data scientists, analysts, and business stakeholders to deliver high-quality, production-grade data solutions.
• Ensure data quality, lineage, and observability using Databricks tools and best practices.
• Integrate with AWS S3 or Google Cloud Storage (GCS) for data ingestion and storage.
• Follow Agile methodologies and use Git for version control and CI/CD pipelines.

• Bachelor's degree in Computer Science, Information Technology, or a related field.
• 6+ years of experience as a Data Engineer, with 3+ years of hands-on experience with Databricks.
• Strong expertise in Databricks Lakehouse Platform, including:

• Delta Lake
• Databricks SQL
• Unity Catalog
• Databricks Workflows
• Delta Live Tables (DLT)
• Proficiency in PySpark, Python, and SQL.
• Experience with cloud storage services like AWS S3 or Google Cloud Storage.
• Solid understanding of data modeling, data warehousing, and ETL/ELT best practices.
• Exposure to BI tools like Tableau, Power BI, or Databricks SQL Dashboards.
• Experience with CI/CD pipelines, Git, and automated testing for data pipelines.

Diverse Lynx