Azure Data Engineer - Microsoft Fabric (Initial 6-12 months Dubai) then India Remote
- India
- Permanent
- Full-time
- Design, develop and manage data pipelines on Microsoft Azure for internal and external data sources.
- Develop ingestion frameworks for batch and real-time data.
- Ensure data pipelines are optimized for reliability and scalability.
- Ensure the scalability, security, and performance of our cloud architecture while aligning with business goals.
- Work closely with solution architect, backend developers, data governance specialists
- Collaborate with business stakeholders to translate functional requirements into scalable technical solutions.
- Hands-on experience with Microsoft Fabric platform in building end-to-end data solutions.
- Hands-on experience with building pipelines using MS Fabric.
- Build and optimize data lake zones (raw, curated, consumption)
- Proficiency in developing business and dataflow logic for data processing using tools like PySpark, TSQL, Stored Procedures
- Experience implementing message queues using Azure Service Bus
- Integrate cloud-based solutions with on-premises systems.
- Implement infrastructure as code (IaC) solutions using Terraform for consistent, automated provisioning
- In-depth knowledge of Azure architecture components, including compute, storage, networking, and security.
- Optimize cloud storage solutions for performance and cost.
- Establish monitoring and logging solutions using Azure Monitor, Log Analytics, Application Insights, and third-party tools.
- Configure automated scaling solutions to handle workload fluctuations.
- Create and maintain comprehensive architecture documentation, including diagrams, processes, and technical specifications