Databricks Platform, Engineering & Operations Lead
- Hyderabad, Telangana
- Permanent
- Full-time
IT ManagementGeneral Summary:Data Platform, Engineering and Operations lead will play a pivotal role in shaping the future of our data capabilities as part of strategic and critical Enterprise Data Transformation (EDT) program. We're seeking a visionary, hands-on leader to drive the build out of a modern enterprise-wide lake house, ensuring reliability, scalability, security, and compliance. and enabling cutting edge analytics solutions. This role will be leading a talented team of platform and data engineers, fostering a culture of excellence while balancing speed of delivery with innovation and operational excellence. This role will also collaborate with cross functional teams (Solution engineers, SMEs) to align data platform, engineering and operational efforts with business objectives.Key Responsibilities:
- Vision and Strategy: Define and lead the vision, strategy, roadmap for data platforms (data lakes, warehouses, governance, data quality, onservability etc.), aligning with business objectives and technological advancements to enable data platforms as the strategic enabler for AI
- Relentless Execution of Data Platform(s) Development: Oversee rapid incremental delivery of design, development, and deployment of data platforms/products/solutions, including platform engineering to facilitate frameworks, self-service templates and agentic automation ensuring seamless data integration, quality, compliance and security.
- Data Engineering: Lead core technical Data Engineering team for all building reusable frameworks for data pipeline development from ingestion to analytics/AI ready data for diverse data volumes, variety and frequency (streaming, batch, microbatches) with build in governance and security. Lead the migration of pipelines from legacy warehouses to a modern lakehouse hub, embrace GenAI and low code/no code mindset to drive velocity of pipeline development.
- Self Service and CoE Enablement: Partner with business stakeholders and internal data solutions team to empower business for self-service. Establish and lead CoE's for BI with platform and enablement from ideation to adoption at the core
- DecSecOps: Establish robust DevSecOps capabilities with a rigorous focus on automation of data pipeline delivery process with built in security controls
- Oversee Operations: Manage the day-to-day operations of data platforms, ensuring reliability, performance, and scalability. Establish and track data SLA's key performance indicators (KPIs) to measure and drive operational maturity
- Cross-Functional Collaboration: Collaborate with IT and Business with a product mindset to ensure data platforms meet business needs and to drive innovation. Also collaborate with teams across the globe and vendors to ensure high quality and timely delivery.
- Hands-On Leadership with Engineering Mindest: Be actively involved in details of execution (technical, architectural, delivery, troubleshooting) providing technical oversight with an engineering mindset to drive velocity of development work
- Challenge Status Quo: Encourage innovation and continuous improvement by challenging existing processes and proposing new solutions.
- 7+ years of supervisory or leadership experience.
- 3+ years of work experience in a role requiring interaction with senior leadership, leading modernization of Data platforms, Engineering and Operations.
- Hands-on technical experience with AWS, Databricks data platform
- Solid hands-on experience in Data Engineering, Data Analysis, Data Modelling and enabling Analytics solutions
- Passion to explore emerging technologies and perform Proof Of Concepts (POC)
- Ability to work independently and collaborate with business and cross functional teams
- Accountability and Ownership of the delivery in driving the projects to completionMinimum Qualifications: • 7+ years of IT-related work experience with a Bachelor's degree.
OR
9+ years of IT-related work experience without a Bachelor's degree.
- 4+ years in a leadership role in projects/programs.
- 14+ years of experience in data management, data engineering, or data science roles, with at least 7 years in a leadership position, leading data modernization and DW migration efforts.
- Strong understanding and hands-on experience with lakehouse medallion data architecture, data engineering, operations and govenance
- Proven implementation experience with data platforms Databricks, Fivetran, Collibra, Acceldata, AWS and other cloud based BI and GenAI solutions like ThoughtSpot, Tableau on Cloud.
- Thorough knowledge in establishing end-to-end DevSecOps and release management process from Git integration to Deployment
- Strong passion in driving the technical data engineering team with prime focus on reusable frameworks and GenAI agentic co-development for faster development and execution
- Ability to build and scale data operations team with state-of-the-art Observability & Monitoring capabilities for supporting the stakeholders in 24x7 model
- Experience with machine learning, artificial intelligence, and predictive analytics is a plus.
- Excellent leadership, communication, and collaboration skills in a global hybrid delivery model
- Strong problem-solving skills with strategic thought process and self-initiative and proctive soft skills
- Proven track record of leading a strong technical team and collaborating with data engineers, architects, modelers, analysts and data scientists