Lead Data Engineer

FM India

Bangalore, Karnataka
Permanent
Full-time

4 hours ago
Apply easily

Job DescriptionAbout us:We are a highly successful 190-year-old, Fortune 500 commercial property insurance company of 6,000+ employees with a unique focus on science and risk engineering. Businesses worldwide trust our expertise to protect their assets, relying on our comprehensive risk assessments and robust, engineering-based insurance solutions to safeguard against fire, natural disasters, and other perils. Serving over a quarter of the Fortune 500 and major corporations globally, we deliver data-driven strategies that enhance resilience, ensure business continuity, and empower organizations to thrive.FM India is a strategic location for driving our global operational efficiency. Our presence in India allows us to leverage the countrys talented workforce and advance our capabilities to serve our clients better. We have diverse corporate functions that emphasize research, advanced technologies like AI and analytics, risk engineering, research, finance, marketing, HR, etc. working together to provide innovative solutions and nurture lasting relationships from co-workers to clients.Role Title: Lead Data EngineerPosition Summary:The Lead Data Engineer is responsible for data analysis, data modelling, data collection, data integration, data storage and preparation of data for consumption. This includes creating and managing data infrastructure, data pipeline design, implementation and data verification. Along with the team, responsible for ensuring the highest standards of data quality, security and compliance. Additionally, this role will utilize AI to implement methods to improve data reliability and quality, combine raw information from different sources to create consistent data sets. The incumbent will need to be proficient in DataOps and be able to provide technical expertise to other Data Engineers. Incumbents will design and build data solutions and integrations which may involve diverse data platforms, software, hardware, technologies and tools. Uses available approved technologies and recommends solution options. Incumbents may design and build data solutions and integrations which may involve diverse development platforms (including 3rd party systems), software, hardware, technologies and tools. The Lead Data Engineer is the third level position in the Data Engineer job family. Those holding this position are typically assigned to be the technical lead for medium to large projects and initiatives, and provide direction, work effort estimates and delivery coordination to junior data engineers. This role balances independent contribution with mentoring and technical leadership of other data engineers and analytics practitioners.Job Responsibilities:Lead the data acquisition and integration strategy:

Define and implement the strategy for bringing structured and unstructured data sources into Data Analytics. Perform as the technical lead with the Product teams (Underwriting and Risk; Client Service, Sales and Marketing; Claims; Account and Location Engineering) as well as emerging data sources (purchased data sets; external data; etc.)
Partner with product owners, developers, solution architects, business analysts, data engineers, data analysts, data scientists and others to understand data and reporting needs
Assist in or lead the development and implementation of data solutions using data modeling techniques and technologies such as Fabric, Synapse Analytics, SQL Server, Azure services, and others as required
Validate solutions are accurate through detailed and disciplined testing methodologies and taking opportunities to use AI in this process.
Responsible for ensuring the developed solutions are accurate, good quality, and delivered according to schedule
Ensure tables and views are designed for data integrity, efficiency and performance, and are easy to comprehend

Move and Store Data:

Design and implement the data flow, infrastructure pipelines, ETL/ELT, structured and unstructured data movement and storage solutions
Design data models and data flows into and out of Data Analytics databases
Understand and design data relationships between business and data subject areas
Follow standards for naming conventions, code documentation and code review
Support data exploration and transformation needs
Lead the strategy and implementation of data cleansing and mentor other team members on data quality and data cleansing tasks as needed
Define and implement approach to data profiling to identify data anomalies and resolve issues
Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
Lead the build-out of the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL and Azure technologies

Production Support and Operations:

Lead the definition and creation of prototypes and proof of concepts for the selected solutions, and implement complex data engineering projects
Project planning and facilitation
Create the development project implementation plans for strategic data analytics projects
Prioritize work within assigned projects and assign work to development team
Create estimates, timelines and resourcing needs for development team
Lead the team in the development and implementation of the project
Support production applications and reduce maintenance
Identify repeat support issues and implement solutions to reduce those issues
Manage and address operational data issues by determining and fixing root cause
Mentor developers, data analysts and data scientists who need to interact with data
Liaison with dba(s) and Infrastructure Teams on the infrastructure and configuration needed to support Data Analytics solutions.
Monitor system performance and execute opportunities for optimization
Monitor storage capacity and reliability
Remediate production issues quickly, with appropriate validation and deployment steps
Supply ad-hoc data extracts and analysis to respond to tactical business needs
Provide clear and concise communication to users, teammates and leadership

Support planning and execution of team priorities:

Identify work tasks and capture them in team backlog
Participate in backlog and technical debt planning and prioritization
Resolve colliding priorities and escalate as needed
Network with product teams and infrastructure teams to keep abreast of databases, business processes and technology changes which may impact data delivery.

Skill and Experience:

6-7 years of experiencerequired to perform essential job functions.
Ability to read and create data models Relational (3rd Normal Form) and non-relational (Inmon/Kimball) database theory
Strong SQL skills
Build scalable, fault-tolerant data pipelines
Design and support ETL/ELT processes
Design, build, maintain data lakes and data warehouses
Programming languages SQL, Python, Spark, Kafka, Azure, JSON
Implement data governance and data security measures

Must Have Skills:

SQL
Spark / Pyspark, Data lakes, Data warehouses
Fabric Platform,
Ability to read and create data models
ETL/ELT Pipeline development experience
Ability to keep pace with rapidly evolving cloud services and AI technologies
Demonstrate importance of data governance, quality and observability
Preferred: Orchestration and automation knowledge for managing complex data workflows
Communication and collaboration skills

Good to Have Skills:

Python,
Azure Data Factory, Kafka,
Synapse Analytics Platform,
Relational (3NF) and non-relational (Inmon/Kimball) database theory
AI/LLM Integration experience

FM India