
Data Engineer - Director - Data & Analytics Engineering
- Bangalore, Karnataka
- Permanent
- Full-time
The Firmwide Data Office department is recruiting for an enthusiastic, dynamic, hands-on and delivery focused Data and ETL Engineer.
As a member of our Software Development team, we look first and foremost for people who are passionate about solving business problems through innovation and engineering practices. You'll be required to apply your depth of knowledge and expertise to all aspects of the software development lifecycle, as well as partner with stakeholders to stay focused on business goals. We embrace a culture of experimentation and constantly strive for improvement and learning. You'll work in a collaborative, trusting, thought-provoking environment-one that encourages diversity of thought and creative solutions that are in the best interests of our customers globally. You'll combine your design and development expertise with a never-ending quest to create innovative technology through solid engineering practices. You'll work with highly inspired and inquisitive team of technologists who are developing & delivering top quality technology products to our clients & stakeholders.Responsibilities:
- Design, development and support of Data Ingestion Pipelines for Knowledge Graph using Python & Snowflake.
- Collaborate with business and other technology teams to translate business requirements into innovative solutions implementing performant, scalable, resilient distributed applications
- Minimum 5+ Years of strong development experience building robust Data ingestion pipelines using tools like Python, PySpark and Snowflake as well as Autosys/Airflow.
- Strong experience in writing multi-processing scripts using Python and distributed processing scripts using PySpark.
- Strong problem-solving skills, business acumen, and demonstrated excellent oral and written communication skills with both technical and non-technical audiences
- Experience in closely working with application teams and managing all the DB related activities such as DB modeling, design, development, maintenance and support
- 2+ Years of strong experience in python for automating data cleansing, data re-formatting, data transformations
- Strong Skills in Relational Databases including database design, writing complex queries and stored procedures, performance tuning
- Strong Data Modeling skills - Enterprise Data Model (OLTP) and Dimensional modeling. Well versed with principles, techniques and best practices of data modeling.
- Extensive experience in designing and developing of complex mappings, applying various transformations such as lookup, source qualifier, update strategy, router, sequence generator, aggregator, rank, stored procedure, filter, joiner and sorter transformations
- Experience in integration of various data sources like DB2, SQL Server, and Flat Files into the staging area
- Expert in designing Parallel jobs using various stages like Join, Merge, Lookup, Remove duplicates, Filter, Dataset, Lookup file set, Modify, Aggregator, XML parsing stages.
- Knowledge about DevOps implementation in Data space
- Extensive experience working on UNIX/Linux platform, Python scripting and managing
- Bachelor's/Master's Degree in Computer Science, Computer Engineering, Data Analytics or related field
- Experience with Big Data processing and technologies like Hadoop, Spark, Kafka
- Experience with Graph Databases and Redis Data Stores.
- Exposure to LLM Technologies.
- Experience and expertise working with data wrangling and visualization tools like Dataiku, Tableau and Power BI.
- Financial Services experience will be a strong plus