Associate III - Data Engineering

Bangalore, Karnataka
Permanent
Full-time

13 days ago

Job Description:Role Proficiency:This role requires proficiency in data pipeline development including coding and testing data pipelines for ingesting wrangling transforming and joining data from various sources. Must be adept at using ETL tools such as Informatica Glue Databricks and DataProc with coding skills in Python PySpark and SQL. Works independently and demonstrates proficiency in at least one domain related to data with a solid understanding of SCD concepts and data warehousing principles.Outcomes: * Collaborate closely with data analysts data scientists and other stakeholders to ensure data accessibility quality and security across various data sources.rnDesign develop and maintain data pipelines that collect process and transform large volumes of data from various sources.

Implement ETL (Extract Transform Load) processes to facilitate efficient data movement and transformation.
Integrate data from multiple sources including databases APIs cloud services and third-party data providers.
Establish data quality checks and validation procedures to ensure data accuracy completeness and consistency.
Develop and manage data storage solutions including relational databases NoSQL databases and data lakes.
Stay updated on the latest trends and best practices in data engineering cloud technologies and big data tools.

Measures of Outcomes: * Adherence to engineering processes and standards

Adherence to schedule / timelines
Adhere to SLAs where applicable
# of defects post delivery
# of non-compliance issues
Reduction of reoccurrence of known defects
Quickly turnaround production bugs
Completion of applicable technical/domain certifications
Completion of all mandatory training requirementst
Efficiency improvements in data pipelines (e.g. reduced resource consumption faster run times).
Average time to detect respond to and resolve pipeline failures or data issues.

Outputs Expected:Code Development: * Develop data processing code independentlyensuring it meets performance and scalability requirements.Documentation: * Create documentation for personal work and review deliverable documentsincluding source-target mappings
test cases
and results.Configuration: * Follow configuration processes diligently.Testing: * Create and conduct unit tests for data pipelines and transformations to ensure data quality and correctness.

Validate the accuracy and performance of data processes.

Domain Relevance: * Develop features and components with a solid understanding of the business problems being addressed for the client.

Understand data schemas in relation to domain-specific contexts

such as EDI formats.Defect Management: * Raisefix
and retest defects in accordance with project standards.Estimation: * Estimate timeeffort
and resource dependencies for personal work.Knowledge Management: * Consume and contribute to project-related documentsSharePoint
libraries
and client universities.Design Understanding: * Understand design and low-level design (LLD) and link it to requirements and user stories.Certifications: * Obtain relevant technology certifications to enhance skills and knowledge.Skill Examples: * Proficiency in SQL Python or other programming languages utilized for data manipulation.

Experience with ETL tools such as Apache Airflow Talend Informatica AWS Glue Dataproc and Azure ADF.
Hands-on experience with cloud platforms like AWS Azure or Google Cloud particularly with data-related services (e.g. AWS Glue BigQuery).
Conduct tests on data pipelines and evaluate results against data quality and performance specifications.
Experience in performance tuning data processes.
Proficiency in querying data warehouses.

Knowledge Examples:Knowledge Examples * Knowledge of various ETL services provided by cloud providers including Apache PySpark AWS Glue GCP DataProc/DataFlow and Azure ADF/ADLF.

Understanding of data warehousing principles and practices.
Proficiency in SQL for analytics including windowing functions.
Familiarity with data schemas and models.
Understanding of domain-related data and its implications.

Additional Comments:Mandatory Skills: SQL, CDP (TreasureData), Python/Dig-Dag and Presto/Sql, data engineering. Skill to Evaluate: SQL,Phyton,Dig-Dag,Treasure-Data Experience: 4 to 6 Years Job Description: SQL, CDP (TreasureData), Python/Dig-Dag and Presto/Sql, data engineering. Knowledge & experience in cloud technologies - Microsoft Azure, AWS, etc., ETL processes and API integration tools. Working experience in Python & SQL Exposure to Big Data technologies preferrable (such as Presto, Hadoop, Cassandra, MongoDB, etc.) CDP implementation experience with Treasure Data or any other CDP tool like Action IQ, etc. will be a big plus. Experience in data modelling & architecture preferable. Excellent SQL & advanced Sql skills. Knowledge or experience of data visualization (e.g., Power BI, etc.), Understanding or experience in AI/ML - would be a plus. Education Qualificaiton: BE/BTech Job Title: Customer Data Platform Specialist (Treasure Data) Roles & Responsibilities: Involved in requirements gathering, with ability to create technical documentation and having strong analysis and problem-solving skills. Experience in Data engineering, end-to-end implementation of the CDP projects. Involved in CDP BAU, Go-live cut-over & day-to-day CDP application support. Work on automation of existing tasks. Expected to be flexible with working hours and capable of working in a process-oriented environment.Skills:SQL,CDP,Python Or Dig-Dag,Presto or Sql, Data engineeringAbout Company:UST is a global digital transformation solutions provider. For more than 20 years, UST has worked side by side with the world’s best companies to make a real impact through transformation. Powered by technology, inspired by people and led by purpose, UST partners with their clients from design to operation. With deep domain expertise and a future-proof philosophy, UST embeds innovation and agility into their clients’ organizations. With over 30,000 employees in 30 countries, UST builds for boundless impact—touching billions of lives in the process.

UST

Apply Now