Senior Data Engineer with Python (IR-491)
Intellectsoft
- Gurgaon, Haryana
- Permanent
- Full-time
- 6+ years of professional experience, including 2+ years of data engineering with Apache Spark and SQL.
- Proficiency in Python for data processing and automation.
- Knowledge of PySpark, distributed computing, analytical databases and other big data
- technologies.
- Expertise in designing and managing ETL pipelines and distributed data processing frameworks.
- Strong knowledge of database systems, data modeling, and analytical databases.
- Hands-on experience with workflow orchestration tools such as Apache Airflow.
- Familiarity with cloud platforms like AWS, GCP, or Azure.
- Solid understanding of software development lifecycles, including coding standards, version control, and testing.
- Bachelor’s or master’s degree in Computer Science or a related field.
- Familiarity with the data science and machine learning development process.
- Understanding of Machine Learning pipelines or frameworks.
- Design and build highly reliable and scalable data pipelines using PySpark and big data technologies.
- Collaborate with the data science team to develop new features that enhance model accuracy and performance.
- Create standardized data models to improve consistency across various deployments.
- Troubleshoot and resolve issues in existing ETL pipelines and optimize workflows.
- Conduct POCs to evaluate new technologies and integrate additional data sources.
- Follow and promote best practices for software development, ensuring high-quality solutions that meet requirements and deadlines.
- Document development updates and maintain clear technical documentation.
- Employment-based cooperation
- Awesome projects with an impact
- Comprehensive insurance for you and your family (health, life, and accident)
- Paid PTO policy (vacation, sick leaves, and public holidays)
- Tech equipment provided
- Udemy courses, workshops, trainings & expert knowledge-sharing
- Flexible hours & work setup