
Software Engineer
- Telangana
- Permanent
- Full-time
Function/Department: Databricks, SQL
Location: Hyderabad / Bangalore
Employment Type: Full Time
Reports To: Vishal KidderKey Responsibilities: * Data Pipeline Development:
- Design, develop, and optimize scalable ETL/ELT pipelines using Databricks and SQL to process large volumes of structured and unstructured data.
- Implement data ingestion, transformation, and integration workflows from various data sources.
- Data Modeling and Architecture:
- Design and implement efficient data models to support analytics and reporting needs.
- Collaborate with data architects to ensure data solutions align with business requirements and architectural standards.
- Databricks Platform Management:
- Leverage Databricks for data processing, machine learning workflows, and advanced analytics.
- Optimize Databricks clusters for performance and cost efficiency.
- SQL Development:
- Write complex SQL queries for data extraction, transformation, and analysis.
- Optimize SQL queries for performance and scalability.
- Collaboration:
- Work closely with data analysts, data scientists, and business stakeholders to understand data requirements and deliver solutions.
- Partner with DevOps and IT teams to ensure data infrastructure reliability and security.
- Data Quality and Governance:
- Implement data quality checks and monitoring to ensure data accuracy and consistency.
- Adhere to data governance policies and ensure compliance with data security standards.
- Performance Optimization:
- Monitor and improve the performance of data pipelines and workflows.
- Troubleshoot and resolve data-related issues in a timely manner.
- Technical Expertise:
- Proficiency in Databricks, including Spark-based data processing and Databricks notebooks.
- Strong SQL skills, including query optimization and database design.
- Experience with cloud platforms (e.g., Azure, AWS, or GCP) and their data services.
- Familiarity with data lake and data warehouse concepts.
- Programming Skills:
- Proficiency in Python, Scala, or another programming language commonly used in data engineering.
- Experience with version control systems (e.g., Git).
- Data Engineering Knowledge:
- Strong understanding of ETL/ELT processes and data integration techniques.
- Knowledge of data modeling (e.g., star schema, snowflake schema) and data architecture principles.
- Soft Skills:
- Strong problem-solving and analytical skills.
- Excellent communication and collaboration abilities.
- Ability to work in a fast-paced, dynamic environment.
- Experience with other big data tools and frameworks (e.g., Hadoop, Kafka, Airflow).
- Knowledge of machine learning workflows and integration with Databricks.
- Familiarity with BI tools (e.g., Power BI, Tableau) for data visualization.
- Certifications in Databricks, cloud platforms, or related technologies.
- Bachelor's or Master's degree in Computer Science, Data Engineering, Information Systems, or a related field.
- 3+ years of experience in data engineering or a related role.