
Senior Software Engineer
- Hyderabad, Telangana
- Permanent
- Full-time
- Design, develop, and maintain scalable and efficient data pipelines and ETL processes to extract, transform, and load data from various sources into our data lake or warehouse.
- Collaborate with cross-functional teams including data scientists, analysts, and software engineers to understand data requirements, define data models, and implement solutions that meet business needs.
- Ensure the security, integrity, and quality of data throughout the data lifecycle, implementing best practices for data governance, encryption, and access control.
- Develop and maintain data infrastructure components such as data warehouses, data lakes, and data processing frameworks, leveraging cloud services (e.g., AWS, Azure, GCP) and containerization technologies (e.g., Docker, Kubernetes).
- Implement monitoring, logging, and alerting mechanisms to ensure the reliability and availability of data pipelines and systems, and to proactively identify and address issues.
- Work closely with stakeholders to understand business requirements, prioritize tasks, and deliver solutions in a timely manner within an Agile working environment.
- Collaborate with the risk, security and compliance teams to ensure adherence to regulatory requirements (e.g., GDPR, PCI DSS) and industry standards related to data privacy and security.
- Stay updated on emerging technologies, tools, and best practices in the field of data engineering, and propose innovative solutions to improve efficiency, performance, and scalability.
- Mentor and coach junior engineers, fostering a culture of continuous learning and professional development within the team.
- Participate in code reviews, design discussions, and other Agile ceremonies to promote collaboration, transparency, and continuous improvement.
Qualified to Degree, HND or HNC standard in a software engineering and/or data engineering discipline or can demonstrate commercial experienceRequired skills/ experience:
- Experience of the full development lifecycle
- Strong communication skills with the ability to explain solutions to technical and non-technical audiences
- Write clean, scalable and re-usable code that implements SOLID principles, common design patterns where applicable and adheres to published coding standards
- Excellent attention to detail, ability to analyse, investigate and compare large data sets when required.
- 3 or more years of programming using Scala
- 2 or more years of programming using Python
- Some experience of using Terraform to provision and deploy cloud services and components
- Experience of developing on Apache Spark
- Experience of developing with AWS cloud services including (but not limited to) AWS Glue, S3, Step Functions, Lambdas, EventBridge and SQS
- BDD / TDD experience
- Jenkins CI / CD experience
- Application Lifecycle Management Tools - BitBucket & Jira
- Performing Pull Request reviews
- Understanding of Agile methodologies
- Automated Testing Tools
- Mentoring or coaching junior engineers
- Cloud Solution Architecture
- Document databases
- Relational Databases
- Experience with Container technologies (e.g. Kubernetes)
- Java (rather than Scala)
- Google Cloud or Microsoft Azure (rather than AWS)
- Azure Pipelines or TeamCity (rather than Jenkins)
- Github (rather than BitBucket)
- Azure DevOps (rather than Jira)
- CloudFormation (rather than Terraform)