Databricks Developer

Mumbai, Maharashtra
Permanent
Full-time

15 days ago

Seaspan teams are goal-driven and share a high-performance culture, focusing on building services offerings to become a leading asset manager. Seaspan provides many of the world's major shipping lines with alternatives to vessel ownership by offering long-term leases on large, modern containerships and pure car, truck carriers (PCTCs) combined with industry leading ship management serves. Seaspan's fleet has evolved over time to meet the varying needs of our customer base. We own vessels in a wide range of sizes, from 2,500 TEU to 24,000 TEU vessels. As a wholly owned subsidiary of Atlas Corp, Seaspan delivers on the company's core strategy as a leading asset management and core infrastructure company.Position Description:We are seeking a skilled Databricks Developer to join our Data Development team. Reporting to the Team Lead, Data Development, the Databricks Developer implements robust data pipelines using Apache Spark on Databricks, supports advanced data transformation, and enables scalable data products that serve enterprise analytics and reporting needs. You'll work closely with data engineers and analysts to ensure high-performance, reliable data pipelines and quality outputs.This is a hands-on development role focused on engineering scalable, maintainable, and optimized data flows in a modern cloud-based environment.Job Responsibilities:

Design, build, and maintain scalable data pipelines and workflows using Databricks (SQL, PySpark, Delta Lake).
Develop efficient ETL/ELT pipelines for structured and semi-structured data using Azure Data Factory (ADF) and Databricks notebooks/jobs.
Integrate and transform large-scale datasets from multiple sources into unified, analytics-ready outputs.
Optimize Spark jobs and manage Delta Lake performance using techniques such as partitioning, Z-ordering, broadcast joins, and caching.
Design and implement data ingestion pipelines for RESTful APIs, transforming JSON responses into Spark tables.
Apply best practices in data modeling and data warehousing concepts.
Perform data validation and quality checks.
Work with various data formats, including JSON, Parquet, and Avro.
Build and manage data orchestration pipelines, including linked services and datasets for ADLS, Databricks, and SQL Server.
Create parameterized and dynamic ADF pipelines, and trigger Databricks notebooks from ADF.
Collaborate closely with Data Scientists, Data Analysts, Business Analysts, and Data Architects to deliver trusted, high-quality datasets.
Contribute to data governance, metadata documentation, and ensure adherence to data quality standards.
Use version control tools (e.g., Git) and CI/CD pipelines to manage code deployment and workflow changes.
Develop real-time and batch processing pipelines for streaming data sources such as MQTT, Kafka, and Event Hub.

Requirements:

5+ years of experience in data engineering or big data development.
Bachelor's degree in computer science or a relevant field, or equivalent training and work experience.
Strong hands-on experience with Databricks and Apache Spark (PySpark/SQL).
Proven experience with Azure Data Factory, Azure Data Lake, and related Azure services.
Experience integrating with APIs using libraries such as requests and http.
Deep understanding of Delta Lake architecture, including performance tuning and advanced features.
Proficiency in SQL and Python for data processing, transformation, and validation.
Familiarity with data lakehouse architecture and both real-time and batch processing design patterns.
Comfortable working with Git, DevOps pipelines, and Agile delivery methodologies.

Additional Desired Qualifications:

Experience with dbt, Azure Synapse, or Microsoft Fabric.
Familiarity with Unity Catalog features in Databricks.
Relevant certifications such as Azure Data Engineer, Databricks, or similar.
Understanding of predictive modeling, anomaly detection, or machine learning, particularly with IoT datasets.

Job Demands and/or Physical Requirements:

As Seaspan is a global company, occasional work outside of regular office hours may be required.

Compensation and Benefits Package:Seaspan's total compensation is based on our pay-for-performance philosophy that rewards team members who deliver on and demonstrate our high-performance culture.The base salary offered will be commensurate with the incumbent's experience, job-related skills and knowledge, and internal pay equity.Seaspan Corporation is an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to age, race, color, religion, gender, sexual orientation, gender identity, national origin, disability, or protected Veteran status. We thank all applicants in advance. If your application is shortlisted to be included in the interview process, one of our team will be in contact with you.Please note that while this position is open in both Vancouver and Mumbai, it represents a single headcount. The role will be filled in one of the two locations based on candidate availability and suitability, determined by the hiring team.A WEEK AT SEA WITH SEASPANThis video was taken on board our 4250 TEU vessel, Seaspan Santos, during a six-day passage in September 2013.JOIN THE BEST AT SEAWhen you join the Seaspan family, you become a part of a company with one of the newest and most advanced fleets in the industry. Our modern vessels are maintained to the highest standards and offer excellent living conditions for our seafarers.

Seaspan

Apply Now