Lead Data Engineer(pyspark)

Tide Platform

  • Hyderabad, Telangana
  • Permanent
  • Full-time
  • 23 days ago
ABOUT TIDEAt Tide, we are building a business management platform designed to save small businesses time and money. We provide our members with business accounts and related banking services, but also a comprehensive set of connected administrative solutions from invoicing to accounting.Launched in 2017, Tide is now used by over 1 million small businesses across the world and is available to UK, Indian and German SMEs. Headquartered in central London, with offices in Sofia, Hyderabad, Delhi, Berlin and Belgrade, Tide employs over 2,000 employees.Tide is rapidly growing, expanding into new products and markets and always looking for passionate and driven people. Join us in our mission to empower small businesses and help them save time and money.ABOUT THE ROLEWe are seeking a highly skilled and experienced Senior Data Engineer with a deep expertise in PySpark to join our ML/Data engineering team. This team is responsible for feature development, data quality checks, deploying and integrating ML models with backend services and the overall Tide platform.
In this role, you will be instrumental in designing, developing, and optimizing our next-generation data pipelines and data platforms. You will work with large-scale datasets, solve complex data challenges, and contribute to building robust, scalable, and efficient data solutions that drive business value.
This is an exciting opportunity for someone passionate about big data technologies, performance optimization, and building resilient data infrastructure.As a Data Engineer you'll be:
  • Performance Optimization: Identify, diagnose, and resolve complex performance bottlenecks in PySpark jobs and Spark clusters, leveraging Spark UI, query plans, and advanced optimization techniques (e.g., partitioning, caching, broadcasting, AQE, UDF optimization).
  • Design & Development: Lead the design and implementation of highly scalable, fault-tolerant, and optimized ETL/ELT pipelines using PySpark for batch and potentially real-time data processing.
  • Data Modeling: Collaborate with data scientists, analysts, and product teams to understand data requirements and design efficient data models (e.g., star/snowflake schemas, SCDs) for analytical and operational use cases.
  • Data Quality & Governance: Implement robust data quality checks, monitoring, and alerting mechanisms to ensure the accuracy, consistency, and reliability of our data assets.
  • Architectural Contributions: Contribute to the overall data architecture strategy, evaluating new technologies and best practices to enhance our data platform's capabilities and efficiency.
  • Code Review & Best Practices: Promote and enforce engineering best practices, including code quality, testing, documentation, and version control (Git). Participate actively in code reviews.
  • Mentorship & Leadership: Mentor junior data engineers, share knowledge, and contribute to a culture of continuous learning and improvement within the team.
  • Collaboration: Work closely with cross-functional teams including software engineers, data scientists, product managers, and business stakeholders to deliver impactful data solutions.
WHAT ARE WE LOOKING FOR
  • 8+ years of professional experience in data engineering, with at least 4+ years specifically focused on PySpark development and optimization in a production environment.
  • Expert-level proficiency in PySpark including Spark SQL, DataFrames, RDDs, and understanding of Spark's architecture (Driver, Executors, Cluster Manager, DAG).
  • Strong hands-on experience with optimizing PySpark performance on large datasets, debugging slow jobs using Spark UI, and addressing common issues like data skew, shuffles, and memory management.
  • Excellent programming skills in Python with a focus on writing clean, efficient, and maintainable code.
  • Proficiency in SQL for complex data manipulation, aggregation, and querying.
  • Basic understanding of data warehousing concepts (dimensional modeling, ETL/ELT processes, data lakes, data marts).
  • Experience with distributed data storage solutions such as Delta Lake, Apache Parquet etc.
  • Familiarity with version control systems (Git).
  • Strong problem-solving abilities, analytical skills, and attention to detail.
  • Excellent communication and interpersonal skills, with the ability to explain complex technical concepts to both technical and non-technical audiences.
  • Bachelor's or Master's degree in Computer Science, Engineering, or a related quantitative field.
WHAT YOU WILL GET IN RETURNMake work, work for you! We are embracing new ways of working and support flexible working arrangements. With our Working Out of Office (WOO) policy our colleagues can work remotely from home or anywhere in their assigned Indian state. Additionally, you can work from a different country or Indian state for 90 days of the year. Plus, you'll get:
  • Competitive salary
  • Self & Family Health Insurance
  • Term & Life Insurance
  • OPD Benefits
  • Mental wellbeing through Plumm
  • Learning & Development Budget
  • WFH Setup allowance
  • 15 days of Privilege leaves
  • 12 days of Casual leaves
  • 12 days of Sick leaves
  • 3 paid days off for volunteering or L&D activities
  • Stock Options
TIDEAN WAYS OF WORKINGAt Tide, we champion a flexible workplace model that supports both in-person and remote work to cater to the specific needs of our different teams.While remote work is supported, we believe in the power of face-to-face interactions to foster team spirit and collaboration. Our offices are designed as hubs for innovation and team-building, where we encourage regular in-person gatherings to foster a strong sense of community.TIDE IS A PLACE FOR EVERYONEAt Tide, we believe that we can only succeed if we let our differences enrich our culture. Our Tideans come from a variety of backgrounds and experience levels. We consider everyone irrespective of their ethnicity, religion, sexual orientation, gender identity, family or parental status, national origin, veteran, neurodiversity or differently-abled status. We celebrate diversity in our workforce as a cornerstone of our success. Our commitment to a broad spectrum of ideas and backgrounds is what enables us to build products that resonate with our members' diverse needs and lives.We are One Team and foster a transparent and inclusive environment, where everyone's voice is heard.At Tide, we thrive on diversity, embracing various backgrounds and experiences. We welcome all individuals regardless of ethnicity, religion, sexual orientation, gender identity, or disability. Our inclusive culture is key to our success, helping us build products that meet our members' diverse needs. We are One Team, committed to transparency and ensuring everyone's voice is heard.You personal data will be processed by Tide for recruitment purposes and in accordance with .

Tide Platform