BigData and Hadoop Ecosystems - Chennai/Pune/Mumbai
- Mumbai, Maharashtra Pune, Maharashtra
- Permanent
- Full-time
Job Summary: We are seeking a skilled Big Data Engineer with strong expertise in Hadoop ecosystems to design, develop, and maintain big data solutions. The ideal candidate will work closely with data scientists, analysts, and business stakeholders to deliver scalable data pipelines and efficient data processing frameworks.
Key Responsibilities:
- Design and develop scalable big data solutions using Hadoop ecosystem tools.
- Build and maintain robust data pipelines for batch and real-time processing.
- Work with large datasets to extract meaningful insights and support analytics use cases.
- Optimize data storage and processing architectures for performance and cost-efficiency.
- Collaborate with cross-functional teams including data scientists, analysts, and DevOps.
- Ensure data security, privacy, and governance in data solutions.
- Perform data cleansing, transformation, and validation.
- Monitor and troubleshoot production big data environments.
- 3+ years of hands-on experience in Big Data technologies.
- Proficient in Hadoop ecosystem tools such as:
- HDFS, MapReduce, YARN
- Hive, Pig
- HBase
- Sqoop, Flume
- Apache Spark
- Oozie, Kafka, NiFi (a plus)
- Strong programming skills in Java, Scala, or Python.
- Experience with data pipeline tools like Apache Airflow or Oozie.
- Familiarity with cloud platforms (e.g., AWS EMR, Azure HDInsight, GCP Dataproc).
- Working knowledge of SQL and NoSQL databases.
- Knowledge of data modeling, data warehousing, and ETL frameworks.
- Experience with CI/CD pipelines and version control (e.g., Git).
- Bachelor's or Master's degree in Computer Science, Information Technology, or a related field.
- Experience with containerization (e.g., Docker, Kubernetes).
- Certifications in Big Data technologies or cloud platforms (e.g., Cloudera, AWS Big Data Specialty).
- Knowledge of data governance and security frameworks.