Key ResponsibilitiesDesign and implement big data pipelines for ingestion, transformation, and processing of structured and unstructured data.Develop scalable data solutions using technologies like Hadoop, Spark, Kafka, Hive, and HBase.Write efficient and optimized MapReduce, Spark (PySpark/Scala/Java) jobs.Build and maintain ETL processes and data workflows using big data tools.Integrate big data solutions with cloud platforms such as AWS, Azure, or Google Cloud Platform (GCP).Ensure the security, quality, and integrity of data throughout the pipeline.Optimize performance of data processing and storage solutions.Monitor and troubleshoot big data infrastructure and workflows.Work closely with data scientists, analysts, and DevOps teams to support analytical and operational use cases.Stay updated with the latest big data trends and technologies.
