Big Data Technology Engineer - Apache Spark (4-8 yrs)

Job Title : Big Data Engineer Apache Spark Specialist

Location : Pune, India

Experience : 48 years (can be tailored)

Job Description :

We are seeking a passionate and experienced Big Data Engineer with a strong breadth of knowledge in Apache Spark and related big data technologies. As part of our dynamic data engineering team in Pune, you will play a key role in building scalable data pipelines, optimizing large-scale data processing, and supporting analytics-driven decisions.

Key Responsibilities :

- Design, develop, and maintain scalable data pipelines using Apache Spark (Core, SQL, Streaming).

- Process, clean, and transform large volumes of structured and unstructured data.

- Work closely with data scientists, analysts, and business teams to understand data needs.

- Optimize Spark jobs for performance and cost-efficiency in a distributed environment.

- Ensure data quality and implement monitoring and alerting for data pipelines.

- Collaborate in an Agile environment to deliver high-quality solutions iteratively.

Required Skills :

- Strong experience with Apache Spark (RDD, DataFrame, Datasets, Spark SQL, Streaming).

- Proficient in Scala, Java, or Python.

- Solid understanding of Hadoop ecosystem (HDFS, YARN, Hive, etc.).

- Experience with distributed systems, parallel computing, and data partitioning techniques.

- Familiarity with cloud platforms (AWS, Azure, or GCP) is a plus.

- Good grasp of data modeling and ETL best practices.

Preferred Qualifications :

- Bachelors or Masters degree in Computer Science, Engineering, or a related field.

- Experience with Kafka, Airflow, or NiFi.

- Exposure to CI/CD tools and practices in data engineering.

- Knowledge of Delta Lake, Iceberg, or Hudi is a plus.