Big Data Engineer - Hadoop/Spark (5-8 yrs)

Job Summary :

We are seeking a skilled and passionate Big Data Engineer to join our dynamic team in Faridabad. You will play a crucial role in designing, developing, and maintaining our large-scale data processing pipelines and infrastructure. The ideal candidate will possess deep hands-on experience with a wide range of Big Data technologies and a strong understanding of data warehousing principles.

You will leverage your expertise in tools like Hadoop, Spark, Hive, and Impala, along with programming proficiency in Scala, Java, and Python, to build robust and efficient data solutions. Your ability to integrate relational databases like Oracle and Postgres into our Big Data ecosystem and your knowledge of stream and batch processing frameworks such as Kafka and Flink will be essential to your success in this role.

While cloud experience is a plus, your core technical expertise and drive to build innovative data solutions are what we value most.

Key Responsibilities :

- Data Pipeline Development : Design, develop, and implement scalable and robust data processing pipelines using technologies such as Spark, Hadoop (MapReduce, HDFS), Hive, and Impala.

- Programming and Scripting : Utilize strong programming skills in Scala, Java, and Python to develop efficient data processing logic and automation scripts.

- Data Integration : Integrate data from various sources, including relational databases (Oracle, Postgres) and other structured and unstructured data sources, into our Big Data platform.

- Data Warehousing : Apply data warehousing principles and techniques to design and build efficient data models for both batch and real-time data processing.

- Stream and Batch Processing : Implement and manage stream processing solutions using Kafka and batch processing frameworks using Spark and Hadoop. Experience with Flink is highly desirable.

- Performance Optimization : Optimize data processing pipelines for performance, scalability, and reliability.

- Data Quality and Governance : Implement data quality checks and contribute to data governance initiatives.

- Collaboration : Work closely with data scientists, analysts, and other engineers to understand data requirements and deliver effective data solutions.

- Troubleshooting and Support : Identify, troubleshoot, and resolve issues within the Big Data environment.

- Documentation : Create and maintain clear and concise technical documentation for data pipelines, architectures, and processes.

- Continuous Learning : Stay up-to-date with the latest Big Data technologies, trends, and best practices.

- Cloud Exploration (Optional) : Explore and potentially contribute to the adoption of cloud platforms and cloud-native data services (AWS, Azure, GCP) for enhanced scalability and cost-effectiveness.

Requirements :

- Proven experience as a Big Data Engineer with a strong understanding of the Big Data ecosystem.

- Extensive hands-on experience with Hadoop, Spark, Hive, and Impala.

- Strong proficiency in programming languages such as Scala, Java, and Python.

- Deep understanding of relational databases like Oracle and Postgres and experience with data integration.

- Solid knowledge of data warehousing concepts and techniques.

- Experience with stream processing frameworks (Kafka) and batch processing frameworks (Spark, Hadoop).

- Excellent problem-solving and analytical skills.

- Ability to work collaboratively in a fast-paced, team-oriented environment.

- Strong communication skills, both written and verbal.

Preferred Skills (Not Mandatory) :

- Experience with cloud platforms (AWS, Azure, GCP) and related Big Data services (e.g., EMR, Databricks, Azure HDInsight, Google Cloud Dataflow).

- Familiarity with containerization technologies (Docker, Kubernetes).

- Experience with data visualization tools.

- Knowledge of data governance and security best practices in Big Data environments.