Lead Data Engineer - Spark/Hadoop (7-9 yrs)

Location : Chennai.

Shift : 11AM 8 PM or 1PM 10PM.

Skills : Python, Big-Data, spark/pyspark, databricks, ADF or ADL, team management experience.

Role Description :

- As a Technical Lead Big Data Platform, you will be responsible for designing and implementing big data platforms to support large-scale data integration projects.

- You will work with data architects and data engineers to define the platform architecture and build the necessary infrastructure.

- You will be skilled in big data technologies such as Hadoop, Spark, and Kafka and have experience in cloud computing platforms such as AWS or Azure.

- You will be responsible for ensuring the performance, scalability, and security of the big data platform and troubleshooting any issues that arise.

Roles & Responsibilities :

- Designing, developing and maintaining large-scale big data platforms using technologies like Hadoop, Spark and Kafka.

- Creating and managing data warehouses, data lakes and data marts.

- Implementing and optimizing ETL processes and data pipelines.

- Developing and maintaining security and access controls.

- Troubleshooting and resolving big data platform issues.

- Collaborating with other teams to ensure the consistency and integrity of data.

Technical Skills :

Skills Requirements :

- Experience with big data processing technologies such as Apache Hadoop, Apache Spark, or Apache Kafka.

- Understanding of distributed computing concepts such as MapReduce, Spark RDDs, or Apache Flink data streams.

- Familiarity with big data storage solutions such as HDFS, Amazon S3, or Azure Data Lake Storage.

- Knowledge of big data processing frameworks such as Apache Hive, Apache Pig, or Apache Impala.

- Must have excellent communication skills and be able to communicate complex technical information to non-technical stakeholders in a clear and concise manner.

- Must understand the company's long-term vision and align with it.

- Should be open to new ideas and be willing to learn and develop new skills.

- Should also be able to work well under pressure and manage multiple tasks and priorities.

Qualifications :

Qualifications :

- 7-9 years of work experience in relevant field.Tech/B.