Lead Data Engineer (5-10 yrs)

Job Description :

Responsibilities :

- Lead the implementation of data engineering solutions using Databricks.

- Implement applied AI solutions using GenAI.

- Collaborate with data scientists, analysts, and other engineers to design and build data pipelines and models.

- Optimize data processing security and performance to ensure the scalability of the solutions.

- Implement test-first solutions. Pair with junior engineers to help them appreciate the values of test-first engineering.

- Practice clean code practices while implementing code. Pair with junior engineers to help them appreciate the value of clean code.

- Mentor and guide junior engineers, fostering a collaborative and innovative environment.

- Work closely with stakeholders to understand their data needs and translate them into technical solutions.

- Ensure data integrity, security, and compliance with industry standards.

Requirements :

- Bachelor's/Master's degree in Computer Science, Engineering, or a related field.

- 8+ years of experience in data engineering, with a focus on Databricks.

- Strong proficiency in Python/Java/Scala, and SQL

- Intermediate experience with Spark is highly desirable

- Exposure to DQ tools like Great Expectations would be a strong positive

- Hands-on experience with any of the cloud platforms, like AWS, Azure, or GCP, is a must.

- Proven track record of working on data engineering projects from conception to delivery.

- Strong analytical and problem-solving skills.

- Candidates with experience working on consulting engagements and experience interacting with business stakeholders would be preferred.

- Excellent communication and teamwork abilities.