Databricks Engineer - Apache Spark (4-8 yrs)

Databricks Consultant

We are seeking a skilled Databricks Consultant with hands-on experience in AWS for our client, a leading consulting firm in India.

The ideal candidate will be responsible for designing, developing, and optimizing big data solutions using Databricks, leveraging cloud environments for scalable and efficient data processing.

The role requires a strong understanding of cloud platforms, data engineering, and big data technologies.

To apply for this position, it is mandatory to register on our platform at www.curataid.com and give 10 minutes technical quiz on AWS/Azure Data Engineering skill.

Responsibilities

Key Responsibilities:

- Build scalable data pipelines using AWS services and Databricks.

- Collaborate with cross-functional teams to gather requirements and deliver robust ETL/ELT workflows.

- Develop and maintain large-scale data processing jobs using Apache Spark and Python.

- Optimize data pipelines for performance, cost-efficiency, and reliability.

- Design and manage data lakes, data warehouses, and structured/unstructured data integrations.

- Implement and enforce data governance, quality, and security best practices.

- Ensure compliance with industry regulations (e.g., GxP, HIPAA) related to data handling and privacy.

Qualifications

- 4 to 8 years of experience with Databricks for data engineering and analytics.

- Hands-on experience with AWS services such as S3, Lambda, Glue, Redshift, and EMR.

- Proven track record of building and optimizing workflows in Databricks.

- Expertise in Apache Spark and advanced SQL for processing large datasets.

- Strong knowledge of ETL/ELT design and implementation across structured and unstructured data sources.

- Proficiency in Python (or Scala) for developing data engineering solutions and automating workflows.

- Experience with data modeling, data lakes, warehouse design, and governance practices in regulated environments.

- Familiarity with data privacy and compliance frameworks such as GxP, HIPAA, or other relevant regulations.

Unfortunately, due to high volume of applicants, not everybody will receive a response.

About Us

CuratAId is a hiring platform for the tech industry in India, providing pre-vetted candidates to recruiters.

Our expert interviewers screen candidates based on their primary and secondary technical skills, as well as communication and behavioral traits, to provide recruiters with high-quality, qualified candidates.

As a tech hiring platform, our goal is to simplify and streamline the recruitment process for both recruiters and job seekers.