BigTapp Analytics - Senior Data Engineer - Azure Databricks (5-7 yrs)

Job Title : Senior Data Engineer.

Location : Hyderabad, Chennai & Bangalore.

Experience : 5+ Years.

Job Summary :

We are looking for a highly skilled and experienced Senior Data Engineer to join our growing data engineering team.

The ideal candidate will have a strong background in building and optimizing data pipelines and data architecture, as well as experience with big data technologies and Azure cloud services.

You will work closely with cross-functional teams to ensure data is accessible, reliable, and ready for analytics and business insights.

Mandatory Skills :

- Advanced SQL.

- Python or Scala for data engineering.

- ETL pipeline development.

- Cloud platforms (AWS/GCP/Azure).

- Azure 1st party services (ADF, Azure Databricks, Synapse, etc.

- Big data tools (Spark, Hadoop).

- Data warehousing (Redshift, Snowflake, Big Query).

- Workflow orchestration tools (Airflow, Prefect, or similar).

Key Responsibilities :

- Design, develop, and maintain scalable and reliable data pipelines.

- Demonstrate experience and leadership across two full project cycles using Azure Data Factory, Azure Databricks, and PySpark.

- Collaborate with data analysts, scientists, and software engineers to understand data needs.

- Implement data quality checks and monitoring systems.

- Optimize data delivery and processing across a wide range of sources and formats.

- Ensure security and governance policies are followed in all data handling processes.

- Evaluate and recommend tools and technologies to improve data engineering capabilities.

- Lead and mentor junior data engineers as needed.

- Work with cross-functional teams in a dynamic and fast-paced environment.

Qualifications :

- Bachelor's or Master's degree in Computer Science, Engineering, or related field.

- Certification in cloud platforms or big data technologies (preferred).

- 5+ years of data engineering experience.

Technical Skills :

- Programming : Python, Scala, SQL.

- Big Data : Spark, Hadoop, Hive.

- Stream Processing : Storm, Spark Streaming.

- Data Warehousing : Snowflake, Big Query, Redshift.

- Cloud : AWS (S3, Lambda, Glue), GCP, Azure (ADF, Azure Databricks).

- Orchestration : Apache Airflow, Prefect, Luigi.

- Databases : PostgreSQL, MySQL, NoSQL (MongoDB, Cassandra).

- Tools : Git, Docker, Kubernetes (basic), CI/CD.

Soft Skills :

- Strong problem-solving and analytical thinking.

- Excellent verbal and written communication.

- Ability to manage multiple tasks and deadlines.

- Collaborative mindset with a proactive attitude.

- Strong analytic skills related to working with unstructured datasets.

Good to Have.

- Experience with real-time data processing (Kafka, Flink).

- Knowledge of data governance and privacy regulations (GDPR, HIPAA).

- Familiarity with ML model data pipeline integration.

Work Experience :

- Minimum 5 years of relevant experience in data engineering roles.

- Experience with Azure 1st party services across at least two full project lifecycles.

Compensation & Benefits :

- Competitive salary and annual performance-based bonuses.

- Comprehensive health and optional Parental insurance.

- Retirement savings plans and tax savings plans.

- Work-Life Balance : Flexible work hours.

Key Result Areas (KRAs) :

- Timely development and delivery of high-quality data pipelines.

- Implementation of scalable data architectures.

- Collaboration with cross-functional teams for data initiatives.

- Compliance with data security and governance standards.

Key Performance Indicators (KPIs) :

- Uptime and performance of data pipelines.

- Reduction in data processing time.

- Number of critical bugs post-deployment.

- Stakeholder satisfaction scores.

- Successful data integrations and migrations.