Senior Data Engineer - Python/PySpark (7-12 yrs)

Job Summary :

We are seeking a highly skilled Senior Data Engineer to join our data engineering team and lead the design and development of robust, scalable, and high-performance data pipelines. You will play a key role in building end-to-end data solutions that drive critical business insights. This role demands expertise in Databricks, PySpark, Azure Data Factory, SQL, and modern cloud data platforms, along with experience in data modeling and governance.

As a Senior Data Engineer, you will work closely with data scientists, business analysts, and stakeholders to ensure the delivery of high-quality, reliable, and efficient data systems.

Key Responsibilities :

- Design, develop, and maintain large-scale, distributed data pipelines using Databricks, PySpark, and Python.

- Build reusable and modular ETL/ELT frameworks for structured and semi-structured data.

- Implement batch and real-time data ingestion and transformation logic.

- Deploy and manage data pipelines in Azure Databricks and optimize for performance, scalability, and cost-efficiency.

- Integrate data across hybrid systems (cloud/on-prem) using Azure Data Factory, Logic Apps, or other orchestration tools.

- Support and extend data platform capabilities in any leading cloud provider: Azure, AWS, or GCP.

- Perform conceptual, logical, and physical data modeling using tools such as Snowflake, MSBI, AWS Redshift, or Google BigQuery.

- Design data marts, warehouses, and lakes with well-structured schemas (star/snowflake models).

- Define and enforce data retention, archival, and purging strategies.

- Write, debug, and optimize complex SQL queries for analytical use cases.

- Implement data quality checks, validation logic, and monitoring dashboards to ensure consistency and accuracy.

- Ensure compliance with data governance, privacy regulations, and security best practices.

- Implement role-based access control (RBAC), encryption, and lineage tracking where necessary.

- Work closely with product teams, analysts, and business stakeholders to gather and refine requirements.

- Participate in architecture and design discussions, suggesting best practices and scalable solutions.

- Mentor junior engineers and promote knowledge sharing within the team.

Required Skills & Qualifications :

- Bachelors or Masters degree in Computer Science, Engineering, or related field.

- 6+ years of experience in data engineering or software development with a strong focus on data.

- Expert-level knowledge of Databricks, PySpark, and Python for data pipeline development.

- Proficient in SQL and database performance tuning.

- Experience with Azure Data Factory or equivalent orchestration tools.

- Strong understanding of cloud data platforms Azure (preferred), AWS, or GCP.

- Solid experience in data modeling using tools like Snowflake, MSBI, Redshift, or BigQuery.

- Knowledge of CI/CD, version control (Git), and infrastructure-as-code (Terraform, ARM templates) is a plus.