Data Engineer - ETL (3-6 yrs)

Responsibilities :

• Design, build, and maintain robust, scalable, and efficient real-time data pipelines using Databricks, PySpark, Kafka, and Delta Lake.
• Architect and implement data ingestion processes for high-volume streaming and batch data into Amazon S3 and Delta Lake.
• Optimize data pipelines and workflows for performance, scalability, and cost-effectiveness.
• Process and analyze terabytes of structured and unstructured data to enable real-time decision-making.
• Work closely with stakeholders to define data requirements and ensure the integrity, security, and availability of the data.
• Implement advanced data transformations, deduplication, and enrichment logic.
• Continuously improve data engineering best practices and automation workflows.
• Monitor and troubleshoot data pipelines to ensure high availability and reliability.

• Proven experience working with Databricks and PySpark for large-scale data processing.
• In-depth knowledge of Delta Lake for real-time and batch data processing.
• Expertise in Apache Kafka for real-time data streaming.
• Hands-on experience with AWS S3 or equivalent cloud storage services.
• Strong understanding of distributed computing principles and performance optimization techniques.
• Experience handling terabytes of data in production environments.
• Proficiency in optimizing data pipelines for throughput, latency, and cost.
• Solid understanding of ETL/ELT processes and data modeling principles.
• Ability to write clean, efficient, and maintainable code in Python (PySpark).
• Familiarity with DevOps practices, CI/CD pipelines, and Kubernetes.
• Knowledge of data security and governance best practices.
• Background in monitoring and alerting systems such as Prometheus, Grafana, or CloudWatch.