Senior Data Engineer - Python/SQL (6-10 yrs)

Job Description :

We are seeking a highly skilled and motivated Senior Data Engineer to join our clients dynamic team. In this role, you will play a critical part in designing, implementing, and optimizing robust data architectures and pipelines that power scalable and high-performance data solutions across the organization. Your expertise will directly support data-driven decision-making and strategic initiatives.

Qualifications :

- 5-8 years of experience in data engineering, with a focus on building and managing data pipelines.

- Strong proficiency in relational databases (e.g., MySQL, PostgreSQL) and NoSQL databases (e.g., MongoDB, Cassandra).

- Experience in building data pipelines with data warehouses like Snowflake, Redshift

- Experience in processing unstructured data stored from S3 using Athena, Glue etc.

- Hands-on experience with Kafka for real-time data streaming and messaging.

- Solid understanding of ETL processes, data integration, and data pipeline optimization.

- Proficiency in programming languages like Python, Java, or Scala for data processing.

- Experience with Apache Spark for big data processing and analytics is an advantage

- Familiarity with cloud platforms like AWS, GCP, or Azure for data infrastructure is a plus.

- Excellent problem-solving skills and attention to detail.

- Strong communication and collaboration skills, with the ability to work effectively in a team environment.

Key Responsibilities :

- Design, build, and maintain efficient and scalable data pipelines to support data integration and transformation across various sources.

- Work with relational databases (e.g., MySQL, PostgreSQL, etc.) and NoSQL databases (e.g., MongoDB, Cassandra, etc.) to manage and optimize large datasets.

- Utilize Apache Spark for distributed data processing and real-time analytics.

- Implement and manage Kafka for data streaming and real-time data integration between systems.Collaborate with cross-functional teams to gather and translate business requirements into technical solutions.

- Monitor and optimize the performance of data pipelines and architectures, ensuring high availability and reliability.

- Ensure data quality, consistency, and integrity across all systems.

- Stay up-to-date with the latest trends and best practices in data engineering and big data technologies.