Data Engineer

Job Overview: We seek a highly skilled and motivated Data Engineer with strong expertise in Python, PySpark, AWS, Databricks, and Snowflake to join our dynamic team.

The ideal candidate should have hands-on experience with Spark optimization, SQL data processing, and AWS tools and technologies. Exposure to Informatica and data streaming tools is a plus.

Key Responsibilities:

• Design, develop, and optimize scalable data processing pipelines using Spark and PySpark.
• Implement Spark optimization techniques to enhance performance and efficiency.
• Work with complex SQL queries for data manipulation and transformation.
• Collaborate with data scientists, analysts, and engineering teams to meet data needs.
• Work on AWS Cloud services like Redshift, AWS Glue, SQL Server, and Databricks.
• Utilize Python for data processing tasks, ensuring modularization and packaging.
• Basic knowledge of Informatica.
• Engage in real-time data streaming and integration using tools such as NiFi, Kafka, and EventHub (optional but preferred).
• Hands-on experience with Snowflake for data warehousing and analytics.

Must Have Experience _ Python, Pyspark, SQL, AWS, Spark

Required Skills:

• Spark Expertise: Strong experience in Spark and PySpark, including optimization techniques.
• Complex SQL: Advanced SQL knowledge and data processing skills.
• Cloud Exposure: Familiarity with AWS services and Databricks.
• Python Proficiency: Experience with Python for data processing and scripting.
• Data Streaming: Experience with NiFi, Kafka, and EventHub is desirable.
• Snowflake: Hands-on experience with Snowflake for data warehousing and analytics.

Qualifications:

• Bachelors or Masters degree in Computer Science, Information Technology, or related fields.
• 4-8 years of professional experience in Data Engineering, Data Analytics, or related roles.