AWS Glue Developer - ETL/Data Warehousing (3-5 yrs)

Job Description :

We are hiring a skilled AWS Glue Developer to join our team for an exciting opportunity with a leading multinational company. The ideal candidate will have a strong background in Apache PySpark, AWS Stack, and Dimensional Modeling. As an AWS Glue Developer, you will work on building scalable data pipelines, developing ETL processes, and integrating various data sources for business intelligence solutions.

The role requires a strong technical background, experience with AWS services, and a solid understanding of DevOps, DataOps, and CloudOps principles, along with CI/CD and SRE practices.

Key Responsibilities :

- Design, develop, and implement ETL pipelines using AWS Glue, Apache PySpark, and SparkSQL.

- Work with various AWS services such as S3, Redshift, Glue, Athena, Lambda, and others to design end-to-end data workflows.

- Develop, test, and optimize data pipelines for real-time and batch processing.

- Transform raw data into a usable format, adhering to best practices in data engineering and processing.

- Collaborate with data architects to create and implement dimensional modeling for structured and unstructured data.

- Build and manage data lakes and data warehouse solutions, utilizing Lakehouse architecture principles for scalable and efficient storage.

- Design and implement data integration strategies that align with business intelligence and analytics needs.

- Work with the AWS cloud platform to create scalable and cost-effective data solutions.

- Utilize AWS Glue for data transformation, loading, and extraction tasks.

- Design and implement cloud-native solutions and services that optimize data workflows on AWS.

- Ensure high availability, fault tolerance, and scalability of data services.

- Integrate data pipelines into CI/CD pipelines for automated deployments using tools like Jenkins, GitLab, or AWS CodePipeline.

- Collaborate with DevOps and CloudOps teams to implement infrastructure-as-code (IAC) and automating the deployment process.

- Ensure the application of DataOps practices, managing the lifecycle of data pipelines in a version-controlled environment.

- Develop microservices and APIs to interact with data systems, ensuring scalability and maintainability.

- Write clean, efficient, and reusable code in Python and other programming languages as required.

- Ensure adherence to coding standards, security practices, and version control with Git.

- Work closely with business analysts, data scientists, and other stakeholders to translate business requirements into scalable data systems.

- Provide detailed documentation on architecture, processes, and solutions.

- Communicate complex technical concepts to non-technical stakeholders and ensure alignment with business objectives.

- Identify bottlenecks in data pipelines and work on performance optimization.

- Troubleshoot and debug data-related issues, ensuring the smooth operation of data flows.

Required Skills & Qualifications :

- 3-5 years of experience in data engineering, with expertise in AWS Glue, Apache PySpark, SparkSQL, and ETL development.

- Strong experience in AWS Cloud services, including but not limited to AWS Glue, S3, Redshift, Lambda, and Athena.

- Hands-on experience with dimensional modeling for data warehouses and lakes.

- Familiarity with DevOps principles, including CI/CD, DataOps, and CloudOps practices.

- Strong understanding of Lakehouse architecture and Data Warehousing concepts.

- Proficiency in version control systems (specifically Git).

- Solid experience in API development and microservices.

- Strong Python programming skills, with knowledge of Spark and PySpark frameworks.

- Excellent analytical and problem-solving skills with a keen eye for optimizing performance.

- Experience with data governance and security best practices in cloud environments.

- Ability to translate business needs into technical solutions and collaborate effectively with cross-functional teams.