Associate Data Engineer - Python/ETL (3-7 yrs)

Role Overview :

We are seeking a highly motivated and skilled Associate Data Engineer with 3-7 years of experience to join our dynamic data team.

This role is crucial in building, maintaining, and optimizing robust, scalable, and efficient data pipelines and infrastructure.

Depending on experience and demonstrated capabilities, the role can evolve into a Lead or Tech Lead position, guiding junior engineers and shaping technical strategies.

The ideal candidate will have a strong foundation in data warehousing, ETL/ELT processes, cloud platforms, and big data technologies.

You will work closely with data scientists, analysts, and product teams to understand data requirements and deliver high-quality data solutions that support business objectives.

Key Responsibilities :

Data Pipeline Development & Management :

- Design, develop, and maintain scalable and reliable ETL/ELT pipelines using various data integration tools and programming languages (e.g., Python, Scala).

- Ingest, transform, and load data from diverse sources (e.g., relational databases, APIs, streaming data, flat files) into data warehouses or data lakes.

- Implement data orchestration workflows using tools like Apache Airflow, Azure Data Factory, or AWS Step Functions.

Data Modeling & Database Design :

- Collaborate with data architects and analysts to design and implement optimal data models (e.g., dimensional modeling, 3NF) for analytical and operational databases.

- Work with both SQL and NoSQL databases (e.g., PostgreSQL, MySQL, MongoDB, Cassandra) and data warehousing solutions (e.g., Snowflake, Amazon Redshift, Google BigQuery).

Performance Optimization & Scalability :

- Optimize existing data pipelines and queries for performance, cost-efficiency, and scalability.

- Identify and resolve data-related issues, ensuring data accuracy, consistency, and completeness.

- Implement best practices for data governance, security, and compliance.

Data Quality & Governance :

- Develop and implement data quality checks and monitoring mechanisms to ensure the integrity and reliability of data.

- Establish and enforce data governance policies and procedures.

Collaboration & Mentorship (for Senior/Lead/Tech Lead levels) :

- For Senior Developer (3-5 years) : Act as a subject matter expert in specific data domains, contribute to technical discussions, and mentor junior team members.

- For Lead/Tech Lead (5-7 years) :

- Provide technical leadership and guidance to a team of data engineers.

- Drive architectural decisions and define technical standards and best practices for data engineering.

- Lead code reviews, ensure code quality, and foster a culture of continuous improvement.

- Participate in strategic planning for data platform evolution and technology adoption.

- Act as a liaison between the data engineering team and other departments.

Troubleshooting & Support :

- Proactively monitor data systems and troubleshoot issues, ensuring high availability and reliability of data assets.

- Provide support for data consumers, resolving data access or usage problems.

Documentation :

- Create and maintain comprehensive documentation for data pipelines, data models, and data processes.

Required Skills & Qualifications :

- Education : Bachelor's or Master's degree in Computer Science, Engineering, Information Technology, or a related quantitative field.

- Experience : 3-7 years of professional experience in data engineering, ETL development, or a similar role.

- Programming : Strong proficiency in at least one programming language, preferably Python or Scala, for data manipulation and automation.

- Big Data Technologies : Hands-on experience with big data frameworks such as Apache Spark, Hadoop, Kafka, or Flink.

- Cloud Platforms : Demonstrated experience with at least one major cloud platform (AWS, Azure, or GCP) and their data services (e.g., AWS S3, Glue, EMR, Redshift; Azure Data Lake, Databricks, Synapse; GCP Cloud Storage, Dataflow, BigQuery).

- Databases : Solid understanding of relational databases (SQL) and strong SQL querying skills. Experience with NoSQL databases is a plus.

- Data Warehousing : Experience with data warehousing concepts and tools (e.g., Snowflake, Redshift, BigQuery, Teradata).

- Orchestration : Experience with workflow orchestration tools like Apache Airflow, Luigi, or equivalent.

- Version Control : Proficiency with Git and collaborative development workflows.

- Problem-Solving : Excellent analytical and problem-solving skills with a keen eye for detail.

- Communication : Strong verbal and written communication skills, with the ability to explain complex technical concepts to non-technical stakeholders.

- Teamwork : Ability to work effectively in a collaborative team environment and independently.

Preferred Qualifications :

- Experience with real-time data processing and streaming technologies.

- Familiarity with data governance tools and practices.

- Knowledge of machine learning pipelines and MLOps concepts.

- Cloud certifications (e.g., AWS Certified Data Analytics, Azure Data Engineer Associate, Google Professional Data Engineer).

- Experience with containerization technologies (Docker, Kubernetes).

- Prior experience in a Lead or Tech Lead role, mentoring junior engineers and driving technical initiatives