Senior Data Engineer - Python/SQL (5-8 yrs)

Requirements :

- BE / BTECH | MTech | MCA.

- Minimum 3+ years of work experience in development / Migration projects.

- Experience working with distributed technology tools for developing Batch and streaming pipelines using SQL, Spark, Python, Airflow Scala.

- Experience in Cloud Computing, e. g., AWS, GCP, Azure, etc.

- Able to quickly pick up new programming languages, technologies, and frameworks.

- Strong skills in building positive relationships across Product and Engineering.

- Able to influence and communicate effectively, both verbally and in writing, with team members and business stakeholders.

- Experience with creating/ configuring Jenkins pipeline for smooth CI/CD process for Managed Spark jobs, building Docker images, etc.

- Working knowledge of Data warehousing, Data modelling, Governance, and Data Architecture.

Good to have :

- Experience working with Data platforms, including EMR, Airflow, Data bricks (Data Engineering and Delta Lake components, and Lakehouse Medallion architecture), etc.

- Experience working in Agile and Scrum development processes.

- Experience in EMR/ EC2 Data bricks, etc.

- Experience working with Data warehousing tools, including SQL database, Presto, and Snowflake.

- Experience architecting data products in Streaming, Serverless less and Microservices Architecture and platform.

Primary Responsibilities :

- Understand Caredata architecture/domain and start contributing for new/existing business requests

- Experience in Data Integration and Data Warehousing, Cloud

- Develop efficient and high performing ETL solutions

- Design, develop, test and performance tune complex ETLs

- Develops and maintains scalable data pipelines in response to customer requirements

- Performs data analysis required to troubleshoot data related issues and assist in the resolution of data issues

- Design, develop, implement, and run data solutions that improve data efficiency, reliability, and quality

- Participate in design review sessions and peer ETL reviews

- Assess and profile source and target data (data structure, quality, completeness, schema, nulls, etc.) and requested business use cases

- Summarizes testing and validation results and can communicate and make recommendations/decisions on the best course of action to remediate

- Resourceful at coming up with solutions using existing or available resources based on knowledge of the organization and level of execution effort

- Participate in agile work environment, attend daily scrums, and complete sprint deliverables on time

- Supports practices, policies and operating procedures and ensures alignment to departmental objectives and strategy

- Ensure the code is meeting the desired quality checks using Sonar

- To make sure all the cloud infra is intact and resolve any issues encountered

- Schedule the deployed pipelines using Airflow following proper dependency hierarchy of jobs

- Move the code/application to higher environments(stage/prod) from non-prod for go-live

- Build and maintain pipelines and automation through Git Ops (GitHub Actions, Jenkins, etc)

- Identifies solutions to non-standard complex requests and problems and creates solution using available technologies

- Builds solid relationship with IT operational leaders to ensure connectivity to the business

- Supports a work environment in which people can perform to the best of their abilities.

- Holds self-accountable for technical abilities, productive results, and leadership attributes

Comply with the terms and conditions of the employment contract, company policies and procedures, and any and all directives (such as, but not limited to, transfer and/or re-assignment to different work locations, change in teams and/or work shifts, policies in regards to flexibility of work benefits and/or work environment, alternative work arrangements, and other decisions that may arise due to the changing business environment). The Company may adopt, vary or rescind these policies and directives in its absolute discretion and without any limitation (implied or otherwise) on its ability to do so