Senior Data Engineer - Python/SQL/ETL (4-8 yrs)

Various business units in the bank have data requirements, including regulatory, operational, and ad hoc reporting. Alongside this, the business excellence team needs data for machine learning and customer profiling scorecards to run on data engineering jobs. The purpose of this requirement is to fulfill all those data requirements on time and with accuracy.

Responsibilities :

- Design, build, and optimize data pipelines using AWS EMR (must have), S3 and Glue.

- Develop and maintain scalable ETL workflows for campaign, customer, deposit, and credit card domains.

- Write and maintain complex data transformation logic using PySpark (must have), SQL (must have and Python scripts for ETL, data quality, and automation tasks.

- Debug code issues efficiently and utilize Git (must have for versioning.

- Monitor, troubleshoot, and resolve issues in scheduled data jobs (root cause analysis, escalations, bug fixes) using Airflow.

- Automate failure notifications, job monitoring, and maintenance history tracking.

- Coordinate with the source and Data Lake teams for missing data or fixes.

- Manage password rotation and SSL upgrades on S3 buckets.

- Design and implement automated data quality checks and reporting solutions.

- Investigate and resolve data quality issues

- Collaborate with QA and production support teams for continuous process improvement.

- Use GoCD, Bitbucket (must have), and related DevOps tools for CI/CD and deployment automation.

- Participate in Agile/Scrum ceremonies, manage Jira tasks, and contribute to technical documentation.

- Work on end-to-end data lifecycle from Data Ingestion, Data Transformation, and Data Consumption layer. Versed in API and its usability.

- Suitable candidates will also be proficient in Spark, Spark Streaming, Hive, and SQL.

- Suitable candidates will also demonstrate experience with big data infrastructure, inclusive of MapReduce, Hive, HDFS, YARN, HBase, Oozie, etc.

Secondary Responsibilities :

- Ability to work independently and handle your development effort.

- Identify key performance indicators and create an educational/deliverables path to achieve the same.

- Use an educational background in data engineering and perform data mining analysis.

- Work with BI analysts/engineers to create prototypes.

- Engage in the delivery and presentation of solutions.

Requirements :

- Good understanding of Data Lake/Lakehouse architecture and experience/exposure to Hadoop(Cloudera, hortonworks) and/or AWS.

- Excellent oral and written communication skills.

- Learn and use internally available analytic technologies.

- 4-8 years in data engineering roles with exposure to project management.

- Proficiency in SQL, PySpark, data processing, and data integration.

- Experience with big data tools (MapReduce, Hive, HDFS, YARN, etc. ) and the AWS cloud platform.

- Strong problem-solving and debugging skills, with a proven track record of delivering projects on time.

- Demonstrated ability to lead cross-functional teams and manage stakeholder expectations.

- Excellent communication and interpersonal skills for effective team management and collaboration.

Technical Stack :

- AWS : EMR, EC2 S3 Glue, Athena.

- Programming : PySpark, SQL, Python.

- Version Control : Git.

- Data Integration : Kafka, API, flat files, vendor feeds.

- DevOps : GoCD, Bitbucket, Jira.

- Orchestrator : Apache Airflow.