Senior Data Engineer

High-Level Skill Set Requirements

Must-Have Skills

1. Programming Languages

• Proficiency in Python for automating processes and supporting services.

Shell scripting for task automation and system-level troubleshooting.

1. ETL Tools

• Experience with ETL tools like Informatica Cloud.

Ability to manage and optimize data extraction, transformation, and loading (ETL) processes.

1. SQL and Data Querying

• Strong SQL skills for querying data, creating custom checks, and monitoring services.

• Experience managing and monitoring data pipelines in various environments (upstream, batch, streaming).

Familiarity with pipeline orchestration tools like Luigi or Airflow.

1. Operational Support and Troubleshooting

• Experience providing operational support, including troubleshooting and resolving incidents.

• Knowledge of cloud platforms like AWS, GCP, or Azure.

Experience with OpenShift and Kubernetes environments.

1. Monitoring and Automation Tools

• Proficiency in using Prometheus, Grafana, or similar tools for monitoring service health.

Ability to set up monitoring dashboards and perform anomaly detection.

1. Security and Access Management

• Knowledge of user access management, including account setup and permissions validation.

• Ability to understand and document data flows and dependencies between different data services.

Familiarity with DevOps principles and Agile environments.

1. Collaboration and Communication

• Proficiency in using Google Docs and Jira for documentation and incident tracking.

Strong communication skills for working with cross-functional teams.

1. Incident Management and Optimization

• Experience in incident management and root cause analysis. Ability to identify and fix common failure points.

• Knowledge of Apache Kafka and experience with streaming services.

• Awareness of data governance principles for maintaining data quality and compliance.