Data Engineer - Hive/Spark (7-10 yrs)

Job Description :

Key Responsibilities :

- Design and implement scalable data pipelines to support big data transformations and analytics.

- Work with cloud platforms such as Azure (preferred), AWS, or Google Cloud to deploy and manage data systems.

- Utilize orchestration tools like Azure Data Factory and other cloud-native services for data pipeline automation.

- Write efficient, high-performance scripts using SQL, PySpark, Scala, and Python (Databricks experience preferred).

- Work with big data technologies like Hadoop, Spark, Hive, Snowflake, and Kafka to process and manage large datasets.

- Design and implement dimensional data models for data warehouses and data marts.

- Use monitoring and logging tools (Prometheus, Grafana, Splunk) to ensure system performance and troubleshoot issues.

- Collaborate with cross-functional teams using project management tools like JIRA and Azure DevOps.

- Develop reports and dashboards using Power BI or similar visualization tools to deliver actionable insights.

Qualifications :

- Bachelor's degree in Computer Science, Engineering, or a related field.

- 5-7 years of experience in data engineering, specializing in scalable data transformations and pipeline design.

- Strong proficiency with cloud platforms (Azure preferred), including experience with services like Azure Data Factory.

- Expertise in scripting languages such as SQL, Python, PySpark, and/or Scala; experience with Databricks is a plus.

- Experience with big data technologies including Hadoop, Spark, Hive, Snowflake, and Kafka.

- Strong understanding of dimensional modeling and experience designing data warehouses and data marts.

- Experience with monitoring/logging tools such as Prometheus, Grafana, or Splunk.

- Familiarity with project management tools like JIRA and Azure DevOps.

- Proficiency in Power BI or similar data visualization tools to create insightful reports and dashboards.

Must-Haves :

- Acknowledges the presence of choice in every moment and takes personal responsibility for their life.

- Possesses an entrepreneurial spirit and continuously innovates to achieve great results.

- Communicates with honesty and kindness, creating space for others to do the same.

- Leads with courage, knowing the possibility of greatness is bigger than the fear of failure.

- Fosters connection by putting people first and building trusting relationships.

- Integrates fun and joy as a way of being and working, i.e., doesn't take themselves too seriously.