Lead Data Engineer

Provides leadership for the overall architecture, design, development, and deployment of a full-stack cloud native data analytics platform.

Designing & Augmenting Solution architecture for Data Ingestion, Data Preparation, Data Transformation, Data Load, ML & Simulation Modelling, Java BE & FE, State Machine, API Management & Intelligence consumption using data products, on cloud

Understand Business Requirements and help in developing High level and Low-level Data Engineering and Data Processing Documentation for the cloud native architecture

Developing conceptual, logical and physical target-state architecture, engineering and operational specs.

Work with the customer, users, technical architects, and application designers to define the solution requirements and structure for the platform

Model and design the application data structure, storage, and integration

Lead the database analysis, design, and build effort

Work with the application architects and designers to design the integration solution

Ensure that the database designs fulfill the requirements, including data volume, frequency needs, and long-term data growth

Able to perform Data Engineering tasks using Spark

Knowledge of developing efficient frameworks for development and testing using (Sqoop/Nifi/Kafka/Spark/Streaming/ WebHDFS/Python) to enable seamless data ingestion processes on to the Hadoop/BigQuery platforms.

Enabling Data Governance and Data Discovery

Exposure of Job Monitoring framework along validations automation

Exposure of handling structured, Un Structured and Streaming data.

Experience with building data platform on cloud (Data Lake, Data Warehouse environment, Databricks)

Strong technical understanding of data modeling, design and architecture principles and techniques across master data, transaction data and derived/analytic data

Proven background of designing and implementing architectural solutions which solve strategic and tactical business needs

Deep knowledge of best practices through relevant experience across data-related disciplines and technologies, particularly for enterprise-wide data architectures, data management, data governance and data warehousing Highly competent with database design

Highly competent with data modeling

Strong Data Warehousing and Business Intelligence skills or including: Handling ELT and scalability issues for enterprise level data warehouse

Creating ETLs/ELTs to handle data from various data sources and various formats

Strong hands-on experience of programming language like Python, Scala with Spark and Beam.

Solid hands-on and Solution Architecting experience in Cloud Technologies Aws, Azure and GCP (GCP preferred)

Hands on working experience of data processing at scale with event driven systems, message queues (Kafka/ Flink/Spark Streaming)

Hands on working Experience with GCP Services like BigQuery, DataProc, PubSub, Dataflow, Cloud Composer, API Gateway, Datalake, BigTable, Spark, Apache Beam,

Feature Engineering/Data Processing to be used for Model development

Experience gathering and processing raw data at scale (including writing scripts, web scraping, calling APIs, write SQL queries, etc.)

Experience building data pipelines for structured/unstructured, real-time/batch, events/synchronous/ asynchronous using MQ, Kafka, Steam processing

Hands-on working experience in analyzing source system data and data flows, working with structured and unstructured data

Must be very strong in writing SparkSQL queries

Strong organizational skills, with the ability to work autonomously as well as leading a team

Pleasant Personality, Strong Communication & Interpersonal Skills

A bachelor's degree in computer science, computer engineering, or a related discipline is required to work as a technical lead

Certification in GCP would be a big plus

Individuals in this field can further display their leadership skills by completing the Project Management Professional certification offered by the Project Management Institute.