Solution Architect - Data & Machine Learning Platform (12-15 yrs)

Experience : 12+ years.

Position Overview :

We are looking for a highly skilled Solution Architect to lead the design and implementation of a Data and Machine Learning Platform spanning edge, cloud, and on-prem components.

The ideal candidate will have deep experience in Azure cloud services, data engineering, edge computing, and ML lifecycle management.

Technical Expertise Azure Cloud Stack & DevOps :

- Azure Databricks (including ML workspace for Feature Store and Model Store).

- Azure Data Factory (ADF) for orchestration and compute.

- Azure Data Lake Storage (ADLS) implementing medallion architecture (raw, bronze, silver, gold).

- Azure Event Hub : Experience in defining topics, managing consumer groups, and integrating ETL events.

- Azure Streaming Analytics : Real-time data processing for telemetry and operational data.

- Azure Key Vault.

- Azure App Service.

- Azure Container Registry (ACR).

- Azure IoT Hub for connecting edge devices.

- Azure DevOps & GitHub Actions (for CI/CD pipelines).

- GitHub self-hosted runners for ML workflow automation.

Edge and On-Prem Integration :

- Strong experience in OT-IT integration and data extraction from industrial systems.

- Edge VM deployment using :

- Docker and Portainer for container orchestration.

- RabbitMQ for messaging (read/write services from edge).

- OPC UA for interfacing with PLCs (e.g., FX Filter, NH3 Compressor).

- IDMZ deployment practices and edge-to-cloud data service integration.

Machine Learning Platform (MLP) and MLOps :

- End-to-end ML lifecycle implementation : Feature Engineering, Model Training & Validation, Model Export, Versioning, and Deployment.

- Hands-on with ADB ML workspace, Feature Store, Model Store.

- Monitoring deployed models at 1-minute intervals.

- Understanding of training vs inference, cloud vs edge deployment.

- Cadence for ML models (Weekly Refresh, Monthly Retrain, Quarterly Revamp).

- Use of GitHub monorepo structure for managing model code.

Data Architecture & Integration :

- Implementation of medallion architecture in the data platform.

- Integration with Unity Catalog (UC) for governance, data sharing, and cataloging.

- Experience with CDC tools (e.g., Aecorsoft) for real-time SAP data ingestion.

- Consumption layer design for BI, ML, and operational workloads.

- Familiarity with streaming and API-based ingestion from external environments.

- Template-driven ingestion and mapping using configurations.

Governance and Data Modeling :

- Define and enforce data governance standards using Unity Catalog and enterprise frameworks.

- Design scalable data models to support operational analytics and ML features.

- Implement policies for access control, quality, and metadata tagging across DLZ/zones.

Key Responsibilities :

- Architect Integrated Solutions : Lead architectural design across edge, cloud, and ML across zones.

- Build and Govern Data Platform : Oversee ingestion, transformation, and cataloging across Raw ? Gold layers, aligned to UC.

- Enable Scalable ML Platform : Support ML teams with infrastructure for feature storage, model ops, and deployment pipelines.

- Edge Integration and Automation : Design robust and secure OT-IT interfaces with RabbitMQ, OPC UA, and container orchestration tools.

- Monitor and Optimize Pipelines : Set up real-time monitoring for ML and ETL pipelines; optimize for performance and cost.

- Governance and Security Compliance : Ensure enterprise compliance, tagging, and secure access across all zones and services.

- Lead CI/CD Automation : Use GitHub Actions and Azure DevOps to streamline deployment of ML workflows and platform components.