Prepare for Your Wadhwani Foundation Interview with Real Experiences!
View interviews
AmbitionBox
Search
Jobs
- Reviews
- Salaries
- Interview Questions
- About Company
- Benefits
- Jobs
- Office Photos
- Community
Add office photos
All Filters
76 Wadhwani Foundation Jobs
Data Engineer - WGDT
Wadhwani Foundation
posted 6 days ago
Job Role Insights
Flexible timing
Key skills for the job
Job Description
This is an exciting opportunity to join a dynamic and growing organization,working at the forefront of technology trends and developments in social impactsector. Wadhwani Center for Government Digital Transformation (WGDT) works withthe government ministries and state departments in India with a mission of \u201cEnablingdigital transformation to enhance the impact of government policy, initiativesand programs \u201d.
We are seeking a highly motivated anddetail-oriented individual to join our team as a Data Engineer withexperience in the designing, constructing, and maintaining the architecture andinfrastructure necessary for data generation, storage and processing andcontribute to the successful implementation of digital government policies andprograms. You will play a key role in developing, robust, scalable, andefficient systems to manage large volumes of data, make it accessible foranalysis and decision-making and driving innovation & optimizing operationsacross various government ministries and state departments in India.
Key Responsibilities:
a. Data Architecture Design : Design, develop, and maintain scalable data pipelines andinfrastructure for ingesting, processing, storing, and analyzing large volumesof data efficiently. This involves understanding business requirements andtranslating them into technical solutions.
b. Data Integration: Integrate data from various sources such as databases, APIs, streamingplatforms, and third-party systems. Should ensure the data is collectedreliably and efficiently, maintaining data quality and integrity throughout theprocess as per the Ministries/government data standards.
c. Data Modeling: Design and implement data modelsto organize and structure data for efficient storage and retrieval. They usetechniques such as dimensional modeling, normalization, and denormalizationdepending on the specific requirements of the project.
d. Data Pipeline Development/ ETL (Extract,Transform, Load): Develop data pipeline/ETL processes toextract data from source systems, transform it into the desired format, andload it into the target data systems. This involves writing scripts or usingETL tools or building data pipelines to automate the process and ensure dataaccuracy and consistency.
e. Data Quality and Governance: Implement data quality checks and data governance policies to ensuredata accuracy, consistency, and compliance with regulations. Should be able to designand track data lineage, data stewardship, metadata management, building businessglossary etc
f. Data lakes or Warehousing: Design and maintain data lakes and data warehouse to store and managestructured data from relational databases, semi-structured data like JSON orXML, and unstructured data such as text documents, images, and videos at anyscale. Should be able to integrate with big data processing frameworks such asApache Hadoop, Apache Spark, and Apache Flink, as we'll as with machine learningand data visualization tools.
g. Data Security : Implement security practices, technologies, and policies designed toprotect data from unauthorized access, alteration, or destruction throughoutits lifecycle. It should include data access, encryption, data masking andanonymization, data loss prevention, compliance, and regulatory requirementssuch as DPDP, GDPR, etc
h. Database Management: Administer and optimize databases, both relational and NoSQL, to managelarge volumes of data effectively.
i. Data Migration: Plan and execute data migration projects to transfer data betweensystems while ensuring data consistency and minimal downtime.
a. Performance Optimization : Optimize data pipelines and queries for performance and scalability. Identifyand resolve bottlenecks, tune database configurations, and implement cachingand indexing strategies to improve data processing speed and efficiency.
b. Collaboration: Collaborate with data scientists, analysts, and other stakeholders tounderstand their data requirements and provide them with access to thenecessary data resources. They also work closely with IT operations teams todeploy and maintain data infrastructure in production environments.
c. Documentation and Reporting: Document their work including data models, data pipelines/ETLprocesses, and system configurations. Create documentation and provide trainingto other team members to ensure the sustainability and maintainability of datasystems.
d. Continuous Learning: Stay updated with the latest technologies and trends in data engineeringand related fields. Should participate in training programs, attendconferences, and engage with the data engineering community to enhance theirskills and knowledge.
Desired Skills/ Competencies
- Education: A Bachelors or Masters degree in Computer Science, Software Engineering, Data Science, or equivalent with at least 5 years of experience.
- Database Management: Strong expertise in working with databases, such as SQL databases (eg, MySQL, PostgreSQL) and NoSQL databases (eg, MongoDB, Cassandra).
- Big Data Technologies: Familiarity with big data technologies, such as Apache Hadoop, Spark, and related ecosystem components, for processing and analyzing large-scale datasets.
- ETL Tools: Experience with ETL tools (eg, Apache NiFi, Talend, Apache Airflow, Talend Open Studio, Pentaho, Infosphere) for designing and orchestrating data workflows.
- Data Modeling and Warehousing: Knowledge of data modeling techniques and experience with data warehousing solutions (eg, Amazon Redshift, Google BigQuery, Snowflake).
- Data Governance and Security: Understanding of data governance principles and best practices for ensuring data quality and security.
- Cloud Computing: Experience with cloud platforms (eg, AWS, Azure, Google Cloud) and their data services for scalable and cost-effective data storage and processing.
- Streaming Data Processing: Familiarity with real-time data processing frameworks (eg, Apache Kafka, Apache Flink) for handling streaming data.
KPIs:
- Data Pipeline Efficiency: Measure the efficiency of data pipelines in terms of data processing time, throughput, and resource utilization. KPIs could include average time to process data, data ingestion rates, and pipeline latency.
- Data Quality Metrics: Track data quality metrics such as completeness, accuracy, consistency, and timeliness of data. KPIs could include data error rates, missing values, data duplication rates, and data validation failures.
- System Uptime and Availability: Monitor the uptime and availability of data infrastructure, including databases, data warehouses, and data processing systems. KPIs could include system uptime percentage, mean time between failures (MTBF), and mean time to repair (MTTR).
- Data Storage Efficiency: Measure the efficiency of data storage systems in terms of storage utilization, data compression rates, and data retention policies. KPIs could include storage utilization rates, data compression ratios, and data storage costs per unit.
- Data Security and Compliance: Track adherence to data security policies and regulatory compliance requirements such as DPDP, GDPR, HIPAA, or PCI DSS. KPIs could include security incident rates, data access permissions, and compliance audit findings.
- Data Processing Performance: Monitor the performance of data processing tasks such as ETL (Extract, Transform, Load) processes, data transformations, and data aggregations. KPIs could include data processing time, CPU usage, and memory consumption.
- Scalability and Performance Tuning: Measure the scalability and performance of data systems under varying workloads and data volumes. KPIs could include scalability benchmarks, system response times under load, and performance improvements achieved through tuning.
- Resource Utilization and Cost Optimization: Track resource utilization and costs associated with data infrastructure, including compute resources, storage, and network bandwidth. KPIs could include cost per data unit processed, cost per query, and cost savings achieved through optimization.
- Incident Response and Resolution: Monitor the response time and resolution time for data-related incidents and issues. KPIs could include incident response time, time to diagnose and resolve issues, and customer satisfaction ratings for support services.
- Documentation and Knowledge Sharing : Measure the quality and completeness of documentation for data infrastructure, data pipelines, and data processes. KPIs could include documentation coverage, documentation update frequency, and knowledge sharing activities such as internal training sessions or knowledge base contributions.
Employment Type: Full Time, Permanent
Read full job description
What people at Wadhwani Foundation are saying
What Wadhwani Foundation employees are saying about work life
based on 65 employees
Flexible timing
Monday to Friday
No travel
Day Shift
Wadhwani Foundation Benefits
View more benefits
Compare Wadhwani Foundation with
Magic Bus India Foundation
Compare
CARE
Compare
Dr. Reddy's Foundation
Compare
World Vision
Compare
Labournet Services
Compare
ICICI Foundation for Inclusive Growth
Compare
Save the Children
Compare
Naandi Foundation
Compare
Azim Premji Foundation
Compare
SOS Children's Village
Compare
Childline India Foundation
Compare
Medecins Sans Frontieres
Compare
Pradan
Compare
Don Bosco Tech Society
Compare
Population Services International (PSI)
Compare
Teach For India
Compare
Sankara Nethralaya
Compare
American India Foundation
Compare
BAIF Development Research Foundation
Compare
HCL Foundation
Compare
Similar Jobs for you
Content & QA Specialist
1-3 Yrs
Kolkata, Mumbai, New Delhi +4 more
Animation, QA Engineering, Photoshop +5 more
Wadhwani Foundation - Engineering Director (15-18 yrs)
15-18 Yrs
Javascript, Engineering Management, Full Stack +2 more
Wadhwani Foundation - Senior Data Engineer - ETL (5-10 yrs)
5-10 Yrs
Data Engineering, ETL Testing, DBMS +7 more
Wadhwani Foundation - Platform Frontend Developer - React.js/Javascript (5-8 yrs)
5-8 Yrs
UI and UX, Javascript, Flux +2 more
Wadhwani Foundation - Engineering Manager - Full Stack Development (15-17 yrs)
15-17 Yrs
Azure DevOps, Engineering Management, MongoDB +3 more
Wadhwani Foundation - Agentic AI Implementation Engineer - RAG Pipelines (6-10 yrs)
6-10 Yrs
Artificial Intelligence, Chatgpt, Brokerage +1 more
Wadhwani Foundation - Solution Analyst - Artificial Intelligence/Machine Learning (3-7 yrs)
3-7 Yrs
Data Analytics, Artificial Intelligence, Machine Learning +3 more
Wadhwani Foundation - Quality Assurance Tester - Manual Testing (4-8 yrs)
4-8 Yrs
Manual Testing, Performance Testing, Selenium Testing +5 more
Wadhwani Foundation - Quality Assurance Test Engineer - Manual & Automation Testing (4-8 yrs)
4-8 Yrs
Manual Testing, Postman, Automation Testing +5 more
Share an Interview