Regology - ETL Development Lead/Manager (4-6 yrs)

Job Description : ETL Development Lead/ Manager

Location : Hyderabad, India (Regology office in HITECH City)

Reports to : Head of India Office

Job Type : Full-time

Experience : 4-6yrs

Job Description :

We are seeking an experienced ETL Lead/Manager to manage and direct our web scraping team, specializing in building and maintaining robust data extraction solutions using Scrapy and ScrapyKeeper.

Key Responsibilities :

- Lead a team of data extraction specialists

- Design and architect complex web scraping solutions

- Develop and implement scalable data collection strategies

- Oversee the development and maintenance of web crawlers

- Ensure data quality, reliability, and compliance

- Manage spider infrastructure and deployment

- Implement error handling and resilience mechanisms

- Optimize scraping performance and efficiency

Technical Requirements :

- 4-6 years of experience in web scraping and ETL development

- Knowledge of Scrapy framework

- Familiarity with ScrapyKeeper

- Strong Python programming skills

- Familiarity with data cleaning and transformation processes

Additional Technical Skills

- Database technologies (PostgreSQL, MongoDB)

Qualifications :

- Bachelor's degree in Computer Science or related field

- Proven track record of leading technical teams

- Experience managing complex web scraping projects

Preferred Skills :

- Experience scraping diverse websites

- Understanding of different website structures and challenges