BrightEdge - Senior Software Engineer - Data Pipeline (3-6 yrs)

Location : Remote

Experience : 3 - 6 years

Employment Type : Full Time

About the Role :

We are seeking a highly motivated and experienced Engineer to join our dynamic team. In this role, you will be instrumental in building and maintaining robust, scalable, and efficient data pipelines. You will be responsible for designing, implementing, and optimizing data collection, storage, and processing workflows, ensuring the seamless flow of both structured and unstructured data. Your expertise in web scraping, API integration, database optimization, cloud logging, and anomaly detection will be crucial in maintaining the integrity and performance of our data infrastructure. You will collaborate with cross-functional teams to deliver high-quality, data-driven solutions.

Key Responsibilities :

Data Pipeline Development & Maintenance :

- Design, develop, and maintain efficient data collection pipelines for ingesting and storing diverse data types (structured and unstructured).

- Implement robust data transformation and processing logic to ensure data quality and consistency.

- Build and manage real-time and batch data processing workflows.

Web Scraping & API Integration :

- Develop and implement web scraping techniques for extracting data from various online sources.

- Integrate with various APIs for real-time and batch data retrieval, ensuring data accuracy and reliability.

Database Management & Optimization :

- Design, implement, and optimize database schemas in MySQL, Redis, and other relevant storage solutions.

- Perform query optimization and implement indexing strategies to enhance data retrieval performance.

- Manage database scalability and ensure data integrity.

Logging & Data Streaming :

- Implement and manage comprehensive logging solutions (cloud and local) for monitoring system performance and debugging issues.

- Set up and maintain real-time data streaming pipelines for efficient data distribution and processing.

- Ensure effective monitoring of data streams for potential issues.

Anomaly Detection & Alerting :

- Develop and implement anomaly detection mechanisms to identify data inconsistencies and potential errors.

- Set up automated alerts and notifications for critical events and anomalies.

- Create and maintain monitoring dashboards for data pipeline health and performance.

Data Optimization & Architecture :

- Develop and implement data optimization strategies for faster retrieval and efficient storage.

- Design and maintain scalable, fault-tolerant data architectures to support growing data volumes and processing needs.

- Conduct performance analysis and implement improvements to optimize data workflows.

Collaboration & Communication :

- Collaborate closely with data scientists, analysts, and other engineers to ensure data integrity and performance.

- Communicate effectively with stakeholders to understand data requirements and provide technical updates.

- Participate in code reviews and contribute to improving development processes.

Required Skills & Experience :

Data Collection & Integration :

- Proven experience in web scraping and API integration for data acquisition.

- Strong understanding of data collection methodologies and best practices.

Logging & Streaming :

- Hands-on experience with cloud logging, data logging, and data streaming technologies.

- Ability to implement and manage logging and streaming pipelines.

Database Expertise :

- Extensive knowledge of MySQL, Redis, and other relevant database systems.

- Proficiency in database design, query optimization, and performance tuning.

Data Optimization & Retrieval :

- Expertise in data optimization techniques for efficient retrieval and storage.

- Strong understanding of query optimization and indexing strategies.

Data Handling :

- Proven ability to handle both structured and unstructured data.

- Experience in data transformation and processing.

Data Pipelines & Architecture :

- Ability to design and implement robust and scalable data pipelines.

- Understanding of data architecture principles and best practices.

Problem Solving & Monitoring :

- Strong problem-solving skills with a focus on performance and scalability.

- Experience in implementing scripts, alerts, and anomaly detection mechanisms.

Programming & Scripting :

- Proficiency in at least one scripting language (e.g., Python) and SQL.

Preferred Qualifications :

- Experience with cloud platforms (AWS, GCP, or Azure) and their data services.

- Knowledge of Big Data frameworks (Kafka, Spark, etc.).

- Familiarity with containerization and orchestration technologies (Docker, Kubernetes).

- Experience in developing and deploying microservices.

- Familiarity with data warehousing and data lake concepts.