StreamSets Lead - SQL/Data Pipeline (6-7 yrs)

Position : StreamSet Lead.

Permanent Opportunity.

Location : Remote.

Duration : Full Time.

Level of experience : 6+ years.

Salary : 15-23 LPA based on level of experience.

Notice Period : Immediate Joiner/ Currently Serving/ Less than 45 days.

Relevant years of experience in Streamsets : 4+ Years.

(Open for Freelancer/Contractor as well if someone can support on need basis with remote option but strong expertise in StreamsetsMin 4+ years in StreamSet and SQL).

Job Responsibilities :

StreamSets Developer Lead :

- Designing, developing, and deploying data pipelines using StreamSets.

- Configuring data sources, transformations, and destinations within pipelines.

- Monitoring and troubleshooting pipeline performance and errors.

- Collaborating with data architects and stakeholders to understand data requirements and ensure pipeline efficiency.

- Implementing best practices for data governance, security, and compliance.

- Basic understanding of Apache Kafka architecture, including topics, partitions, brokers, and consumer groups.

Responsibilities :

Pipeline Development :

- Designing and developing data pipelines using StreamSets based on business requirements.

- Implementing data transformations, validations, and enrichments within pipelines.

- Testing pipelines for performance, reliability, and data quality.

Pipeline Monitoring and Maintenance :

- Monitoring pipeline performance metrics such as throughput, latency, and error rates.

- Troubleshooting and resolving issues related to pipeline failures or data inconsistencies.

- Optimizing pipelines for efficiency and scalability as data volumes grow.

Security and Compliance :

- Ensuring that data handling practices within StreamSets comply with organizational security policies and regulatory requirements.

- Implementing encryption, access controls, and auditing mechanisms to protect sensitive data.

Documentation and Knowledge Sharing :

- Documenting pipeline configurations, dependencies, and operational procedures.

- Sharing knowledge and best practices with team members through training sessions, documentation, and collaboration tools.

Performance Optimization :

- Analyzing pipeline performance bottlenecks and optimizing configurations for improved efficiency.

- Scaling pipelines to handle increasing data volumes and processing requirements.

Collaboration and Communication :

- Collaborating with cross-functional teams including developers, data scientists, and business analysts to understand data requirements and ensure alignment with business objectives.

- Communicating updates, issues, and resolutions effectively with stakeholders.