Data Engineer - Apache Flink (8-13 yrs)

The Lead Data Engineer will lead the development of complex and exciting platforms in our Arctic Wolf Labs department by contributing to Security Research and Threat Intelligence products and initiatives, developing cutting-edge systems, services, and frameworks for Arctic Wolf Labs. We are looking for experienced full-stack developers to join our team and work with us to continuously evolve and expand the capabilities of our security products and services.

Responsibilities :

- Own R& D of distributed, highly scalable, and fault-tolerant microservices.

- Use test-driven development techniques to develop beautiful, efficient, and secure code.

- Create and scale high-performance services that bring new capabilities to Arctic Wolf's Security and Threat Research organizations.

- Document and maintain software functionality and specifications.

- Identify problems proactively and propose novel solutions to solve them.

- Continuously learn and expand your technical horizons.

Requirements:

- Experience researching and implementing new and innovative software technologies and standards.

- Experience with designing and developing highly scalable and fault-tolerant distributed services for high throughput and low latency from the ground up in a complex production system.

- Experience with SQL and NoSQL databases (MongoDB, Elastic Search), KV stores (Redis, etc. ), Message Queues (RabbitMQ, Kafka, etc. ), Event-Driven Compute Frameworks (Step Functions, Lambda Functions), Stream and Batch Data Processing (Apache Spark, Flink) in production.

- Excellent understanding of Object-oriented, Reactive, and Event-Driven programming concepts.

- Has working knowledge of Data Lake technologies, data storage formats (Parquet, ORC, Avro), and query engines (Athena, Presto, Dremio), and associated concepts for building optimized solutions at scale.

- Maintains a proficient level in one of the following programming languages or similar- Python, Java, Go.

- Has experience with data pipeline tools (Flink, Spark, or Ray) and orchestration tools such as Airflow, Dagster, or Step Functions.

- Is an expert in implementing data streaming and event-based data solutions (Kafka, Kinesis, SQS/SNS or the like).

- Good understanding of functional programming concepts.

- Willingness and ability to pick up new object-oriented and functional programming languages suitable for the job.

- Experience building and shipping scalable microservices and APIs.

- Demonstrable experience with integrating, refactoring, and troubleshooting services in a large production system.

- Experience with developing and deploying assets in the cloud - preferably AWS and Kubernetes using IAC (infrastructure as code).

- Comfortable deploying software with CI / CD tools including Jenkins, Harness, Terraform, etc.

- Strong understanding of the SDLC (Software Development Lifecycle) and product development processes.

- Experience architecting, designing, developing, testing, and delivering software in an agile environment.

- Familiarity with cybersecurity concepts is an asset.

- 5+ years of experience in building and shipping scalable microservices and APIs.

- 2+ years of experience leading/mentoring other technical contributors on standards and best practices.