MBOS - Data Engineer

• We are looking for experienced data engineers to join our team. You will use various methods to transform raw data into useful data systems.
• To succeed in this data engineering position, you should have strong analytical skills and the ability to combine data from different sources. Data engineer skills also include familiarity with several programming languages and knowledge of learning machine methods.

Responsibilities

• Experience in building and managing data pipelines.
• experience with development and operations of data pipelines in the cloud (Preferably Azure.)
• Experience with distributed data/computing tools: Map/Reduce, Hadoop, Hive, Spark
• Deep expertise in architecting and data pipelines in cloud using cloud native technologies.
• Good experience in both ETL and ELT Ingestion patterns
• Hands-on experience working on large volumes of data(Petabyte scale) with distributed compute frameworks.
• Good Understanding of container platforms Kubernetes and docker
• Excellent knowledge and experience with object-oriented programming
• Familiarity developing with RESTful API interfaces.
• Experience in markup languages such as JSON and YAML
• Proficient in relational database design and development
• Good knowledge on Data warehousing concepts
• Working experience with agile scrum methodology

Technical Skills

• Strong skills in distributed cloud Data analytics platforms like Databricks, HD insight, EMR cluster etc.
• Strong in Programming Skills -Python/Java/R/Scala etc.
• Experience with stream-processing systems: Kafka, Apache Storm, Spark-Streaming, Apache Flink, etc.
• Hands-on working knowledge in cloud data lake stores like Azure Data Lake Storage.
• Data pipeline orchestration with Azure Data Factory, Amazon Data Pipeline
• Good Knowledge on File Formats like ORC, Parquet, Delta, Avro etc.
• Good Experience in using SQL and No-SQL. databases like MySQL, Elasticsearch, MongoDB, PostgreSQL and Cassandra running huge volumes of data
• Strong experience in networking and security measures
• Proficiency with CI/CD automation, and specifically with DevOps build and release pipelines
• Proficiency with Git, including branching/merging strategies, Pull Requests, and basic command line functions
• Strong experience in networking and security measures
• Good Data Modelling skills

Job Responsibilities

• Cloud Analytics, Storage, security, resiliency and governance
• Building and maintaining the data architecture for data Engineering and data science projects
• Extract Transform and Load data from sources Systems to data lake or Datawarehouse leveraging combination of various IaaS or SaaS components
• Perform compute on huge volume of data using open-source projects like Databricks/spark or Hadoop
• Define table schema and quickly adapt with the pipeline
• Working with High volume unstructured and streaming datasets
• Responsible to manage NoSQL Databases on Cloud (AWS, Azure etc.)
• Architect solutions to migrate projects from On-premises to cloud
• Research, investigate and implement newer technologies to continually evolve security capabilities
• Identify valuable data sources and automate collection processes
• Implement adequate networking and security measures for the data pipeline
• Implement monitoring solution for the data pipeline
• Support the design, and implement data engineering solutions
• Maintain excellent documentation for understanding and accessing data storage
• Work independently as well as in teams to deliver transformative solutions to clients
• Be proactive and constantly pay attention to the scalability, performance and availability of our systems
• Establishes privacy/security hierarchy and regulates access
• Collaborate with engineering and product development teams
• Systematic problem-solving approach with strong communication skills and a sense of ownership and drive