Datavail Infotech Senior Technical Specialist Interview Questions and Answers

Spark DAG (Directed Acyclic Graph) represents the execution plan for Spark jobs, optimizing data processing tasks.

• A DAG is a graph structure that consists of nodes (operations) and edges (data flow) without cycles.

• In Spark, when an action (like count or collect) is called, it triggers the creation of a DAG.

• Each transformation (like map, filter) creates a new RDD, forming a lineage that Spark uses to optimize execution.

• ...read more

Data validation in Hive involves using built-in functions and custom scripts to ensure data accuracy and consistency.

• Use built-in functions like IS NULL, IS NOT NULL, and COALESCE to check for missing or null values

• Use regular expressions and pattern matching to validate data formats

• Write custom scripts to perform more complex data validation tasks

• Perform data profiling to identify potential data quality issues

• Use data...read more

I have worked on various storage arrays including EMC VNX, NetApp FAS, and Dell EMC Unity.

• EMC VNX

• NetApp FAS

• Dell EMC Unity

Performing a test activity on a remote site

• Ensure proper communication channels are established

• Define clear objectives and scope of the test

• Select appropriate tools and equipment for the test

• Conduct a pre-test site survey to identify potential issues

• Execute the test plan and document results

• Analyze the results and report findings to stakeholders

Storages can be classified into three types: direct-attached, network-attached, and storage area networks.

• Direct-attached storage (DAS) is directly connected to a server or computer, such as a hard drive or USB drive.

• Network-attached storage (NAS) is a file-level storage connected to a network, allowing multiple users to access the same data simultaneously.

• Storage area networks (SAN) are block-level storage connected t...read more

Replication types include snapshot, transactional, and merge. Basic task procedures involve setting up publications and subscriptions.

• Snapshot replication copies a snapshot of the entire database to the subscriber

• Transactional replication copies only the changes made to the database since the last replication

• Merge replication allows for bidirectional synchronization of data

• Setting up publications involves selecting the...read more