Cognizant Associate Data Engineer Interview Questions and Answers

To kill a job in Datastage

• Stop the job manually from the Director client

• Terminate the job from the command line using the dsjob command

• Kill the job process from the operating system level

• Delete the job from the Datastage repository

SQL queries with window functions

• Window functions perform calculations across a set of rows that are related to the current row

• Common window functions include ROW_NUMBER, RANK, DENSE_RANK, and NTILE

• Window functions are used with the OVER() clause to define the window or subset of rows to perform the calculation on

SORT BY, ORDER BY, CLUSTER BY, and DISTRIBUTE BY are SQL clauses used for data sorting and partitioning.

• SORT BY is used to sort the result set in ascending or descending order based on one or more columns.

• ORDER BY is used to sort the result set in ascending or descending order based on one or more columns. It is similar to SORT BY but can be used with other clauses like LIMIT and OFFSET.

• CLUSTER BY is used to group data...read more

Spark OutOfMemory (OOM) issues occur when the application exceeds memory limits, causing failures in processing data.

• Increase executor memory: Use the configuration 'spark.executor.memory' to allocate more memory to executors.

• Optimize data partitioning: Use 'repartition()' or 'coalesce()' to manage the number of partitions effectively.

• Use broadcast variables: For large lookup tables, use 'sc.broadcast()' to reduce memo...read more

Small file problem refers to the issue of having a large number of small files in a storage system.

• Small files can cause inefficiencies in storage and processing.

• Solutions include consolidating small files into larger ones or using a different storage system.

• Examples include Hadoop's SequenceFile format and Amazon S3's object size optimization.

RDS, VA, DF, VS, and DS are all acronyms related to data engineering.

• RDS stands for Relational Database Service, a managed database service by AWS.

• VA stands for Virtual Assistant, a software program that can assist with tasks.

• DF stands for Dataflow, a managed service by Google Cloud for data processing.

• VS stands for Virtual Server, a server that runs on a virtual machine.

• DS stands for Datastore, a NoSQL document databa...read more

Reading data from a .log file and extracting columns with a specific regex.

• Use Python's built-in 're' module to define the regex pattern.

• Open the .log file using Python's 'open' function.

• Iterate through each line of the file and extract the desired columns using the regex pattern.

• Store the extracted data in a data structure such as a list or dictionary.

Calculate the count and profit from data over the last four years using SQL or data analysis tools.

• Use SQL queries like 'SELECT COUNT(*)' to get the total count of records.

• To calculate profit, use 'SUM(profit_column)' grouped by year.

• Example SQL: 'SELECT YEAR(date_column), COUNT(*), SUM(profit_column) FROM sales WHERE date_column >= DATE_SUB(CURDATE(), INTERVAL 4 YEAR) GROUP BY YEAR(date_column);'

• Consider filtering ...read more

Optimizations for data engineering

• Use indexing to speed up queries

• Partition data to improve query performance

• Use caching to reduce data retrieval time

• Optimize data storage format for faster processing

• Use parallel processing to speed up data processing

• Optimize network bandwidth usage

• Use compression to reduce storage and network usage

Answering how to read JSON in Python.

• Use the json module to load and parse JSON data

• Use the json.loads() method to load JSON data from a string

• Use the json.load() method to load JSON data from a file

• Access JSON data using keys or indexes

• Use the json.dumps() method to convert Python objects to JSON strings