Infosys Data Engineer Interview Questions and Answers

Pardo in dataflow refers to a parallel data processing model that optimizes performance and resource utilization.

• Pardo stands for 'Parallel Do', enabling distributed processing of data across multiple nodes.

• It allows for efficient handling of large datasets by breaking them into smaller chunks.

• For example, in Apache Beam, Pardo can be used to apply a function to each element in a collection in parallel.

• This model ...read more

Calculate trailing zeros in a factorial using Python by counting factors of 5 in the numbers leading to n.

• Trailing zeros in n! are produced by factors of 10, which are made from pairs of 2 and 5.

• Since there are usually more factors of 2 than 5, we only need to count the factors of 5.

• The formula to calculate trailing zeros is: n // 5 + n // 25 + n // 125 + ... until n // 5^k is 0.

• Example: For 100!, trailing zeros =...read more

Identify the city with the highest revenue by analyzing data from various regions.

• Aggregate revenue data from all regions within the city.

• Use SQL queries like 'SELECT city, SUM(revenue) FROM sales GROUP BY city ORDER BY SUM(revenue) DESC LIMIT 1;'

• Consider factors like population, economic activity, and industry presence in each region.

• Example: If Region A has $1M and Region B has $2M, the total for the city is $3M...read more

SQL query to compare today's sales with yesterday's sales using aggregation and date functions.

• Use a table with sales data that includes a date column.

• Aggregate sales by date using SUM() function.

• Use a Common Table Expression (CTE) or subquery to get sales for today and yesterday.

• Calculate the difference between today's and yesterday's sales.

Spark optimization techniques enhance performance and resource utilization in distributed data processing tasks.

• Use DataFrames and Datasets for optimized execution plans.

• Leverage Catalyst Optimizer for query optimization.

• Apply Tungsten for memory management and code generation.

• Utilize partitioning to minimize data shuffling, e.g., using 'repartition' or 'coalesce'.

• Cache intermediate results with 'persist()' to avo...read more

ETL is a data integration process that involves extracting data, transforming it for analysis, and loading it into a target system.

• Extract: Gather data from various sources like databases, APIs, or flat files. Example: Pulling customer data from a CRM system.

• Transform: Clean and format the data to meet business requirements. Example: Converting date formats or aggregating sales data.

• Load: Insert the transformed da...read more

Efficiently joining large and small datasets requires strategic approaches to optimize performance and resource usage.

• Use a distributed computing framework like Apache Spark to handle large datasets efficiently.

• Consider filtering the larger dataset before the join to reduce its size, e.g., using a WHERE clause.

• Leverage indexing on the join keys to speed up the join operation.

• Use a broadcast join if the smaller dat...read more

Developed a data pipeline to ingest, process, and analyze customer feedback data for product improvement.

• Designed and implemented ETL processes to extract data from various sources

• Utilized Apache Spark for data processing and analysis

• Built data visualizations to present insights to stakeholders