What is the difference between repartition and coalesce?

Question

Accepted Answer

Repartition and coalesce are methods used in Apache Spark to manage the number of partitions in a dataset.

Repartition is used to increase or decrease the number of partitions in a dataset by shuffling the data across the cluster.
Coalesce is used to decrease the number of partitions in a dataset without shuffling the data, which can improve performance.
Repartition is more expensive as it involves data shuffling, while coalesce is less expensive as it avoids shuffling.
Example: df.repartition(10) - repartitions the DataFrame into 10 partitions.
Example: df.coalesce(5) - coalesces the DataFrame into 5 partitions.

Top Application Lead Interview Questions Asked at Accenture