Physical Plans in Spark SQL—continues - David Vrba (Socialbakers)
Why Stack Overflow recommends it
Stack Overflow users link it in Spark performance answers and comments when explaining how operators like window, shuffle, sort-merge join and Catalyst optimizer rules appear in the physical plan.
Recommended by 2 people across 3 different months, 2020 to 2026.
What it is
A 39-minute Spark Summit talk by David Vrba of Socialbakers on reading and interpreting physical query plans in Spark SQL. It walks through common plan operators, what happens during physical planning, and how to rewrite queries for better performance, drawing on Spark source code and real queries.
What people said
Recommends David Vrba · 2020-06 · Stack Overflow · on “Spark Window aggregation vs. Group By/Join performance”
It depends on the data. More specifically here it depends on the cardinality of the name column. If the cardinality is small, the data will be small after the aggregation and the aggregated result can be broadcasted in the join. In that case, the join will be faster than the window . On the other hand, if the cardinality is big and the data is large after the aggregation, so the join will be planed with SortMergeJoin , using window will be more efficient. In the case of window we have 1 total shuffle + one sort. In the case of SortMergeJoin we have the same in the left branch (total shuffle + sort) plus additional reduced shuffle and sort in the right branch (by reduced I mean that the data
Recommends Matt Andruff · 2022-05 · Stack Overflow · on “Difference Beetween Window function and OrderBy in Spark”
May be of interest to you, not specifically on topic, but shows another area where windows out performs groupby+join [this video]
Recommends Matt Andruff · 2022-06 · Stack Overflow · on “replace or optimize the join in spark sql”
This has been answer here [LINK] : It depends on the data. More specifically here it depends on the cardinality of the name column. If the cardinality is small, the data will be small after the aggregation and the aggregated result can be broadcasted in the join. In that case, the join will be faster than the window. On the other hand, if the cardinality is big and the data is large after the aggregation, so the join will be planed with SortMergeJoin, using window will be more efficient. In the case of window we have 1 total shuffle + one sort. In the case of SortMergeJoin we have the same in the left branch (total shuffle + sort) plus additional reduced shuffle and sort in the right branch
Cites Arjunlal M.A · 2021-09 · Stack Overflow · on “Apache Spark Dataframe How to turn off partial aggregation when using groupBy?”
Thanks for the help, this makes sense now. I've been trying to turn off Catalyst's PredicatePushDown optimization as shown in @8:48. I used ` spark.conf.set( "spark.sql.optimizer.excludeRules", "org.apache.spark.sql.catalyst.optimizer.PushDownPredicate" ) ` This doesn't work. I guess it's because Spark hasn't added the feature for such a configuration. Or am I missing something?
Cites Matthew Thomas · 2026-02 · Stack Overflow · on “Window Vs GroupBy Performance in Spark”
Many window function queries can be alternatively performed by aggregating data (i.e. groupBy ) followed by join with the original data. Whether one approach will be faster than the other will be case by case, however, there is a simple heuristic that will guide the approach to take -- based off of your data and query. If the cardinality of the key by which you will partitionBy (window function approach) or groupBy ("aggregating -> joining" approach) is large choose the former, and if it is small choose the latter. Source: AWS Prescriptive Guidance [LINK] Another way to think about this: if the original data is large and the aggregated data is large (because you aggregated by a high cardinal