Apache Spark Core—Deep Dive—Proper Optimization Daniel Tomes Databricks
Why Stack Overflow recommends it
Stack Overflow users link it when answering questions about shuffle partition counts, partition sizing, and the localCheckpoint-then-repartition pattern for tuning Spark performance.
Recommended by 2 people across 2 different months, 2020 to 2023.
What it is
A 90-minute Spark + AI Summit 2019 talk by Daniel Tomes of Databricks on tuning Spark jobs by understanding Spark core. It explains read, shuffle and write partitions, sizing partitions and files, where shuffle data goes between stages, and why jobs stall on a few straggler tasks or fail to speed up when nodes are added.
What people said
Recommends Alex Ott · 2020-09 · Stack Overflow · on “what happens when shuffle partition is greater than 200( spark.sql.shuffle.partitions 200(by default) in dataframe)”
The number 200 was selected as default based on the typical workloads on the relative big clusters with enough resources allocated for jobs. Otherwise this number should be selected based on the 2 factors - number of available cores, and partition size (it's recommended to keep partitions close to 100Mb). The selected number of partitions should be the multiply of the number of available cores, but shouldn't be very big (typically it's 1-3 x of number of cores). If number of partitions is greater than default, shouldn't change Spark's behavior - it will just increase the number of tasks that Spark will need to execute). You can watch this talk from Spark + AI Summit 2019 [this video] - it covers
Recommends Kafels · 2021-09 · Stack Overflow · on “Most efficient way of pivoting huge dataframes pyspark”
@Mike There's this video [this video] published by Databricks telling tips to improve your performance. It's not a silver bullet, but it really helps to understading key points to tuning
Cites danielcahall · 2022-10 · Stack Overflow · on “PySpark LSH approxSimilarityJoin taking too much time”
Without being able to view the Spark UI and inspect the plan to identify bottlenecks, it seems likely the issue is the join. There are a few mechanisms for optimizing Spark performance when dealing with joins: Broadcast Joins Depending on the size of each hashed dataframe, if one is much smaller than the other, you could do a broadcast join, where the smaller dataframe is copied to each executor and joined to each partition of the larger dataframe. For example, in the snippet provided, since df_2 is much smaller than df_1 (500 << 89k) and by extension db_hashed_final is much smaller than db_hashed : from pyspark.sql.functions import broadcast model.stages[-1].approxSimilarityJoin(broadcast(d
Cites John Stud · 2022-12 · Stack Overflow · on “What is the proper use of localCheckpoint()”
I am in need of using localCheckpoint() to resize shuffle partitions, but I am unsure how to use it properly. I searched through Github as well as Stackoverflow and found effectively everyone repeating this block of code (with no further context, situation, reasoning, etc) which is from this databricks presentation: [this video] df.localCheckpoint(...).repartition(n).write(...) How do I use this properly? For instance, do I: df = df.localCheckpoint() or df.localCheckpoint() Under what conditions would I repartition? Is specifying the write necessary? Let's say I have specified my number of shuffle partitions like so: spark.conf.set("spark.sql.shuffle.partitions", 1200); # bunch of work here # loc
Cites krezno · 2023-06 · Stack Overflow · on “Spark force a distributed join”
You need to create what is called a stage barrier, something that will force spark not to push the coalesce and the join into the same stage. If you can manage holding the whole result on a single partition, the join result probably isn't too large. Consider replacing coalesce with repartition , which will do a full shuffle but also create a stage barrier and perform the join in parallel. This was fast enough for my use cases but might not be best practice as repartition is an expensive operation. It might be faster to call df.localCheckpoint which writes intermediate results to disk (but unlike cache creates a stage barrier) then coalesce but I haven't personally tried it. More elaboration