Advanced Apache Spark Training - Sameer Farooqui (Databricks)
Why Stack Overflow recommends it
Stack Overflow users link it when explaining Spark internals, such as shuffle spill, partition counts, DAG scheduling and how many threads to run per core.
Recommended by 4 people across 3 different months, 2015 to 2019.
What it is
A six-hour recorded Spark Summit 2015 training session in New York where Databricks evangelist Sameer Farooqui covers Spark core in depth. Topics include the big data ecosystem, RDDs, YARN and standalone cluster managers, Spark internals, accumulators, broadcast variables, Spark Streaming and how the 2014 100 TB sort record was won.
What people said
Strong rec Indrajit Swain · 2017-12 · Stack Overflow · on “How to find out the task dependency in apache spark”
You have already almost answered your question . You can check the RDD dependencies using API also Spar todebug String [LINK] gives the facilities to check the RDD Dependencies . Please check spark-rdd-dependencies [LINK] . At high level, when any action is called on the RDD, Spark creates the DAG and submits it to the DAG scheduler. The DAG scheduler divides operators into stages of tasks. A stage is comprised of tasks based on partitions of the input data. The DAG scheduler pipelines operators together. For e.g. Many map operators can be scheduled in a single stage. The final result of a DAG scheduler is a set of stages. The Stages are passed on to the Task Scheduler.The task scheduler lau
Recommends Alister Lee · 2015-06 · Stack Overflow · on “How to optimize shuffle spill in Apache Spark application”
Learning to performance-tune Spark requires quite a bit of investigation and learning. There are a few good resources including this video [this video] . Spark 1.4 has some better diagnostics and visualisation in the interface which can help you. In summary, you spill when the size of the RDD partitions at the end of the stage exceed the amount of memory available for the shuffle buffer. You can: Manually repartition() your prior stage so that you have smaller partitions from input. Increase the shuffle buffer by increasing the memory in your executor processes ( spark.executor.memory ) Increase the shuffle buffer by increasing the fraction of executor memory allocated to it ( spark.shuffle.memor
Recommends Maksud · 2015-06 · Stack Overflow · on “Apache Spark architecture”
Here are the answers to your questions Spark will try to load 128Mb chunk into memory and process it in RAM. Keep in mind that that the size in memory can be several times larger than the original size of the raw file due to Java overhead (Java headers, etc). From my experience, it can be 2-4 time larger. If there is not enough memory (RAM) Spark will spill the data to local disk. You may want to tweak these two parameters to minimize the spill: spark.shuffle.memoryFraction and spark.storage.memoryFraction . Spark will always try to use local data from in your HDFS. If the chunk not available locally it will retrieve it from another node in the cluster. more info [LINK] On shuffle, you do no
Recommends Luis Miguel Mejía Suárez · 2019-09 · Stack Overflow · on “how to understand the foreach function in scala in my case?”
@jayWong Yeah of course, checkout the original spark paper [LINK] , this Coursera course [LINK] , this blog post [LINK] about memory management was very helpful too me, I also watched this _looooong_ video [this video] when I was starting, not sure if it is still updated for today, but I guess the main concepts still apply. Finally obviously the official documentation, and remember the scaladoc is your friend.
Cites Sathish · 2015-06 · Stack Overflow · on “How DAG works under the covers in RDD?”
Even i have been looking in the web to learn about how spark computes the DAG from the RDD and subsequently executes the task. At high level, when any action is called on the RDD, Spark creates the DAG and submits it to the DAG scheduler. The DAG scheduler divides operators into stages of tasks. A stage is comprised of tasks based on partitions of the input data. The DAG scheduler pipelines operators together. For e.g. Many map operators can be scheduled in a single stage. The final result of a DAG scheduler is a set of stages. The Stages are passed on to the Task Scheduler.The task scheduler launches tasks via cluster manager (Spark Standalone/Yarn/Mesos). The task scheduler doesn't know ab
Cites Sathish · 2015-06 · Stack Overflow · on “How Spark works internally”
Spark revolves around the concept of a resilient distributed dataset (RDD), which is a fault-tolerant collection of elements that can be operated on in parallel. RDDs support two types of operations: transformations, which create a new dataset from an existing one, and actions, which return a value to the driver program after running a computation on the dataset. Spark translates the RDD transformations into something called DAG (Directed Acyclic Graph) and starts the execution, At high level, when any action is called on the RDD, Spark creates the DAG and submits to the DAG scheduler. The DAG scheduler divides operators into stages of tasks. A stage is comprised of tasks based on partitions
Cites zero323 · 2016-03 · Stack Overflow · on “pyspark partitioning data using partitionby”
Not exactly. Spark, including PySpark, is by default using hash partitioning [LINK] . Excluding identical keys there is no practical similarity between keys assigned to a single partition. There is no simple answer here. All depends on amount of data and available resources. Too large [LINK] or too low number of partitions will degrade the performance. Some resources [this video] claim the number of partitions should around twice as large as the number of available cores. From the other hand a single partition typically shouldn't contain more than 128MB and a single shuffle block cannot be larger than 2GB (See SPARK-6235 [LINK] ). Finally you have to correct for potential data skews. If some keys
Cites scott miles · 2017-05 · Stack Overflow · on “Number of threads vs processor core?”
Per mine current understanding, Ideally for best perormance, we should not create threads more than number of cores in processor as there will be overheads like thread switching. But this video Apache Spark Video [this video] says at 91 minute, we should oversubscribe the thread i.e. create more threads than cores for better performaance . Only Advantage i can think of creating extra thread all tasks will get cpu cycles in kind of round robin fashion instead of limited threads finishing first set of task then pick another set. I am not sure which one is correct ?
Cites Tiffany · 2017-08 · Stack Overflow · on “Number of cores used by local[*]”
In this talk on YouTube [this video] , the speaker states that we should specificy a higher number than the number of logical cores when using master as local[N] . He states than local[*] isn't optimal then. However this video is a bit old. Is it still relevant to do so ?
Cites zero323 · 2018-02 · Stack Overflow · on “How does Spark determine the number of slave node simultaneous threads?”
Core in Spark is __an alias for thread__. We don't manage "physical" processor cores here, but declared parallelism capability. 8 core worker is free to advertise 1000 `CORES` - you can start with old but still good [this video] _I will get at most 8 threads on an 8 core machine_ - no. You'll get at most advertised `CORES` threads.