Deep Dive: Apache Spark Memory Management
Why Stack Overflow recommends it
Stack Overflow users link it when answering questions about Spark executor memory, caching limits, spilling to disk and per-task resource sharing.
Recommended by 1 people across 1 different months, 2016 to 2020.
What it is
A conference talk from Spark Summit 2016 that explains how Apache Spark manages memory inside executors, including the split between storage and execution memory and the unified memory model. It also covers how tasks running in parallel on one executor share memory.
What people said
Recommends Constantine · 2018-10 · Stack Overflow · on “Executor Memory vs Java Heap Size memory”
You never get entire executor memory for caching. You can watch the video on Unified Memory Model here : [this video]
Cites Jacek Laskowski · 2016-12 · Stack Overflow · on “What is and how to control Memory Storage in Executors tab in web UI?”
CAUTION : You use the very, very old and currently unsupported Spark 1.5.2 (which I noticed after I had posted the answer) and my answer is about Spark 1.6+. The tooltip of Storage Memory may say it all: Memory used / total available memory for storage of data like RDD partitions cached in memory. [LINK] It is part of Unified Memory Management feature that was introduced in SPARK-10000: Consolidate storage and execution memory management [LINK] that (quoting verbatim): Memory management in Spark is currently broken down into two disjoint regions: one for execution and one for storage. The sizes of these regions are statically configured and fixed for the duration of the application. There ar
Cites Khozzy · 2017-05 · Stack Overflow · on “Assigning fixed resources for single task on executor”
According to Deep Dive: Apache Spark Memory Management [this video] there is a contention for tasks running in parallel on the same executor. From Spark 1.0+ there are two possible options: Option 1: Static assignment - resources are shared across tasks equally, Option 2: Dynamic assignment - resources are shared across tasks dynamically AFAIK Spark uses by default second option. Is there a possibility to manually specify maximum resources for each task?
Cites Vikash Pareek · 2020-06 · Stack Overflow · on “If Spark support memory spill to disk, how can Spark Out of Memory happen?”
>>"Thus storage portion of the memory can be cleared but not the execution portion" I politely challenge this statement, the document you referred says storage portion can't evict execution portion rather it will evict storage portion on LRU basis. But if all the executor memory is occupied by execution data then on further need execution data can be spill to disk. As per [this video] (at 7:11), execution data can also be spill to disk.