Introduction to AmpLab Spark Internals
Why Stack Overflow recommends it
Stack Overflow users link it when answering questions about how Spark builds the DAG, schedules stages and returns executor results to the driver.
Recommended by 1 people across 1 different months, 2015 to 2019.
What it is
A 74-minute talk by Matei Zaharia of UC Berkeley's AmpLab, recorded at Yahoo in Sunnyvale in December 2012. It walks through Spark's internals: the RDD abstraction, the DAG scheduler, the life of a job, task and worker details, and how to extend or contribute to Spark.
What people said
Recommends tarun · 2019-11 · Stack Overflow · on “How Apache Spark collects and coordinate the results from executors”
The executors process the task and the result sends back to the driver through the cluster manager.It is job of each each worker to sump there partition data and return the result to driver. [LINK] You can also have a go through [this video]
Cites Sathish · 2015-06 · Stack Overflow · on “How DAG works under the covers in RDD?”
Even i have been looking in the web to learn about how spark computes the DAG from the RDD and subsequently executes the task. At high level, when any action is called on the RDD, Spark creates the DAG and submits it to the DAG scheduler. The DAG scheduler divides operators into stages of tasks. A stage is comprised of tasks based on partitions of the input data. The DAG scheduler pipelines operators together. For e.g. Many map operators can be scheduled in a single stage. The final result of a DAG scheduler is a set of stages. The Stages are passed on to the Task Scheduler.The task scheduler launches tasks via cluster manager (Spark Standalone/Yarn/Mesos). The task scheduler doesn't know ab
Cites Sathish · 2015-06 · Stack Overflow · on “How Spark works internally”
Spark revolves around the concept of a resilient distributed dataset (RDD), which is a fault-tolerant collection of elements that can be operated on in parallel. RDDs support two types of operations: transformations, which create a new dataset from an existing one, and actions, which return a value to the driver program after running a computation on the dataset. Spark translates the RDD transformations into something called DAG (Directed Acyclic Graph) and starts the execution, At high level, when any action is called on the RDD, Spark creates the DAG and submits to the DAG scheduler. The DAG scheduler divides operators into stages of tasks. A stage is comprised of tasks based on partitions
Cites maverik · 2018-04 · Stack Overflow · on “What is the best way to collect the Spark job run statistics and save to database”
Spark uses Lazy Evaluation, allowing the engine to optimize RDD transformations at a very granular level. When you execute val DF1= spark.sql("select x,y from A,B ") nothing happens except the transformation is added to the Directed Acyclic Graph. Only when you perform an Action, such as DF1.count, the driver is forced to execute a physical execution plan. This is deferred as far down the chain of RDD transformations as possible. Therefore it is not correct to ask 1.How much time it really took to compute DF1 2.How much time to compute DF2 and at least based on the code examples you provided. Your code did not "compute" val DF1. We may not know how long processing just DF1 took, unless you s