Improving Apache Spark with S3 - Ryan Blue
Why Stack Overflow recommends it
Stack Overflow users link it when explaining why Spark jobs writing directly to S3 can fail or silently produce bad results because of rename-based commits and listing inconsistency.
Recommended by 1 people across 1 different months, 2017 to 2018.
What it is
A conference talk by Ryan Blue on running Apache Spark against Amazon S3, covering why S3 behaves differently from a real filesystem and what that means for committing job output. It is a Spark Summit session published by Databricks.
What people said
Recommends stevel · 2018-06 · Stack Overflow · on “Spark throws java.io.IOException: Failed to rename when saving part-xxxxx.gz”
It's not safe to use S3 as a direct destination of work without a "consistency layer" (Consistent EMR, or from the Apache Hadoop project itself, S3Guard), or a Special output committer designed explicitly for work with S3 (Hadoop 3.1+ "the S3A committers"). Rename is where things fail, as listing inconsistency means that the scan for files to copy may miss data, or find deleted files which it can't rename. Your stack trace looks exactly how I'd expect this to surface: job commits failing apparently at random. Rather than go into the details, here's a video of Ryan Blue on the topic [this video] Workaround: write to your local cluster FS then use distcp to upload to S3. PS: for Hadoop 2.7+, switch
Cites Wade Jensen · 2017-09 · Stack Overflow · on “Setting spark.speculation in Spark 2.1.0 while writing to s3”
Update: If you use AWS Elastic MapReduce, clusters with version >= 5.19 can now safely use speculative execution, but your Spark job can still fail part-way through and leave incomplete results . The partial data from your incomplete results are queryable if you are directly scanning AWS S3 which can lead to incorrect results for downstream jobs, so you need a strategy to deal with that! If your are running Spark 2.3.0 or greater I would recommend writing new partitions to a deterministic location using SaveMode.Overwrite and retrying on failure, this will avoid duplicate or corrupt data in your output. If you are using SaveMode.Append then retrying a Spark job will produce duplicate data in
Cites stevel · 2017-09 · Stack Overflow · on “Writing Spark dataframe as parquet to S3 without creating a _temporary folder”
You can't eliminate the _temporary file as that's used to keep the intermediate work of a query hidden until it's complete But that's OK, as this isn't the problem. The problem is that the output committer gets a bit confused trying to write to the root directory (can't delete it, see) You need to write to a subdirectory under a bucket, with a full prefix. e.g. s3a://mybucket/work/out . I should add that trying to commit data to S3A is not reliable, precisely because of the way it mimics rename() by what is something like ls -rlf src | xargs -p8 -I% "cp % dst/% && rm %" . Because ls has delayed consistency on S3, it can miss newly created files, so not copy them. See: Improving Apache Spark
Cites stevel · 2017-11 · Stack Overflow · on “Is Namenode still necessary if I use S3 instead of HDFS?”
No, provided you have a means to deal with the fact that S3 lacks the consistency needed by the shipping work committers. Every so often, if S3's listings are inconsistent enough, your results will be invalid and you won't even notice . Different suppliers of Spark on AWS solve this in their own way. If you are using ASF spark, there is nothing bundled which can do this. [this video]
Cites stevel · 2017-12 · Stack Overflow · on “Trying to read and write parquet files from s3 with local spark”
I mean, you cannot use S3 as a direct destinatiion of work because the code to commit work assumes it is a filesystem with things like rename(), and really its an object store which needs its own commit protocol. Reading files: fine, but direct writes to S3. Danger. See [this video]
Cites Kiwy · 2018-11 · Stack Overflow · on “Writing Spark dataframe as parquet to S3 without creating a _temporary folder”
@SteveLoughran Hello Steeve, I've just see this conference [this video] where they thank you by the way, is there an better way 1 year after or is it still the same issue ?