Top 5 Mistakes When Writing Spark Applications
Why Stack Overflow recommends it
Stack Overflow users link it when answering Spark performance and configuration questions, such as skewed joins, partition sizing, and classpath or jar conflicts that call for shading.
Recommended by 4 people across 4 different months, 2016 to 2020.
What it is
A 30-minute Spark Summit 2016 talk on the most common mistakes people make when writing Apache Spark applications. It covers practical issues such as sizing executors, memory and partitions, shuffle and data skew, and dependency conflicts.
What people said
Recommends taotao.li · 2016-03 · Stack Overflow · on “Spark code organization and best practices”
I think you can subscribe Apache Spark , databricks channel on youtube, listen more and know more, especially for the experiences and lessons from others. Apache Spark [LINK] databricks [LINK] Spark Technology Center [LINK] here is some videos recommended: SparkUI Visualization [LINK] slide SparkUI Visualization [LINK] Spark in Production: Lessons from 100+ Production Users [LINK] slide Spark in Production: Lessons from 100+ Production Users [LINK] Spark Tuning for Enterprise System Administrators [LINK] Spark Tuning for Enterprise System Administrators [LINK] Building, Debugging, and Tuning Spark Machine Learning Pipelines - Joseph Bradley (Databricks) [LINK] slide Building, Debugging, and
Recommends Biplob Biswas · 2018-04 · Stack Overflow · on “Error while using "spark.executor.userClassPathFirst" configuration”
We ended up solving this by shading our Jackson dependency in our pom. You can check this video where they talk about exactly this problem : [this video]
Recommends Sahil J · 2018-06 · Stack Overflow · on “PySpark: How to speed up sqlContext.read.json?”
The short answer is : It depends (on the underlying infrastructure ) and the distribution within data (called the skew which only applies when you're performing anything that causes a shuffle). If the code you posted is being run on say: AWS' EMR or MapR, it's best to optimize the number of executors on each cluster node such that the number of cores per executor is from three to five. This number is important from the point of reading and writing to S3. Another possible reason, behind the slowness, can be the dreaded corporate proxy. If all your requests to the S3 service are being routed via a corporate proxy, then the latter is going to be huge bottleneck. It's best to bypass proxy via th
Recommends abiratsis · 2020-02 · Stack Overflow · on “Dilemma about Spark partitions”
Here are some resources that may help understanding partitioning Top 5 Mistakes When Writing Spark Apps [this video] , Tune Apache Spark Jobs [LINK] , Executors, Cores and Memory [LINK] and the last one [LINK] . By the way you should avoid posting 5 question or more question in one
Cites John Engelhart · 2016-08 · Stack Overflow · on “Apache Spark Handling Skewed Data”
I have two tables I would like to join together. One of them has a very bad skew of data. This is causing my spark job to not run in parallel as a majority of the work is done on one partition. I have heard and read and tried to implement salting my keys to increase the distribution. [this video] at 12:45 seconds is exactly what I would like to do. Any help or tips would be appreciated. Thanks!
Cites rdeboo · 2017-02 · Stack Overflow · on “Ignore Spark Cluster Own Jars”
You can try to use the Maven shade plugin to relocate the conflicting packages. This creates a separate namespace for the newer version of the mllib jar. So both the old and the new version will be on the classpath, but since the new version has an alternative name you can refer to the newer package explicitly. Have a look at [LINK] : If the uber JAR is reused as a dependency of some other project, directly including classes from the artifact's dependencies in the uber JAR can cause class loading conflicts due to duplicate classes on the class path. To address this issue, one can relocate the classes which get included in the shaded artifact in order to create a private copy of their bytecod
Cites ravee · 2018-10 · Stack Overflow · on “Partitioning a large binary file using pyspark”
Thanks for the replies. Basically it is a tar file which contains some bunch of medical images called dicom(it is a standard medical imaging format) files. As it is a binary file, I guess single file cannot be partitioned right? And from this link, got to know that a block cannot be more than 2GB ( [this video] ). I am using python plugins for reading tar and dicom file.
Cites Saawan · 2020-04 · Stack Overflow · on “How to override spark jars while running spark-submit command in cluster mode? (okhttp3)”
I have solved the issue using maven shade plugin. Ignore Spark Cluster Own Jars [LINK] Reference video: [this video] I followed answer given here and added the following. Even in the source code for SparkSubmit , you will see jar getting appended to total jar list if we give --jar , so it will never override with those options but it will add jar. [LINK] <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-shade-plugin</artifactId> <executions> <execution> <phase>package</phase> <goals> <goal>shade</goal> </goals> <configuration> <relocations> <relocation> <pattern>okio</pattern> <shadedPattern>com.shaded.okio</shadedPattern> </relocation> <relocation> <pattern>okhttp3</pattern>