Code-level Deep Dive into the optimizations that allowed Spark to win the Daytona GraySort Challenge:

https://databricks.com/blog/2014/10/10/spark-petabyte-sort.html

We’ll discuss the following at a code level:

1) Sort-based Shuffle (less OS resources)

https://issues.apache.org/jira/browse/SPARK-2045

2) Netty-based Network module (epoll, async, ByteBuffer reuse)

https://issues.apache.org/jira/browse/SPARK-2468

3) External Shuffle Service (also allows for auto-scaling of Worker nodes)

https://issues.apache.org/jira/browse/SPARK-3796

4) AlphaSort style cache locality optimizations

http://www.slideshare.net/SparkSummit/deep-dive-into-project-tungsten-josh-rosen (slide 22)

https://issues.apache.org/jira/browse/SPARK-7082

5) https://issues.apache.org/jira/browse/SPARK-9850