Aaron joined Facebook’s data platform team four years ago where he designed and implemented core components of Cosco and helped drive it to wide internal adoption. Before Facebook, Aaron studied math and computer science as an undergraduate at Caltech.
Cosco is an efficient and reliable shuffle-as-a-service that powers Spark jobs at Facebook warehouse scale. Cosco is built on in-memory aggregation across a shared pool of distributed memory and provides much more efficient disk usage than Spark's built-in shuffle. In this talk, we present how adding a little flash to Cosco goes a long way in further improving shuffle efficiency: Flash decreases memory usage, and larger write-ahead (aggregation) buffers further help decrease disk IO. We also demonstrate, via careful experiments and analysis, that dynamically leveraging both memory and flash protects flash endurance even for write-once/read-once workloads like shuffle. Finally, the long time-scale at which flash's endurance bottleneck applies allows it to gracefully absorb short-term spikes in workload. We discuss how flash fits into Cosco's architecture and deployment model at Facebook, learnings from deploying at scale in production, and potential future work. We first presented Cosco at Spark+AI Summit 2019.