The capacity of data grows rapidly in big data area, more and more memory are consumed either in the computation or holding the intermediate data for analytic jobs. For those memory intensive workloads, end-point users have to scale out the computation cluster or extend memory with storage like HDD or SSD to meet the requirement of computing tasks. Intel Optane DC persistent memory (DCPM) brings higher capacity than memory and higher bandwidth & lower latency than storage like SSD or HDD. Extension storage memory with Intel Optane DCPM to cache RDDs, memory intensive workloads benefit from large memory capacity. In this session, we will talk more about how we can use DCPM to extend Spark storage memory as well as its performance advantage.
Xie Qi is a senior architect of Intel Big Data team. He once worked for IT Flags at Intel and joined Intel Big Data team in 2016 and has a broad experience across Big Data, Multi Media and Wireless.