Finding relevant and related publications is an important task of researchers’ activities. At Mendeley, we have tens of millions of research articles that we try to recommend to millions of researchers, requiring a large scale solution to this problem. Spark’s implementations of recommender systems have recently attracted much attention. In this presentation, we demonstrate how Spark can be used to generate scientific article recommendations for researchers. We share Mendeley’s experiences of moving from other machine learning libraries to Spark, the challenges that we faced and the solutions that we put in place.
Scala, Distributed Computing, Hadoop, Big Data, Spark, Data Mining, Networking, Stochastic Mathematical Modelling