Strava Labs: Exploring a Billion Activity Dataset from Athletes with Apache Spark – Databricks

Strava Labs: Exploring a Billion Activity Dataset from Athletes with Apache Spark

Download Slides

At Strava we have extensively leveraged Apache Spark to explore our data of over a billion activities, from tens of millions of athletes. This talk will be a survey of the more unique and exciting applications: A Global Heatmap gives a ~2 meter resolution density map of one billion runs, rides, and other activities consisting of three trillion GPS points from 17 billion miles of exercise data. The heatmap was rewritten from a non-scalable system into a highly scalable Spark job enabling great gains in speed, cost, and quality. Locally sensitive hashing for GPS traces was used to efficiently cluster 1 billion activities. Additional processes categorize and extract data from each cluster, such as names and statistics. Clustering gives an automated process to extract worldwide geographical patterns of athletes.

Applications include route discovery, recommendation systems, and detection of events and races. A coarse spatiotemporal index of all activity data is stored in Apache Cassandra. Spark streaming jobs maintain this index and compute all space-time intersections (“flybys”) of activities in this index. Intersecting activity pairs are then checked for spatiotemporal correlation, indicated by connected components in the graph of highly correlated pairs form “Group Activities”, creating a social graph of shared activities and workout partners. Data from several hundred thousand runners was used to build an improved model of the relationship between running difficulty and elevation gradient (Grade Adjusted Pace).

Session hashtag: #EntSAIS18



« back
About Drew Robb

Drew is a Staff Infrastructure Engineer at Strava where he has worked since 2013. He is the lead for Strava's experimental Labs project-- https://labs.strava.com. On the infrastructure team he has driven adoption of Apache Spark for processing spatial data, recommendation and content ranking systems, and streaming applications. He studied Physics, Mathematics, and Computer Science at Harvard (BA).