Astronomical Data Processing on the LSST Scale with Apache Spark - Databricks

Astronomical Data Processing on the LSST Scale with Apache Spark

The next decade promises to be exciting for both astronomy and computer science with a number of large-scale astronomical surveys in preparation. One of the most important ones is Large Scale Survey Telescope, or LSST. LSST will produce the first “video” of the deep sky in history by continually scanning the visible sky and taking one 3.2 giga-pixel image every 20 seconds. In this talk we will describe LSST’s unique design and how its image processing pipeline produces catalogs of astronomical objects. To process and quickly cross-match catalog data we built AXS (Astronomy Extensions for Spark), a system based on Apache Spark. We will explain its design and what is behind its great cross-matching performance.



« back
About Petar Zecevic

Petar started out as a Java developer almost 20 years ago, and worked as a Software Architect, Team Leader and IBM software consultant. After switching to the exciting new field of Big Data technologies, he wrote the Spark in Action book (Manning 2016) and these days primarily works on Apache Spark and Big Data projects. Today he is CTO of SV Group in Zagreb, Croatia, while also pursuing his PhD at the University of Zagreb. He is collaborating with Astronomy Department at the University of Washington on building new methods for processing images and data from future astronomical surveys.

About Mario Juric

Mario Juric is a Washington Research Foundation Data Science Professor of Astronomy at the Department of Astronomy at the University of Washington, and a Senior Data Science Fellow of the University of Washington eScience Institute. He is also the Data Management Project Scientist for the Large Synoptic Survey Telescope. He holds a Ph.D. in Astrophysical Sciences from Princeton University; was a postdoctoral member at the Institute for Advanced Study; served as a Hubble Fellow at Harvard University; and was an associate scientist at LSST/AURA.