Scaling Up Fast: Real-time Image Processing and Analytics using Spark - Databricks

Scaling Up Fast: Real-time Image Processing and Analytics using Spark

Download Slides

Imaging experiments involving complex specimens like full-animals, vascular structures in brain, or cellular material rheology are difficult or impossible to accurately characterize by eye and thus require computationally intensive algorithms to extract meaningful quantitative information from them. With improvements in flux, detector efficiency, and reconstruction algorithms, the rate at which image data is produced using Synchrotron-based X-ray Tomographic Microscopy is staggering. At the TOMCAT beamline of the Swiss Light Source, this rate reaches 8GB per second of image data[1], higher than even multinational companies with dedicated IT staff like Facebook or Instagram usually handle. We have developed a scalable framework based on Apache Spark and the Resilient Distributed Datasets proposed in [2] for parallel, distributed, real-time image processing and quantitative analysis. The cluster-/cloud-based evaluation tool performs filtering, segmentation and shape analysis enabling data exploration and hypothesis testing over millions of structures with the time frame of an experiment. The tools have been tested with clusters containing thousands of machines and images containing more than 100 billion voxels. The flexible infrastructure offers a full spectrum of shape, distribution, and connectivity metrics for cellular materials and networks and can be adapted to a wide variety of new studies requiring high sample counts ranging from long-term dynamics and evolution experiments to investigations of drug-gene interactions. Finally we examine the potential for real-time approximate analysis as compared with region of interest selection and down-sampling as a superior approach for speeding up post-processing.

Additional Reading:

  • A Vision for Making Deep Learning Simple
  • About Kevin Mader

    Kevin Mader studied Electrical Engineering (BSc) and Photonics (MSc) at Boston University and did a PhD in Biomedical Engineering at ETH Zurich and Paul Scherrer Institut. His PhD brought him into the world of big data as it focused on developing large-scale computing solutions for analyzing images of millions of images to better understand genetic components of disease.