Databricks
MENUMENU
  • Platform
        • Overview
        • Unified Analytics Platform
          • Collaboration Workspace
          • Delta
          • Runtime or ML
          • Cloud service
        • Genomics
        • Enterprise Security
        • Pricing
        • Why Databricks
        • BY CLOUD ENVIRONMENT
        • Azure
        • AWS
        • BY ROLE
        • Data Engineering
        • Data Science
        • BY TECHNOLOGY
        • Apache Spark
        • TensorFlow
        • MLflow
        • R
        • Snowflake
        • PyTorch
        • Scikit-learn
        • 2019 Registration Now Open: Spark + AI Summit in San Francisco

          REGISTER NOW >

  • Solutions
        • By Use Case
        • Cybersecurity Analytics
        • Deep Learning
        • GDPR
        • Internet of Things
        • Machine Learning & Graph Processing
        • By Industry
        • Advertising & Marketing Technology
        • Energy and Utilities
        • Enterprise Technology & Software
        • Federal Government
        • Financial Services
        • Gaming
        • By Industry Cont.
        • Healthcare & Life Sciences
        • Media & Entertainment
        • Retail & Consumer Packaged Goods
        • Accelerate Discovery with Unified Analytics for Genomics

          LEARN MORE >

  • Customers
  • Learn
        • Dive In
        • Resources
        • Documentation
        • FAQ
        • Forums
        • Certification & Training

        • Training
          • Private Corporate Training
          • Public Training
          • Self-Paced Training
        • Certification
        • Learn Apache Spark Programming, Machine Learning and Data Science, and more

          REGISTER NOW >

  • Partners
        • FEATURED PARTNERS
        • Microsoft
        • AWS
        • RStudio
        • Snowflake
        • Partners
        • Technology
        • Service
        • 2019 Registration Now Open: Spark + AI Summit in San Francisco

          REGISTER NOW >

  • Events
        • Spark + Ai Summit
        • North America 2019
        • Europe 2018
        • Video Archive
        • Other Events
        • Upcoming Local Events
        • Webinars
        • Meetups
        • Global Events
        • Tradeshows
        • WORKSHOPS
        • Data + ML
        • Spark Live
        • Spark + AI Summit Training
        • 2019 Registration Now Open: Spark + AI Summit in San Francisco

          REGISTER NOW >
  • Open Source
        • Apache Spark
        • What is Spark
        • Comparing Spark & Databricks
        • Use Cases
        • Resources
        • Meetups
        • Technical Blogs
        • Other Technologies
        • PyTorch
        • MLflow
        • TensorFlow
        • 2019 Registration Now Open: Spark + AI Summit in San Francisco

          REGISTER NOW >

  • Company
        • Databricks
        • Our Story
        • Careers
        • Leadership
        • Board of Directors
        • Newsroom
        • Blogs
        • All Posts
        • Company Posts
        • Engineering Posts
        • Contact
        • Contact Databricks
        • Join us to help data teams solve the world's toughest problems

          SEE JOBS >
  • Toggle Search
  • Support
  • Contact
  • Log In
  • Try Databricks
Databricks Logo
  • Free Trial
  • Support
  • Log In
  • MENUMENU
    • Platform
          • Overview
          • Unified Analytics Platform
            • Collaboration Workspace
            • Delta
            • Runtime or ML
            • Cloud service
          • Genomics
          • Enterprise Security
          • Pricing
          • Why Databricks
          • BY CLOUD ENVIRONMENT
          • Azure
          • AWS
          • BY ROLE
          • Data Engineering
          • Data Science
          • BY TECHNOLOGY
          • Apache Spark
          • TensorFlow
          • MLflow
          • R
          • Snowflake
          • PyTorch
          • Scikit-learn
          • 2019 Registration Now Open: Spark + AI Summit in San Francisco

            REGISTER NOW >

    • Solutions
          • By Use Case
          • Cybersecurity Analytics
          • Deep Learning
          • GDPR
          • Internet of Things
          • Machine Learning & Graph Processing
          • By Industry
          • Advertising & Marketing Technology
          • Energy and Utilities
          • Enterprise Technology & Software
          • Federal Government
          • Financial Services
          • Gaming
          • By Industry Cont.
          • Healthcare & Life Sciences
          • Media & Entertainment
          • Retail & Consumer Packaged Goods
          • Accelerate Discovery with Unified Analytics for Genomics

            LEARN MORE >

    • Customers
    • Learn
          • Dive In
          • Resources
          • Documentation
          • FAQ
          • Forums
          • Certification & Training

          • Training
            • Private Corporate Training
            • Public Training
            • Self-Paced Training
          • Certification
          • Learn Apache Spark Programming, Machine Learning and Data Science, and more

            REGISTER NOW >

    • Partners
          • FEATURED PARTNERS
          • Microsoft
          • AWS
          • RStudio
          • Snowflake
          • Partners
          • Technology
          • Service
          • 2019 Registration Now Open: Spark + AI Summit in San Francisco

            REGISTER NOW >

    • Events
          • Spark + Ai Summit
          • North America 2019
          • Europe 2018
          • Video Archive
          • Other Events
          • Upcoming Local Events
          • Webinars
          • Meetups
          • Global Events
          • Tradeshows
          • WORKSHOPS
          • Data + ML
          • Spark Live
          • Spark + AI Summit Training
          • 2019 Registration Now Open: Spark + AI Summit in San Francisco

            REGISTER NOW >
    • Open Source
          • Apache Spark
          • What is Spark
          • Comparing Spark & Databricks
          • Use Cases
          • Resources
          • Meetups
          • Technical Blogs
          • Other Technologies
          • PyTorch
          • MLflow
          • TensorFlow
          • 2019 Registration Now Open: Spark + AI Summit in San Francisco

            REGISTER NOW >

    • Company
          • Databricks
          • Our Story
          • Careers
          • Leadership
          • Board of Directors
          • Newsroom
          • Blogs
          • All Posts
          • Company Posts
          • Engineering Posts
          • Contact
          • Contact Databricks
          • Join us to help data teams solve the world's toughest problems

            SEE JOBS >
  • Company Blog
    • Announcements
    • Customers
    • Events
    • Partners
    • Product
    • Security
  • Engineering Blog
    • Apache Spark
    • Ecosystem
    • Machine Learning
    • Platform
    • Streaming
  • See All
Follow @databricks on Twitter
Collapse

Subscribe

  • Blog
  • Newsletter

Follow

  • Follow @databricks on Twitter
  • Follow Databricks on LinkedIn
  • Follow Databricks on Facebook

Accelerating Discovery with a Unified Analytics Platform for Genomics

June 6, 2018 Ion Stoica by Ion Stoica, Frank Austin Nothaft, Ram Sriharsha and Michael Ortega Posted in Company Blog June 6, 2018
  • Share article on Twitter
  • Share article on LinkedIn
  • Share article on Facebook

Today we are proud to introduce the Databricks Unified Analytics Platform for Genomics. With a unified platform for genomic data processing, tertiary analytics, and machine learning at massive scale, healthcare and life sciences organizations can accelerate the discovery of life changing treatments and further advancements in personalized and preventative care.

The Genomic Data Explosion

The first human genome took 13 years and over $2.5 billion to sequence. Today, a human genome can be sequenced in a couple days for less than the price of the latest iPhone. As a result, pharmaceutical and healthcare organizations are profiling the genomes of millions of patients, generating thousands of petabytes of data. To put this in perspective, genomic sequencing is expected to generate up to 40 exabytes of data per year by 2025. This will dwarf the amount of data generated by Youtube and Twitter combined.

When paired with clinical data, genomic data offers huge potential to accelerate drug discovery, predict disease risks, and personalize treatments. This will enable healthcare providers to significantly improve patient outcomes.

The Technology Challenges of Modern Genomic Workflows

The potential to impact care has never been greater. Yet, the tools and systems that genomic researchers rely on do not provide the speed, scale and flexibility needed to draw insight from these massive genomic datasets. These challenges fall into three key areas:

  • Messy sequence data and complex pipelines – a genomic pipeline for processing the petabytes of messy data coming off of a DNA sequencer is typically constructed of 5-10 tools (e.g., GATK, BWA, etc) that are stitched together in up to 40 stages. Building and maintaining these complex pipelines is costly, time consuming, and leads to unnecessary delays in downstream analytics.

  • Antiquated analytics tools – many of the genomics tools used today are command line programs built for single node machines. They were not designed for the cloud and trying to run them as such is incredibly challenging and adds multiple processing steps. Additionally, bioinformatics tools typically have complex dependencies, leading users to package them into monolithic Docker containers, which are deployed via a workflow management system. This makes developing and deploying pipelines labor intensive and time consuming, and makes scaling analytics to current data volumes infeasible. A basic set of queries can take days or weeks to execute. And advanced analytics like cross-cohort joint variant calling and machine learning are simply not possible.
  • Siloed research teams – whether you’re developing a new drug or prescribing a personalized treatment, diverse teams must work together to process, analyze, and interpret insights from genomic and clinical data. The disjointed nature of today’s tools force bioinformaticians, computational biologists, and clinicians to work in silos which further hampers the discovery process.

At Databricks, we’ve seen these problems manifest at organizations across the health ecosystem from pharmaceutical companies to healthcare providers to biotech startups. Based on this experience, we’ve been looking for ways to radically simplify and scale genomic analysis. In short, what if we could enable teams to explore today’s largest genomic datasets using cutting edge analytics and machine learning without any of the wait times? This is exactly what we set out to solve.

Accelerating Discovery with Unified Analytics for Genomics

The Databricks Unified Analytics Platform for Genomics provides the speed and scale bioinformatics teams need to unlock insights buried in their genomic data. We’ve extended the existing capabilities of the Databricks Unified Analytics Platform with genomic-specific toolkits and optimizations to deliver on three specific goals:

  • Simplified genomic pipelines – prebuilt best practice pipelines are provided out-of-the-box ready in a hosted cloud platform. Instead of manually configuring a complex pipeline, all it takes is a few clicks to provision cloud resources, connect to read, variant, and feature data in Azure or AWS and kick-off bulk processing jobs.
  • Interactive tertiary analytics and AI at scale – prepackaged genomic analytics—such as Joint Variant Calling, GWAS, PheWas, eQTL—and machine learning frameworks are provided in a unified platform. Genomic queries are optimized to run in parallel at speeds 60-100x faster than open source tools so you can interactively explore your genomic data at scale.
  • Improved productivity across integrated teams – shared workspaces with detailed revision tracking fosters collaboration across the discovery and diagnosis lifecycle.  Support for SQL, R, Python, Java, and Scala enable bioinformatics teams and data scientists to explore data with their favorite scripting language while interactive dashboards can be deployed making it easy for business users, clinicians and researchers to review findings.

All of these capabilities are built on an Apache SparkTM-optimized engine that improves the performance of genomic tasks by up to 100x. By unifying these capabilities in a single platform we enable healthcare and life sciences organizations to fully leverage their genomic data to reduce drug development timelines and deliver on precision care initiatives.

Getting Started with Unified Analytics for Genomics

The Unified Analytics Platform for Genomics is currently in preview with multiple Databricks customers. The platform will be generally available on Databricks Azure and AWS later this year, but if you’re interested in participating in the preview, please sign up on the product page www.databricks.com/genomics and we will be in touch!

Conclusion

While genomic data has become widely available over the last decade, the processing and downstream analytics required to turn these massive datasets into life changing insight has become the new bottleneck. With the Databricks Unified Analytics Platform for Genomics, we’ve made significant leaps in addressing these challenges. Instead of wrestling with complex pipelines and reducing the scope of research due to rigid tools, healthcare and life sciences organizations can now accelerate critical discovery with a single, collaborative platform for genomic data processing, tertiary analytics and AI at massive scale.

Try it!

  • Sign-up for the Preview
  • Read our Solution Primer

Databricks Blog

Try Databricks for free. Get started today

Read More

New Databricks Delta Features Simplify Data Pipelines

Continued Innovation and Expanded Availability for the Next-gen Unified Analytics Engine Databricks Delta the next generation unified analytics engine, built on top of Apache Spark,...

Databricks Security Advisory: Critical Runc Vulnerability (CVE-2019-5736)

Databricks became aware of a new critical runc vulnerability (CVE-2019-5736) on February 12, 2019 that allows malicious container users to gain root access to the...

A Guide to Developer, Deep Dive, and Continuous Streaming Applications Talks at Spark + AI Summit

In January 2013 when Stephen O’Grady, an analyst at RedMonk, published “The New Kingmakers: How Developers Conquered the World,” the book’s central argument (then and...

See all Company Blog posts

Share Post

  • Share article on Twitter
  • Share article on LinkedIn
  • Share article on Facebook
  • Product
    • Databricks
    • Feature Comparison
    • Pricing
    • Security
    • Documentation
    • FAQ
    • Forums
  • Apache Spark
    • About Apache Spark
    • SparkHub (Community)
    • Developer Resources
    • Certification
    • Instructor-Led Apache Spark Training
  • Solutions
    • Industries
    • Data Science Teams
    • Data Engineering Teams
    • Use Cases
  • Customers
  • Company
    • About Us
    • Leadership
    • Board of Directors
    • Partners
    • Newsroom
    • Careers
    • Contact Us
  • Blog
    • See All
    • Company Blog
    • Engineering Blog
  • Resources

Databricks Inc.
160 Spear Street, 13th Floor
San Francisco, CA 94105
1-866-330-0121

Contact Us

  • Follow @databricks on Twitter
  • Follow Databricks on LinkedIn
  • Follow Databricks on Facebook
  • Databricks Blog RSS feed
  • Follow Databricks on Youtube

© Databricks . All rights reserved. Apache, Apache Spark, Spark and the Spark logo are trademarks of the Apache Software Foundation.
Privacy Policy | Terms of Use