All posts

Today, we announce that we are making a new whole genome regression method available to the open source bioinformatics community as part of Project Glow. Large cohorts of individuals with paired clinical and genome sequence data enable unprecedented insight into human disease biology. Population studies such as the UK Biobank, Genomics England, or Genome Asia...

This morning at Spark and AI Summit, we announced that Databricks has acquired Redash, the company behind the popular open source project of the same name. With this acquisition, Redash joins Apache Spark, Delta Lake, and MLflow to create a larger and more thriving open source system to give data teams best-in-class tools. I would...

Koalas was first introduced last year to provide data scientists using pandas with a way to scale their existing big data workloads by running them on Apache SparkTM without significantly modifying their code. Today at Spark + AI Summit 2020, we announced the release of Koalas 1.0. It now implements the most commonly used pandas...

Today, we announced Delta Engine, which ties together a 100% Apache Spark-compatible vectorized query engine to take advantage of modern CPU architecture with optimizations to Spark 3.0’s query optimizer and caching capabilities that were launched as part of Databricks Runtime 7.0. Together, these features significantly accelerate query performance on data lakes, especially those enabled by...

At Databricks, we are extremely excited to have Microsoft as a Diamond sponsor of the first virtual Spark + AI Summit. Microsoft and Azure Databricks customers are coming together at Summit. Rohan Kumar, Corporate Vice President of Azure Data, will deliver a keynote on Thursday morning. Additional Microsoft speakers and Azure Databricks practitioners will present...

At Databricks, we are excited to have Amazon Web Services (AWS) as a sponsor of the first virtual Spark + AI Summit. Our work with AWS continues to make Databricks better integrated with other AWS services, making it easier for our customers to drive huge analytics outcomes.   As part of Spark + AI Summit,...

SPARK + AI Summit, the world’s biggest gathering of data and artificial intelligence (AI) professionals, has arrived, and this year’s theme is ‘Data Teams Unite!’  So what better time to announce the finalists for the inaugural Databricks Data Team Awards? The Databricks Data Team Awards celebrate the data teams of engineers, scientists and analysts who...

Recently, a gentleman named Scott Galloway was featured in an article in New York Magazine about how Covid-19 is breaking the higher education business model.  The article led to an invitation to a live  interview with Anderson Cooper (full transcript of that interview can be found here).  When I watched that segment and saw Anderson...

We’re excited to share the announcement of our partnership with Avanade to enable enterprise clients to scale their Azure Data and artificial intelligence (AI) investments and to generate positive business results. In addition to the hundreds of trained Microsoft Azure Databricks specialists, Avanade has a set of solution accelerators to help operationalize data engineering, data...

We’re excited to announce that the Apache SparkTM 3.0.0 release is available on Databricks as part of our new Databricks Runtime 7.0. The 3.0.0 release includes over 3,400 patches and is the culmination of tremendous contributions from the open-source community, bringing major advances in Python and SQL capabilities and a focus on ease of use...