All posts

Today we are proud to introduce the Databricks Unified Analytics Platform for Genomics. With a unified platform for genomic data processing, tertiary analytics, and machine learning at massive scale, healthcare and life sciences organizations can accelerate the discovery of life changing treatments and further advancements in personalized and preventative care. The Genomic Data Explosion The...

View the MLflow Spark+AI Summit keynote Everyone who has tried to do machine learning development knows that it is complex. Beyond the usual concerns in the software development, machine learning (ML) development comes with multiple new challenges. At Databricks, we work with hundreds of companies using ML, and we have repeatedly heard the same concerns:...

Databricks is thrilled to announce the Databricks Runtime for Machine Learning, including ready-to-use Machine Learning frameworks, simplified distributed training, and GPU Support. Register for our upcoming webinar to learn more. Today more than ever, data scientists and Machine Learning practitioners have the opportunity to transform their business by implementing sophisticated models for recommendation engines, ads targeting,...

The AI Dilemma Artificial Intelligence (AI) has massive potential to drive disruptive innovations affecting most enterprises on the planet. However, most enterprises are struggling to succeed with AI​. Why is that? Simply put, AI and Data are siloed in different systems and different organizations.​ Enterprise data is siloed across hundreds of systems such as data...

Customers today leverage a variety of data sources to perform complex queries to drive business insight. For example, to better understand user behavior, an analyst might correlate click-through analytics streams from their data warehouse with customer metadata from Salesforce. The Databricks Runtime provides a variety of built-in Apache Spark™ data source connectors (AWS | Azure),...

We have recently shipped the new Databricks Runtime version 4.1 powered by Apache Spark™. Version 4.1 brings improved performance on read/write from sources like S3 or Parquet, improved caching, and a great deal of quality and feature improvements for the preview of Databricks Delta focused on faster query execution and adaptive schema and type validation....

Databricks is thrilled to announce our new optimized autoscaling feature. The new Apache Spark™-aware resource manager leverages Spark shuffle and executor statistics to resize a cluster intelligently, improving resource utilization. When we tested long-running big data workloads, we observed cloud cost savings of up to 30%. What’s the problem with current state-of-the-art autoscaling approaches? Today,...

With GDPR enforcement rapidly approaching (May 25, 2018), many companies are still trying to figure out how to comply.  A big pain point, particularly for companies who utilize data lakes to store vast amounts of data, is how to comply with one of the main requirements under the GDPR - data subject requests, also known...