Databricks Runtime

Databricks Runtime is the set of software artifacts that run on the clusters of machines managed by Databricks. It includes Spark but also adds a number of components and updates that substantially improve the usability, performance, and security of big data analytics. The primary differentiations are:

1. Better Performance with DBIO: The Databricks I/O module, or DBIO, leverages the vertically integrated stack to significantly improve the performance of Spark in the cloud.
2. Stronger Security with DBES: The Databricks Enterprise Security, or DBES, module adds features such as data encryption at rest and in motion, fine-grained data access control, and auditing to satisfy standard compliance (e.g. HIPAA, SOC2) and the most stringent security requirements as one would expect of large enterprises.
3. Significantly lower operational complexity: With features such as auto-scaling of compute resources and local storage, we put Spark on "autopilot" and markedly reduce the operational complexity and management cost.
4. Rapid releases and early access to new features: Compared to upstream open source releases, Databricks' SaaS offering facilitates quicker release cycles, offering our customers the latest features and bug fixes that are not yet available in open source releases.

DATABRICKS RUNTIME OUTPERFORMS OTHER COMPUTE ENGINES