Databricks for Data Engineering

Build Fast and Reliable Data Pipelines

A Cloud-Optimized Platform Powered by Apache Spark™

Databricks Runtime is the core of the Databricks Unified Analytics Platform.
Built on top of a highly-optimized Spark cluster, it increases data processing performance by up to 5x.

Databricks IO

Leverages a vertically integrated stack to optimize the I/O layer and processing layer to significantly improve performance of Spark in the cloud.

Databricks Serverless

A serverless architecture that democratizes infrastructure through the auto-configuration and scaling of compute resources — enabling best-in-class performance at dramatically lower costs.

Fully Managed in the Cloud

A cloud-native platform that abstracts the complexities of big data infrastructure, resulting in a highly elastic, reliable and performant platform to build innovative products.

Databricks Runtime outperforms other compute engines:

5x Faster
Than Vanilla Apache Spark on AWS

Runtime total on 104 queries
(secs — lower is better)

8x Faster
Than Apache Presto on AWS

Runtime geomean on 62 queries
(secs — lower is better)

3x Faster
Than on-premises Impala via Cloudera

Runtime total on 77 impala queries, normalized by cpu cores (secs — lower is better)

Source:Benchmarking Big Data SQL Platforms in the Cloud

Databricks helped us deliver a new feature to market while improving the performance of the data pipeline ten-fold. Today, it powers our entire production pipeline with multi-terabyte Spark clusters.

Chul Lee, Director of Data Engineering & Science at MyFitnessPal

Learn how Databricks can increase the performance of your data pipeline:

Streamline Processes from ETL to Production

Cross-Team Collaboration

Foster collaboration and sharing of insights in real time within and across data engineering, data science, and the business with an interactive workspace.

Production Workflows

A unified platform that streamlines end-to-end workflows from data ingest and ETL, to data exploration and model building, to productionizing models and data products.

Unifying All Analytics

Move seamlessly across various types of analytics including batch, ad hoc, machine learning, deep learning, stream processing, and graph.

Robust Integrations

Plug into a wide variety of AWS tools and data stores with built-in connectors and integrate with other data engineering services to facilitate CI/CD with comprehensive APIs.

Having an agile innovation workflow is critical for McGraw-Hill Education. Databricks Unified Analytics Platform  is at the center of our ecosystem and underpins our innovation pipeline and workflows.

Alfred Essa, VP of Research and Data Science, McGraw-Hill Education

How Databricks streamlined development workflows to improve process efficiencies:

McGraw-Hill improved student pass rates by 13%

Case Study

Notebook Workflows: Implementing Apache Spark Pipelines

Case Study

Using Databricks to implement a CI/CD pipeline

Blog

Protect Enterprise Data on Spark

Strong Data Encryption

Benefit from best-in-class protection at rest and in motion.

Monitor and Auditing

Tap into comprehensive audit logs to monitor and troubleshoot issues.

Unifying All Analytics

Fine-grained management access to every component of the enterprise data infrastructure, including files, clusters, code, application deployments, and dashboards.

Integrated Identity Management

Seamless integration with enterprise identity providers via SAML 2.0 and Active Directory.

Learn how Databricks maintains the highest standard of security:

An overview of Databricks Enterprise Security

Primer

Protecting Enterprise Data on Apache Spark

White Paper

Best Practices for Coarse Grained Data Security

Blog

Managing and Securing Credentials for Apache Spark Jobs

Blog

Our Spark Expertise is our Edge

Expert Support

Unparalleled support by the team that started the Spark research project at UC Berkeley, which later became Apache Spark.

Professional Services

Innovate faster with Databricks and Spark with solution architecting and workload optimization services.

Always Available

Around-the-clock coverage to ensure problems are resolved quickly, with response times as fast as one hour for production tier support.

Technical Resources

Online library of documentation, best practices, user guides, and other technical resources.

Databricks' quality of support and how they've helped our team succeed is absolutely crucial for our business.

Matt Fryer, VP, Chief Data Science Officer, Hotels.com

Get the support you need from our Spark experts:

Learn more about Support

Lower TCO with Smarter Infrastructure Management

Better Performance

Cloud-optimized clusters allow you to complete jobs in a shorter time, reducing cloud compute costs.

Fully Managed Clusters

Further reduce costs by avoiding the time-consuming tasks to build, configure, and maintain complex Spark infrastructure.

Pay for Only What You Use

Billing up to the nearest second keeps your costs down.

Priced for Data Engineering

Lower price point for data engineering production workloads.
See Pricing >

Databricks is our go-to-system for anything requiring deep data processing and analysis. In just a short amount of time, we have been able to increase our data processing speeds by a factor of four without any added operational costs.

Gal Barnea, CTA, Eyeview