Databricks Workflows

Unified orchestration for data, analytics and AI on the Data Intelligence Platform

Try for free Watch a demo

Databricks Workflows is a managed orchestration service, fully integrated with the Databricks Data Intelligence Platform. Workflows lets you easily define, manage and monitor multitask workflows for ETL, analytics and machine learning pipelines. With a wide range of supported task types, deep observability capabilities and high reliability, your data teams are empowered to better automate and orchestrate any pipeline and become more productive.

How does it work?

Unified with the Databricks Data Intelligence Platform

Reliability in production

Deep monitoring and observability

Batch and streaming

Efficient compute

Seamless user experience

Unified with the Databricks Data Intelligence Platform

Unlike external orchestration tools, Databricks Workflows is fully integrated with the Databricks Data Intelligence Platform. This means you get native workflow authoring in your workspace and the ability to automate any platform capability including Delta Live Tables pipelines, Databricks notebooks and Databricks SQL queries. With Unity Catalog, you get automated data lineage for every workflow so you stay in control of all your data assets across the organization.

Reliability at scale

Every day, thousands of organizations trust Databricks Workflows to run millions of production workloads across AWS, Azure and GCP with 99.95% uptime. Having a fully managed orchestration tool built into the Data Intelligence Platform means you don’t need to maintain, update or troubleshoot another separate tool for orchestration.

Deep monitoring and observability

Full integration with the Data Intelligence Platform means Databricks Workflows provides you with better observability than any external orchestration tool. Stay in control by getting a full view of every workflow run and set notifications for failures to alert your team via email, Slack, PagerDuty or a custom webhook so you can get ahead of issues quickly and troubleshoot before data consumers are impacted.

Batch and streaming

Databricks Workflows provides you with a single solution to orchestrate tasks in any scenario on the Data Intelligence Platform. Use a scheduled workflow run for recurring jobs that do batch ingestion on preset times or implement real-time data pipelines that run continuously. You can also set a workflow to run when new data is made available using file arrival triggers.

Efficient compute

Orchestrating with Databricks Workflows gives you better price/performance for your automated, production workloads. Get significant cost savings when utilizing automated job clusters that have a lower cost and are only running when a job is scheduled so you don’t pay for unused resources. In addition, shared job clusters let you reuse compute resources for multiple tasks so you can optimize resource utilization.

Seamless user experience

Define workflows in your preferred environment — easily create workflows right in the Databricks workspace UI or using your favorite IDE. Define tasks that use a version-controlled notebook in a Databricks Repo or in a remote Git repository and adhere to DevOps best practices such as CI/CD.