Serverless Compute for Notebooks, Workflows and Pipelines is now Generally Available on Google Cloud

In the rapidly evolving landscape of data engineering and analytics, speed, scalability, and simplicity are invaluable. Serverless compute addresses these needs by eliminating the complexity of managing infrastructure, allowing you to focus on building impactful data-driven solutions. We’re thrilled to announce serverless compute for notebooks, Workflows, and DLT pipelines is now generally available on Google Cloud. Thus, any user can benefit from distributed compute across all major cloud platforms without having to manage complex infrastructure.

What is Databricks Serverless Compute?

Databricks serverless compute is a scalable and secure execution layer that simplifies running notebooks, Workflows, and Delta Live Tables (DLT) pipelines by eliminating the need to set up, optimize, and manage clusters. With serverless compute, resources are provisioned automatically, so you can:

• Reduce operational overhead: No need to manually configure or manage clusters.
• Optimize performance or costs: Only pay for compute when it’s assigned to a workload. Serverless currently optimizes for performance. Soon, you will also be able to select a cost-optimized mode, which automatically optimizes infrastructure to reduce cost.
• Improve productivity: Scale workloads effortlessly while focusing on solving business problems.

For detailed guidance on taking advantage of serverless compute, explore our best practices for serverless compute.

Monitoring Costs and Usage

Serverless compute’s pay-as-you-go model is cost-efficient, but there are also some easy ways to monitor cost and usage metrics so you can control spend on Google Cloud:

• System tables: Query detailed usage data to identify high-cost workloads and users.
• AI/BI Cost Dashboard: Visualize usage trends by workspace, SKU, or tag, and download reports for further analysis.
• Alerts: Set up budget thresholds to receive notifications when spending exceeds your limits.
• Budget policies: Enforce tag requirements for your workloads.

For detailed instructions, check out our guides on monitoring serverless compute costs and using budgets to monitor account spending.

“Serverless DLT Pipelines on Google Cloud have redefined our approach at Uplight, enabling us to run ETL workloads more than twice as fast while keeping costs down. The ease of use, auto-optimization, and efficiency of serverless compute makes scaling more manageable and allow us to prioritize delivering value to our customers.”

— Micaela Christopher, Director of Data Science & Engineering, Uplight

Google Cloud is a close partner of Databricks. Collaboration across engineering teams has resulted in tight integration between Google Cloud’s Compute Engine and Hyperdisk and the Databricks Data Intelligence Platform. The result is a highly performant and efficient infrastructure for data and AI workloads. Databricks customers can realize these benefits on Google Cloud whether they’re using Serverless or Classic Compute. We’re looking forward to following up on Databricks Serverless availability on Google Cloud with full support for Databricks Classic on advanced Google Axion Processors and C4A VMs in the coming weeks.

“We are excited to bring Databricks Serverless Compute to Google Cloud, which underscores our commitment to  open standards. Through our partnership, we're committed to providing customers with the flexibility to choose the tools they need to accelerate innovation with AI, including access to Google’s advanced Axion processors and C4A virtual machines.” 

— Ritika Suri, Managing Director, AI & Data Partnerships at Google Cloud.

Unlock the Power of Serverless Compute Today

Databricks serverless compute is more than just an infrastructure solution; it’s a productivity enhancer that enables data teams to innovate faster. By following best practices for migration, optimizing ingestion, and monitoring costs, you can fully leverage the potential of serverless computing while keeping budgets in check.

As an account admin, you can easily enable serverless compute for Workflows, notebooks, and DLT pipelines under Settings → Feature Enablement as shown in the screenshot below.

Databricks Serverless Compute on Google Cloud delivers speed, scalability, and simplicity on a secure, AI-ready infrastructure. Leveraging Google's AI-first approach and two decades of AI innovation, it provides access to a global infrastructure optimized for security and AI, enabling powerful synergies between AI/ML systems and data storage. This platform enables businesses to eliminate data silos, leverage deep learning for optimization, and achieve key strategic outcomes. To learn more, dive deeper into our documentation to enable serverless compute and explore its powerful features.