Databricks SQL Serverless is now available on Google Cloud Platform

Today, we are thrilled to announce that Databricks SQL Serverless is now Generally Available on Google Cloud Platform (GCP)! As a key component of our Data Intelligence Platform, Databricks SQL Serverless delivers the best performance with instant and elastic compute, lowers costs, and frees you to focus on delivering business value rather than managing infrastructure. This GA release reinforces our belief that the best data warehouse is a lakehouse, integrating data lakes and warehouses for unified approach. SQL Serverless is now available in 7 GCP regions and 40+ regions across all three major cloud providers (AWS, Azure and GCP).

Benefits of Databricks SQL Serverless on GCP

Over the last couple of years, we've seen over 4x growth and adoption of Databricks SQL - our data warehouse purpose-built for the Lakehouse. SQL Serverless on GCP brings numerous benefits:

• Instant and elastic compute: SQL Serverless: compute brings a truly elastic environment that's instantly available and scales with your needs. You'll benefit from simple usage-based pricing without worrying about idle time charges. Imagine no longer needing to wait for infrastructure resources to become available to run queries, or no longer overprovisioning resources to handle spikes in usage. Databricks SQL Serverless dynamically grows and shrinks resources to handle whatever workload you throw at it.
• Lower infrastructure costs: Under the covers, the serverless compute platform uses machine learning (ML) algorithms to provision and scale compute resources when you need them. SQL Serverless enables substantial cost savings without the need to shut down clusters manually.
• Eliminate management overhead: Serverless capabilities transform Databricks SQL into a fully managed service, eliminating the burden of capacity management, patching, upgrading, and performance optimization of the cluster. You only need to focus on your data and the insights it holds. The simplified pricing model also means there's only one bill to track and only one place to check costs.
• Improved performance: In addition to instant startup times, SQL Serverless is also packed with performance and cost optimization features for all analytics use cases. SQL Serverless comes with three additional performance-improving features giving it best-in-class performance.
  • Predictive I/O uses machine learning-powered optimizations to make point lookups faster and cheaper and data updates/deletes blazing fast.
  • Intelligent Workload Management dynamically upscales and downscales clusters on a warehouse depending on query load to improve query performance and concurrency while reducing cost. This is especially useful for making the most of your warehouse for BI use cases but also benefits mixed workloads. This capability is only available on SQL Serverless.
    

    

  • Enhanced Query Results Cache retains query results by persisting them as workspace system data. This means that identical queries (e.g. from refreshing a dashboard) sent from different warehouses in the same workspace or even after a warehouse has shut down can be served from the query results cache without requiring to re-run the query. This improves query latency while reducing warehouse cost! This capability is only available on SQL Serverless.

During preview, customer response has been tremendous. Many customers adopted early and look forward to scaling up.

"Our user community is thrilled after we switched to Databricks SQL Serverless on GCP. The instant warehouse start times and improved query performance have significantly boosted developer velocity and adoption of our centralized data platform on Databricks. SQL Serverless was easy to enable and due to dyamic scaling we saw immmediate cost savings of over 50% for a high priority project ... which was great!

— Micaela Christopher, Director of Data Science and Engineering, Uplight

New features launched with SQL Serverless

The GA of SQL Serverless on GCP includes many new features, too. These will make it easier to adopt SQL Serverless for various workloads and improve user experience as well.

1. SQL Execution API: The Databricks SQL Statement Execution API allows you to use standard SQL over HTTP to build custom data applications and integrations with a wide range of applications, technologies, and computing devices. The API provides a set of endpoints that allow you to submit SQL statements to a SQL Warehouse for execution and retrieve results.
   

   

2. Python UDFs: Python UDFs allow users to write Python code and invoke it through a SQL function in an easy secure and fully governed way, bringing the power of Python to Databricks SQL.
   

   

3. Notebooks integration: Use SQL warehouses (Serverless or Pro SQL) directly in Notebooks. SQL-optimized compute provides up to a 12x improved price performance over standard interactive clusters. You can also take advantage of the notebook to write and schedule Git-backed, multi-statement, and parameterized SQL.
   

   

4. Workflows Integration: Use the SQL task type in a Databricks job, allowing you to create, schedule, operate, and monitor workflows that include Databricks SQL objects such as queries, legacy dashboards, and alerts. For example, your workflow can ingest data, prepare the data, perform analysis using Databricks SQL queries, and then display the results in a legacy dashboard.
   

   

5. Databricks AI/BI: A new type of business intelligence product built from the ground up to deeply understand your data's semantics and enable anyone to analyze data for themselves is now also GA on GCP. Additionally, AI/BI Genie, a next generation conversational experience for business teams to engage with their data through natural language is coming to GCP soon.
   

   

6. HIPAA compliance: SQL Serverless is now also HIPAA compliant in all GCP regions it is available in.
7. AI Functions: Leverage LLM and ML models directly in SQL by using powerful AI functions either with built-in LLMs or external models [coming soon]

AI_GENERATE_TEXT(
  prompt,
 'azure_openai/gpt-35-turbo',
 'apiKey', SECRET('tokens', 'azure-openai'),
 "deploymentName", "llmbricks",
 "apiVersion", "2023-03-15-preview",
 "resourceName", "lakehouserules",
 "temperature", CAST(0.0 as DOUBLE)
);

Availability and Getting Started

SQL Serverless on GCP is available in seven regions: us-east1, us-central1, us-west1, us-east4, europe-west2, asia-southeast1, and asia-south1. Getting started with SQL Serverless on GCP is straightforward.

1. If you don't have a Databricks account, sign up for Databricks on GCP.
2. Create a workspace in the Databricks UI and select GCP as your cloud provider.
3. After you're set up, you can begin running SQL queries and experience the power of serverless analytics.

The General Availability of SQL Serverless on GCP is a significant advancement in our mission to simplify data analytics and help organizations unlock the full potential of their data. We are excited to see how our customers will harness this new capability to drive innovation and achieve their business objectives. For more information, visit our documentation on how to get started or contact our support team.

Thank you for being a part of our journey!

For more information, visit our documentation on how to get started or contact our support team.