Platform Blog

This is a guest authored post by JT Vega, Support Engineering Manager, Labelbox. While video games and digital content are a source of entertainment, connecting with others, and fun for many around the world, they are also frequently a destination for toxic behavior that can include flaming, trolling, cyberbullying, and hate speech in the form...

A user-defined function (UDF) is a means for a user to extend the native capabilities of Apache Spark™ SQL. Spark SQL has supported external user-defined functions written in Scala, Java, Python and R programming languages since 1.3.0. While external UDFs are very powerful, they also come with a few caveats: Security. A UDF written in...

Since we launched Databricks on Google Cloud earlier this year, we’ve been thrilled to see stories about the value this joint solution has brought to data teams across the globe. One of our favorite quotes is from Douglas Mettenburg, Vice President Analytics at J. B. Hunt: “Ultimately, Databricks on Google Cloud is now the source...

Thousands of Databricks customers have adopted Databricks Repos since its public preview and have standardized on it for their development and production workflows. Today, we are happy to announce that Databricks Repos is now generally available. Databricks Repos was created to solve a persistent problem for data teams: most tools used by data engineering/machine learning...

Transforming into a data-driven organization – which means data has permeated into every facet of your company – is critical for driving meaningful business outcomes. Data literacy is the new buzzword as organizations across industries focus on keeping up with consumer demands and driving innovation, all while meeting ever-evolving compliance measures. Even for organizations that...

Databricks SQL (DB SQL) is a simple and powerful SQL analytics platform for creating and sharing insights at a fraction of the cost of cloud data warehouses. Data analysts can either connect business intelligence (BI) tools of their choice to SQL endpoints, leverage the built-in analytics capabilities (SQL query editor, visualizations and dashboards), or some...

Discussed code can be found here. This is the first part of a two-part series of blog posts that show how to configure and build end-to-end MLOps solutions on Databricks with notebooks and Repos API. This post presents a CI/CD framework on Databricks, which is based on Notebooks. The pipeline integrates with the Microsoft Azure...

Today, we are excited to announce the public preview of Low Shuffle Merge in Delta Lake, available on AWS, Azure, and Google Cloud. This new and improved MERGE algorithm is substantially faster and provides huge cost savings for our customers, especially with common use cases like updating a small number of rows in a given...

Many IT organizations are familiar with the traditional extract, transform and load (ETL) process - as a series of steps defined to move and transform data from source to traditional data warehouses and data marts for reporting purposes. However, as organizations morph to become more and more data-driven, the vast and various amounts of data,...

Originally announced at Data + AI Summit 2020 Europe, Databricks SQL lets you operate a multi-cloud lakehouse architecture that provides data warehousing performance at data lake economics. Our vision is to give data analysts a simple yet delightful tool for obtaining and sharing insights from their lakehouse using a purpose-built SQL UI and world-class support...