CIO Survey: Top 3 Challenges Adopting AI and How to Overcome Them
We recently hosted the webinar — CIO Survey: Enterprise Challenges to AI and How to Overcome Them — featuring Jen Garofalo, Research Director at IDG, the parent company to CIO.com, and Pat McDonough, VP of Customer Success at Databricks. This webinar covered key findings from a recent CIO.com survey of 200 executives on the topic...
Announcing Databricks Runtime 5.0
We’re excited to announce the general availability of Databricks Runtime 5.0. Included in this release is Spark 2.4. This release offers substantial performance increases within key areas of the platform. Benchmarking workloads have shown a 16% improvement in total execution time and Databricks Delta benefits from substantial improvements to metadata caching, improving query latency by...
Simplifying Change Data Capture with Databricks Delta
A common use case that we run into at Databricks is that customers looking to perform change data capture (CDC) from one or many sources into a set of Databricks Delta tables. These sources may be on-premises or in the cloud, operational transactional stores, or data warehouses. The common glue that binds them all is...
Simplify Market Basket Analysis using FP-growth on Databricks
When providing recommendations to shoppers on what to purchase, you are often looking for items that are frequently purchased together (e.g. peanut butter and jelly). A key technique to uncover associations between different items is known as market basket analysis. In your recommendation engine toolbox, the association rules generated by market basket analysis (e.g. if...
Identify Suspicious Behavior in Video with Databricks Runtime for Machine Learning
With the exponential growth of cameras and visual recordings, it is becoming increasingly important to operationalize and automate the process of video identification and categorization. Applications ranging from identifying the correct cat video to visually categorizing objects are becoming more prevalent. With millions of users around the world generating and consuming billions of minutes of...
MLflow On-Demand Webinar and FAQ Now Available!
On August 30th, our team hosted a live webinar—Introducing MLflow: Infrastructure for a complete Machine Learning lifecycle—with Matei Zaharia, Co-Founder and Chief Technologist at Databricks. In this webinar, we walked you through MLflow, a new open source project from Databricks that aims to design an open ML platform where organizations can use any ML library...
Introducing Cluster-scoped Init Scripts
This summer, I worked at Databricks as a software engineering intern on the Clusters team. As part of my internship project, I designed and implemented Cluster-scoped init scripts, improving scalability and ease of use. In this blog, I will discuss various benefits of Cluster-scoped init scripts, followed by my internship experience at Databricks, and the...
Announcing Databricks Runtime 4.3
I'm pleased to announce the release of Databricks Runtime 4.3, powered by Apache Spark. We've packed this release with an assortment of new features, performance improvements, and quality improvements to the platform. We recommend moving to Databricks Runtime 4.3 in order to take advantage of these improvements. In our obsession to continually improve our platform’s...
Building a Real-Time Attribution Pipeline with Databricks Delta
In digital advertising, one of the most important things to be able to deliver to clients is information about how their advertising spend drove results. The more quickly we can provide this, the better. To tie conversions or engagements to the impressions served in an advertising campaign, companies must perform attribution. Attribution can be a...
Loan Risk Analysis with XGBoost and Databricks Runtime for Machine Learning
For companies that make money off of interest on loans held by their customer, it’s always about increasing the bottom line. Being able to assess the risk of loan applications can save a lender the cost of holding too many risky assets. It is the data scientist’s job to run analysis on your customer data...
