All posts

Discussed code can be found here. This is the first part of a two-part series of blog posts that show how to configure and build end-to-end MLOps solutions on Databricks with notebooks and Repos API. This post presents a CI/CD framework on Databricks, which is based on Notebooks. The pipeline integrates with the Microsoft Azure...

Today, we are excited to announce the public preview of Low Shuffle Merge in Delta Lake, available on AWS, Azure, and Google Cloud. This new and improved MERGE algorithm is substantially faster and provides huge cost savings for our customers, especially with common use cases like updating a small number of rows in a given...

Many IT organizations are familiar with the traditional extract, transform and load (ETL) process - as a series of steps defined to move and transform data from source to traditional data warehouses and data marts for reporting purposes. However, as organizations morph to become more and more data-driven, the vast and various amounts of data,...

Originally announced at Data + AI Summit 2020 Europe, Databricks SQL lets you operate a multi-cloud lakehouse architecture that provides data warehousing performance at data lake economics. Our vision is to give data analysts a simple yet delightful tool for obtaining and sharing insights from their lakehouse using a purpose-built SQL UI and world-class support...

Data powers scientific discovery and innovation. But data is only as good as its data management strategy, the key factor in ensuring data quality, accessibility, and reproducibility of results – all requirements of reliable scientific evidence. As large datasets have become more and more important and accessible to scientists across disciplines, the problems of big...

Today, we are excited to announce the availability of Delta Live Tables (DLT) on Google Cloud. With this launch, enterprises can now use DLT to easily build and deploy SQL and Python pipelines and run ETL workloads directly on their lakehouse on Google Cloud. In order to power analytics, data science and machine learning, data...

Question Index What is a Data Lakehouse? How is a Data Lakehouse different from a Data Warehouse? How is the Data Lakehouse different from a Data Lake? How easy is it for data analysts to use a Data Lakehouse? How do Data Lakehouse systems compare in performance and cost to data warehouses? What data governance...

Over the last decade, security incident and event management tools (SIEMs) have become a standard in enterprise security operations. SIEMs have always had their detractors. But the explosion of cloud footprints is prompting the question, are SIEMs the right strategy in the cloud-scale world? Security leaders from HSBC don’t think so. In a recent talk,...