Posts by Ali Ghodsi

Back in 2013, we held the first Spark Summit — a gathering of the Apache Spark™ community with leading contributors and production users sharing their wisdom. Since the first event, Spark’s success has accelerated the evolution of data science, data engineering and analytics. As the data community has expanded, we’ve evolved the content and the...

Our commitment to diversity and inclusion is inherent in our company values at Databricks, but recent events and protests around the world have reminded us that there’s much more we can do to bring awareness to racial injustice and drive meaningful change. This year’s theme at Spark + AI Summit was “DATA TEAMS UNITE!” and...

Some brands start out as, well, brands. A lot of work goes into the concept and painting the picture before the business is ever launched. Databricks is different. It always has been and always will be an engineering-led company. Databricks’ model for innovation is inspired by the open-source community. This is where our roots run...

Dear valued customer, As COVID-19 continues to impact our daily lives, we are thinking about those affected as well as the practitioners who are delivering essential services to so many. As we adjust to a new reality, we want you to know that we have never lost focus on you and have a comprehensive plan...

Over the past few years at Databricks, we've seen a new data management paradigm that emerged independently across many customers and use cases: the lakehouse. In this post we describe this new paradigm and its advantages over previous approaches. Data warehouses have a long history in decision support and business intelligence applications. Since its inception...

The AI Dilemma Artificial Intelligence (AI) has massive potential to drive disruptive innovations affecting most enterprises on the planet. However, most enterprises are struggling to succeed with AI . Why is that? Simply put, AI and Data are siloed in different systems and different organizations. Enterprise data is siloed across hundreds of systems such as...

The confluence of cloud, data, and AI is driving unprecedented change. The ability to utilize data and turn it into breakthrough insights is foundational to innovation today. Our goal is to empower organizations to unleash the power of data and reimagine possibilities that will improve our world. To enable this journey, we are excited to...

99% of organizations still struggle to get valuable analytics from their Big Data and achieve the full potential of AI. Today I’m excited to announce a new partnership with Microsoft that represents a major leap forward in achieving Databricks’ mission of making Big Data and AI easy: Databricks will now be offered by Microsoft as...

When we started Databricks, we thought that extracting insights from big data was insanely difficult for no good reason. You almost needed an advanced degree to be able to get any meaningful work done. As a result, only a select few in each organization could ask questions from their big data, the people who set...

Databricks now includes a new feature called Jobs, enabling support for running production pipelines, consisting of standalone Spark applications. Jobs includes a scheduler that enables data scientists and engineers to specify a periodic schedule for their production jobs, which will be executed according to the specified schedule. Notebooks as Jobs In addition to supporting running...