Welcome Rubicon to Databricks: The Future of AI Storage and Serving Systems

We are incredibly excited to announce that the team behind Rubicon is joining Databricks. Founded by large scale infrastructure builders, Akhil Gupta and Sergei Tsarev, Rubicon is a startup working on building storage systems for AI.

We started Databricks as a company over 10 years ago with the goal to make data and AI applications dramatically easier to build. We soon realized that traditional storage systems such as enterprise data warehouses were not sufficient to process data needed for AI applications. That’s why we built Delta Lake, a core storage foundation for the Lakehouse that combines the best of data warehousing and data lake storage.

With recent innovations in AI like LLMs, there is an even greater need for storage solutions that enable processing diverse types of data and seamlessly integrating both the online stack (for serving) and the offline stack (for training and experimentation). This is reflected in our conversations with customers who have been asking for a simple easy to use platform that enables them to leverage these new exciting technologies. 

Akhil and Sergei are two well known figures in the storage infrastructure space. Akhil began his career at Google almost two decades ago, where he was responsible for the ads infrastructure, which is one of the largest scale use cases of AI on the planet to date. Sergei cofounded Clustrix, one of the first scale-out OLTP database systems for high value, high-transaction applications. The two met at Dropbox, where they helped build, among other things, one of the world’s largest, most efficient, and secure storage systems which served trillions of files and exabytes of data. This included processing immense volumes of unstructured data, e.g. documents, images, and videos, precisely the type of data that customers rely on for AI applications. It is no surprise that given their past experiences, Akhil and Sergei saw the immense potential of AI, as well as the gaps in storage and serving infrastructure needed to support those workloads. This led them to founding a company focused on solving the problem.

"We launched Rubicon.io with the goal of enabling the next generation of storage infrastructure for the AI age. A great developer experience with world class performance at scale is at the core of our mission. We find our vision aligns perfectly with Databricks and we are thrilled that our team will be joining Databricks to deliver an extraordinary experience for developers on the Lakehouse." - Rubicon.io co-founders Akhil Gupta and Sergei Tsarev

When we first met Akhil and Sergei, we quickly realized our shared outlook and the potential for the technologies they have been developing to accelerate our AI infrastructure work.

We are thrilled to bring team Rubicon's pioneering vision into the Databricks Lakehouse Platform. At the upcoming Data + AI Summit, we will unveil how we will integrate Rubicon’s technical approach to dramatically simplify building AI applications. You can still register to join us here.