Posts by Ben Lorica

As companies ramp up machine learning, the growth in the number of models they have under development begins to impact their set of tools, processes and infrastructure. Machine learning involves data and data pipelines, model training and tuning (i.e., experiments), governance, and specialized tools for deployment, monitoring and observability. About three years ago, we started...

Over the years, technical conferences tend to expand beyond their initial focus, adding new technologies, types of attendees, and a broader range of sessions and speakers. From its original focus on Apache Spark™, the Spark + AI Summit has extended its scope to include not only data engineering and infrastructure, but also machine learning automation...

Over the past few years at Databricks, we've seen a new data management paradigm that emerged independently across many customers and use cases: the lakehouse. In this post we describe this new paradigm and its advantages over previous approaches. Data warehouses have a long history in decision support and business intelligence applications. Since its inception...