Skip to main content
Page 1

Offline LLM Evaluation: Step-by-Step GenAI Application Assessment on Databricks

December 13, 2023 by Abe Omorogbe, Liang Zhang, Sunish Sheth, Corey Zumar, Maheswaran Venkatachalam, Emil Lysgaard and Mathias Christiansen in Engineering Blog
Background In an era where Retrieval-Augmented Generation (RAG) is revolutionizing the way we interact with AI-driven applications, ensuring the efficiency and effectiveness of...

Using MLflow AI Gateway and Llama 2 to Build Generative AI Apps

August 23, 2023 by Kasey Uhlenhuth, Xiangrui Meng, Hagay Lupesko, Sean Owen, Corey Zumar, Liang Zhang, Ina Koleva, Vladimir Kolovski and Arpit Jasapara in Data Science and ML
To build customer support bots, internal knowledge graphs, or Q&A systems, customers often use Retrieval Augmented Generation (RAG) applications which leverage pre-trained models...

Introducing MLflow 2.3: Enhanced with Native LLMOps Support and New Features

April 17, 2023 by Ben Wilson, Harutaka Kawamura, Liang Zhang, Corey Zumar, Jin Zhang and Sunish Sheth in Platform Blog
With over 11 million monthly downloads, MLflow has established itself as the premier platform for end-to-end MLOps, empowering teams of all sizes to...

How to Build a Scalable Wide and Deep Product Recommender

June 9, 2021 by Stephan Wernz, Liang Zhang and Bryan Smith in Platform Blog
Download the notebooks referenced throughout this article. I have a favorite coffee shop I've been visiting for years. When I walk in, the...

Simplify Data Conversion from Apache Spark to TensorFlow and PyTorch

June 16, 2020 by Liang Zhang and Weichen Xu in Engineering Blog
Petastorm is a popular open-source library from Uber that enables single machine or distributed training and evaluation of deep learning models from datasets...

100x Faster Bridge between Apache Spark and R with User-Defined Functions on Databricks

August 15, 2018 by Liang Zhang and Hossein Falaki in Engineering Blog
SparkR User-Defined Function (UDF) API opens up opportunities for big data workloads running on Apache Spark to embrace R's rich package ecosystem. Some...