Skip to main content
Page 1
Engineering blog

Offline LLM Evaluation: Step-by-Step GenAI Application Assessment on Databricks

December 14, 2023 by Abe Omorogbe, Liang Zhang, Sunish Sheth, Corey Zumar, Maheswaran Venkatachalam, Emil Lysgaard and Mathias Christiansen in Engineering Blog
Background In an era where Retrieval-Augmented Generation (RAG) is revolutionizing the way we interact with AI-driven applications, ensuring the efficiency and effectiveness of...
Engineering blog

Using MLflow AI Gateway and Llama 2 to Build Generative AI Apps

August 24, 2023 by Kasey Uhlenhuth, Xiangrui Meng, Hagay Lupesko, Sean Owen, Corey Zumar, Liang Zhang, Ina Koleva, Vladimir Kolovski and Arpit Jasapara in Data Science and ML
To build customer support bots, internal knowledge graphs, or Q&A systems, customers often use Retrieval Augmented Generation (RAG) applications which leverage pre-trained models...
Platform blog

Introducing MLflow 2.3: Enhanced with Native LLMOps Support and New Features

April 18, 2023 by Ben Wilson, Harutaka Kawamura, Liang Zhang, Corey Zumar, Jin Zhang and Sunish Sheth in Platform Blog
With over 11 million monthly downloads, MLflow has established itself as the premier platform for end-to-end MLOps, empowering teams of all sizes to...
Engineering blog

How to Build a Scalable Wide and Deep Product Recommender

June 9, 2021 by Stephan Wernz, Liang Zhang and Bryan Smith in Engineering Blog
Download the notebooks referenced throughout this article. I have a favorite coffee shop I've been visiting for years. When I walk in, the...
Engineering blog

Simplify Data Conversion from Apache Spark to TensorFlow and PyTorch

June 16, 2020 by Liang Zhang and Weichen Xu in Engineering Blog
Petastorm is a popular open-source library from Uber that enables single machine or distributed training and evaluation of deep learning models from datasets...
Engineering blog

100x Faster Bridge between Apache Spark and R with User-Defined Functions on Databricks

August 15, 2018 by Liang Zhang and Hossein Falaki in Engineering Blog
SparkR User-Defined Function (UDF) API opens up opportunities for big data workloads running on Apache Spark to embrace R's rich package ecosystem. Some...