Serving Quantized LLMs on NVIDIA H100 Tensor Core GPUs
Quantization is a technique for making machine learning models smaller and faster. We quantize Llama2-70B-Chat, producing an equivalent-quality model that generates 2.2x more...
Page 1
LLM Inference Performance Engineering: Best Practices
In this blog post, the MosaicML engineering team shares best practices for how to capitalize on popular open source large language models (LLMs)...