LIMIT: Less Is More for Instruction Tuning
How should you finetune a large language model for general-purpose question answering? One intriguing approach is that of supervised finetuning on a small...
Page 1
MosaicBERT: Pretraining BERT from Scratch for $20
With the MosaicBERT architecture + training recipe, you can now pretrain a competitive BERT-Base model from scratch on the MosaicML platform for $20...
Efficiently Estimating Pareto Frontiers with Cyclic Learning Rate Schedules
Benchmarking the tradeoff between model accuracy and training time is computationally expensive. Cyclic learning rate schedules can construct a tradeoff curve in a...