How Sorting Your Prompts Speeds Up a Mixture-of-Experts Batch Job

A mixture-of-experts model reads every expert any request in the batch picked, so a batch of similar prompts reads fewer weights. Sorting prompts by topic before an offline job measured 5 to 17% more throughput on unchanged vLLM.