Comparing Batch and Real-Time LLM Inference Economics

Batch inference costs 50% less than real-time inference at Anthropic, OpenAI and Google, in exchange...