Anthropic Interview Question: Every Batch Takes 100 ms. Why Is the API Slow?

A worked system design problem about the waiting time your architecture diagram forgot. Let’s say...