Prefix Caching vs Recomputed Prompts: The Hit-Rate Break-Even
DEV Community
Prefix Caching vs Recomputed Prompts: The Hit-Rate Break-Even
Reusing a shared prompt prefix looks like free compute until cache memory, eviction and tenant isolation are added to the per-token bill.
0 comments
No comments yet.