cached_tokens is 0 because your system prompt isn't stable
DEV Community
cached_tokens is 0 because your system prompt isn't stable
Our OpenAI-compat provider reported cached_tokens 0 of ~16K prompt tokens every turn. The cause was per-turn memory glued into the system prompt. The fix: freeze the prefix, splice context into the message array.
0 comments
No comments yet.