the.bay.news

cached_tokens is 0 because your system prompt isn't stable

DEV Community
cached_tokens is 0 because your system prompt isn't stable
Our OpenAI-compat provider reported cached_tokens 0 of ~16K prompt tokens every turn. The cause was per-turn memory glued into the system prompt. The fix: freeze the prefix, splice context into the message array.

0 comments

Sign in to join the discussion — your thebay.events account works here.

No comments yet.