≈ the.bay.news

Same-window official baselines for hidden model degradation on LLM API relays

programming.dev
Same-window official baselines for hidden model degradation on LLM API relays
A relay can return HTTP 200 with ordinary latency and still change what the caller actually gets. Three patterns are hard to see from uptime checks alone: - Silent model substitution: the request names a flagship model, while some traffic is sent to a cheaper model or an older snapshot. - Reduced reasoning budgets: the model family may be unchanged, but reasoning effort is lowered or internal thinking is cut short so the reply arrives early. - Context or output truncation: earlier turns are dropped or output caps are tightened, which shows up as missed constraints or broken multi-step logic. Two common manual checks do not close this gap. Asking a model to name itself is unreliable, because a gateway system prompt or a smaller model’s memorized phrasing can produce a confident identity statement regardless of the weights that answered. Reusing static puzzles is also weak: those items are widely known, and a gateway can cache them or whitelist probe traffic onto a clean route while ordinary production traffic stays degraded. The method therefore pairs two probe families in the same measurement window. Current-events probes use recent public facts, timelines, and time-sensitive constraints, then ask for structured reasoning over that material. Because the prompt context keeps moving, a frozen answer cache or a stale snapshot is easier to expose than with a fixed question bank. Model-fingerprint probes look at habits that differ across model families under multi-step constraints, strict formatting, and structured reasoning paths. Besides whether the final answer looks right, the method records protocol-level signals when they are available: probe agreement across fingerprint checks; input-token and reasoning-token allocation; and returned model-fingerprint fields plus cache-check observations. A route can still produce polished prose after a quiet drop in reasoning effort; agreement and reasoning-token volume often move together in that case. A shift to a different model series can also show up in formatting discipline and fingerprint telemetry even when short answers still pass. Probes alone are not enough for attribution. If agreement falls in one window, the cause might be relay-side drift, an upstream vendor update, transient lab instability, or the baseline difficulty of a newly written current-events task on the official endpoint itself. The method therefore runs the same current-events and fingerprint probes, in the same window, through isolated official API keys that talk to the provider directly and bypass third-party relays. That live official baseline is the comparison point for each route. Measurement is also kept at model × service-group granularity. One provider domain can expose several pools—discount tiers, dynamic routes, and higher-cost pools—with different upstream capacity. A single site-wide average can let one healthy pool hide a degraded one. Each model and service group should be compared to the official baseline on its own. Even then, the signals are early-warning evidence, not a verdict. They should be read across multiple repeated observations, with sample size, consecutive windows, token distributions, and test conditions in view. A single response, or a single low score, does not prove that a model was swapped. Parallel movement with the official baseline points to upstream or task variance; isolated divergence on one model × service-group route is what makes relay-side drift a more specific hypothesis. Folkbench publishes this telemetry on a website feature called Integrity Monitor. That monitor is a web evaluation view. It is not a local detection feature inside a desktop client. The full method write-up, including limits and the same interpretive boundaries, is here: https://folkbench.com/en/blog/how-folkbench-detects-model-degradation [https://folkbench.com/en/blog/how-folkbench-detects-model-degradation]

0 comments

Sign in to join the discussion — your thebay.events account works here.

No comments yet.