Auditing Token Budget Assumptions Baked Into Chunking Logic
DEV Community
Auditing Token Budget Assumptions Baked Into Chunking Logic
Why a chunker sized in characters or against a stale encoding starts producing oversized chunks after a model change, and how to find and re-derive every budget in the pipeline.
0 comments
No comments yet.