Rate limiting LLM API calls in Node.js: a token bucket for requests and tokens per minute
DEV Community
Rate limiting LLM API calls in Node.js: a token bucket for requests and tokens per minute
LLM APIs limit both requests per minute and tokens per minute. A client-side token-bucket limiter in Node.js that waits for both, keeps callers in order, and corrects its token estimate with real usage. Tested code with a fake clock.
LLM APIs limit both requests per minute and tokens per minute. A client-side token-bucket limiter in Node.js that waits for both, keeps callers in order, and corrects its token…
0 comments
No comments yet.