vLLM Speculative Decoding: 1.8x Faster Alone, Slower at 32 Users
DEV Community
vLLM Speculative Decoding: 1.8x Faster Alone, Slower at 32 Users
vLLM speculative decoding sped up one user 1.8x and slowed 32 users down. The acceptance rate math, and when to turn it off.
0 comments
No comments yet.