Speculative Decoding and MTP: Why Guessing Is Free
DEV Community
Speculative Decoding and MTP: Why Guessing Is Free
A forward pass over five tokens costs about the same as over one. That gap is the entire speedup, and MTP is how the model drafts guesses to fill it.
0 comments
No comments yet.