Expert pruning improves decode throughput up to 1.8 with minimal quality loss
DEV Community
Expert pruning improves decode throughput up to 1.8 with minimal quality loss
Current MoE inference pipelines still stall on expert traffic during batched decoding, limiting...
0 comments
No comments yet.