the.bay.news

Predicting the Speed of a 276B Model Streamed From an SSD

DEV Community
Predicting the Speed of a 276B Model Streamed From an SSD
A from-config cost model for streaming a mixture-of-experts model off an SSD, validated byte-for-byte against a runtime I didn't write. Then the SSD benchmark turned out to be measuring RAM, and the real run missed the speed prediction by 23×, for the one reason the model had declared it couldn't account for.

0 comments

Sign in to join the discussion — your thebay.events account works here.

No comments yet.