โ‰ˆ the.bay.news

llama-server's prompt cache reuses KV computed under a different LoRA scale

DEV Community
llama-server's prompt cache reuses KV computed under a different LoRA scale
TL;DR: llama-server's RAM prompt cache (--cache-ram, on by default) saves a slot's KV when another...

0 comments

Sign in to join the discussion โ€” your thebay.events account works here.

No comments yet.