≈ the.bay.news

Presentation: Scaling LLM-Based Ranking Systems for Latency-Critical Search & Recommendation Workloads

InfoQ
Presentation: Scaling LLM-Based Ranking Systems for Latency-Critical Search & Recommendation Workloads
Sundara Raman Ramachandran shares how LinkedIn productionizes LLM ranking under strict P99 latency SLAs. He discusses distilling a 7B model down to 400M parameters, compressing job contexts into single-token embeddings for a 10x throughput leap, and eliminating hidden system bottlenecks - from custom flash attention masks to Python GIL workarounds, gc.freeze, and non-blocking CUDA memory copies.

0 comments

Sign in to join the discussion — your thebay.events account works here.

No comments yet.