Batch-1 LLM decoding in PyTorch + Triton, approximately at parity with llama.cpp on an RTX 4060: looking for independent reproductions

Problem. Batch-1 decoding is bound by weight bandwidth, but Hugging Face generate() also leaves the GPU idle between kernels. On Qwen3-1.7B with my int8 kernel, the GPU idled about 2 ms per token. Approach. A weight-only int8 Triton GEMV (a torch.library custom op), plus a sync-free decoder that replays one CUDA graph per token, streaming tokens through pinned memory. Measured steps: 91.3 tok/s with generate() → 96.8 with the sync-free CUDA-graph loop → 105.1 with the int8 kernel configuration ...