Built a 380M param LLM from scratch (no PyTorch/TF/JAX) and found a nasty residual connection bug via a custom debugger (that works across all major ml frameworks)

Since December I’ve been building a 380M parameter LLM entirely from scratch, using only NumPy and CuPy (no PyTorch, TensorFlow, or JAX). It started as an exercise to understand nlp, the transformer, and ml much more by implementing the components outlined in “Attention Is All You Need,” but it grew into a full training setup with AdamW, gradient accumulation, and batching, all implemented by hand. The whole thing is designed to run on consumer-grade hardware. Recently I increased the vocab siz...