The Softmax Bottleneck and What It Limits

A rank argument about the final layer: why the set of distributions a model can express is bounded by its hidden width, and what that does and does not stop.