They Put 7 Attention Mechanisms on a Latin Square. Then Removed Them One by One.

A 6.59B MoE with seven sequence mixers across 49 layers, and a parameter-matched ablation with eight seeds per arm. Placement changed loss by 0.16%. Removing three of four mechanisms changed nothing. Removing the fourth cost 2.14%.