RoPE in bfloat16: 8,192 Token Positions Collapsed Into 897
DEV Community
RoPE in bfloat16: 8,192 Token Positions Collapsed Into 897
RoPE in bfloat16 rounds token positions past 256. In my NumPy test, 8,192 positions became 897 and a positional head fell to 3.5%.
0 comments
No comments yet.