Weight Decay 0.05: Catastrophe in Adam, Optimum in AdamW
DEV Community
Weight Decay 0.05: Catastrophe in Adam, Optimum in AdamW
The dose-response series reaches the knob I've been inheriting since September: weight_decay=5e-4, copied from SGD folklore into Adam
0 comments
No comments yet.