Natural Gradients: Why Steepest Descent Depends on the Geometry You Choose

Gradient descent looks like a purely mechanical rule: follow the negative gradient downhill. But...