ASSET Seminar: “When do spectral gradient updates help in deep learning?”

Spectral gradient methods, such as the recently popularized Muon algorithm, are a promising alternative to standard Euclidean gradient descent for training deep neural networks and transformers, but it is still unclear in which regimes they are expected to perform better. In this talk, I’ll argue that spectral gradient methods perform well because they are less sensitive to a pervasive type of ill-conditioning in deep learning optimization problems. This ill-conditioning is induced by the activation functions and the data. I’ll support this argument with synthetic regression experiments and NanoGPT-scale language model training.