Shreshth Rajan, July 2026. ยท Code: verify_theory.py
After you train a model, can it still learn something new? This is defined by the model's plasticity [6], and I've been exploring it through the lens of recent progress in optimizers. Muon [1] does this through orthogonalizing the weight update, setting every singular value of the update matrix to 1, so the update pushes equally hard in every direction. This is one instance of the view that each optimizer is just gradient descent under some norm [2]. In other words, Muon is an 'even push'. I spent the past few weeks testing whether this even push was valuable, and what actually sets plasticity.
I wanted to know whether Muon's update is even where it counts, once it has passed through the network. So I traced what one update does to a layer's outputs. Take \(y = Wx\) with input second moment \(C = \mathbb{E}[xx^\top]\). An update \(dW\) changes the output by \(dW\,x\), which moves with covariance \(dW\,C\,dW^\top\). Write \(B = dW\,C^{1/2}\) for the operator with that covariance; its singular values are the sizes the next layer actually feels. Substitute Muon's update \(dW = UV^\top\): then \(dW\,C\,dW^\top = U(V^\top C V)U^\top\), and since \(U\) and \(V\) are orthonormal its eigenvalues are exactly those of \(C\), so
$$ \sigma(dW\,C^{1/2}) = \sigma(C^{1/2}). $$I found the evenness is gone. The update is flat as a matrix, but the change the network actually feels has the shape of the data. Orthogonalization doesn't do what the story says: if the inputs are low rank, the effective update is low rank too. Muon's update spectrum, on its own, gives no mechanism for plasticity.
If the update's spectrum is the wrong object, the right one should fold the optimizer and the model together. I hypothesized that this was the kernel \(K^\Phi = J\Phi J^\top\), where \(J\) is how the outputs change with the weights and \(\Phi\) is the optimizer's reweighting of the gradient. It earns that role because one small step lowers a probe loss by exactly \(\eta\,r^\top K^\Phi r\), so the number of directions \(K^\Phi\) spreads across is the number of things the model can still learn. There is a catch, though. For plain SGD, \(K^\Phi\) is exactly the activation Gram matrix, so it can only echo the representation; only a real preconditioner, like Adam's, reshapes it into anything new. So the question is whether that reshaping predicts plasticity better than the representation alone. I trained a small transformer through a sequence of shifting tasks under AdamW and Muon, both to the same loss, and at 192 checkpoints measured how much it could still learn: the loss drop after a fixed budget of continued training on held-out tasks. Controlling for the current loss, the correlation was 0.69 for the effective rank of \(K^\Phi\), 0.67 for the plain rank of the representation, and 0.76 for its stable rank [5]. I found the kernel does not beat the representation it is built from. The optimizer's reshaping adds nothing you cannot already read off the representation.
These experiments told me that plasticity lives in the representation, and the optimizer matters only by shaping it; recent work has started to map that shaping directly, showing the optimizer sets the representation's rank as the model scales [10]. This leads to my central curiosity around self-training, where a model learns from its own samples and the representation narrows each round; pushed too far, this becomes model collapse [7], which keeping real data in the mix is the known way to stop [8]. With a fixed weight penalty you can write that narrowing down exactly: each round multiplies the \(k\)-th direction by \(d_k/(d_k + c) < 1\), so the smallest directions die first and the representation contracts at a rate you can compute [9]. But that factor came from the penalty, not the optimizer. Since the optimizer's implicit bias already picks which solution you converge to [4, 11], spectral in Muon's case [3], it should also pick which directions the contraction keeps. That is what I want to answer: does the optimizer set the contraction factor, so that the choice of optimizer decides how many rounds of self-training a model survives? I haven't been able to show this yet because separating geometry from step size needs scale. The few experiments I've run so far have collapsed to a triviality.
The identities in the first two notes are checked in one short NumPy file, verify_theory.py, in the repo.