Shreshth Rajan, July 2026. · Code: verify_theory.py
A model's plasticity is whether it can still learn something new after you've trained it [6]. I've been studying it through recent work on optimizers. Muon [1] orthogonalizes the weight update, setting every singular value of the update matrix to 1, so the update pushes equally hard in every direction — one instance of the view that each optimizer is just gradient descent under some norm [2]. Muon, in other words, is an 'even push'. I spent the past few weeks testing whether that even push is worth anything, and what actually sets plasticity.
I wanted to know whether Muon's update is even where it counts, once it has passed through the network. So I traced what one update does to a layer's outputs. Take \(y = Wx\) with input second moment \(C = \mathbb{E}[xx^\top]\). An update \(dW\) changes the output by \(dW\,x\), which moves with covariance \(dW\,C\,dW^\top\). Write \(B = dW\,C^{1/2}\) for the operator with that covariance; its singular values are the sizes the next layer actually feels. Substitute Muon's update \(dW = UV^\top\): then \(dW\,C\,dW^\top = U(V^\top C V)U^\top\), and since \(U\) and \(V\) are orthonormal its eigenvalues are exactly those of \(C\), so
$$ \sigma(dW\,C^{1/2}) = \sigma(C^{1/2}). $$So the evenness is gone. The update is flat as a matrix, but the change the network actually feels has the shape of the data. Orthogonalization doesn't do what the story says: if the inputs are low rank, the effective update is low rank too. Muon's update spectrum, on its own, gives no mechanism for plasticity.
If the update's spectrum is the wrong object, the right one should fold the optimizer and the model together. I hypothesized the kernel \(K^\Phi = J\Phi J^\top\), where \(J\) is how the outputs change with the weights and \(\Phi\) is the optimizer's reweighting of the gradient. It earns that role because one small step lowers a probe loss by exactly \(\eta\,r^\top K^\Phi r\), so the number of directions \(K^\Phi\) spreads across is the number of things the model can still learn.
There's a catch. For plain SGD, \(K^\Phi\) is exactly the activation Gram matrix, so it can only echo the representation; only a real preconditioner, like Adam's, reshapes it into anything new. The question is whether that reshaping predicts plasticity better than the representation alone.
I trained a small transformer through a sequence of shifting tasks under AdamW and Muon, both to the same loss, and at 192 checkpoints measured how much it could still learn — the loss drop after a fixed budget of continued training on held-out tasks. Controlling for the current loss, the correlation was 0.69 for the effective rank of \(K^\Phi\), 0.67 for the plain rank of the representation, and 0.76 for its stable rank [5]. The kernel doesn't beat the representation it's built from: the optimizer's reshaping adds nothing beyond what the representation already shows.
These experiments told me plasticity lives in the representation, and the optimizer matters only by shaping it. Recent work has started to map that shaping directly, showing the optimizer sets the representation's rank as the model scales [10].
That's what makes self-training interesting to me. A model learns from its own samples and the representation narrows each round; pushed too far, this becomes model collapse [7], and keeping real data in the mix is the known way to stop it [8]. With a fixed weight penalty you can write the narrowing down exactly: each round multiplies the \(k\)-th direction by \(d_k/(d_k + c) < 1\), so the smallest directions die first and the representation contracts at a rate you can compute [9].
But that factor came from the penalty, not the optimizer. Since the optimizer's implicit bias already picks which solution you converge to [4, 11], spectral in Muon's case [3], it should also pick which directions the contraction keeps. That's the question I want to answer: does the optimizer set the contraction factor, so that the choice of optimizer decides how many rounds of self-training a model survives? I haven't shown it yet, because separating geometry from step size needs scale, and the few experiments I've run so far have collapsed to a triviality.
The identities in the first two notes are checked in one short NumPy file, verify_theory.py, in the repo.