I started my PhD a while ago, and I’ve had a pretty hard time figuring out what I want to do. I’ve worked on a niche research area for two consecutive projects, both accepted to good conferences, and I’m currently working on what I now call a bridge between my past and future. But what is my future? I don’t like people telling me what is suitable for me. Whenever someone strongly suggests a direction, I often go the complete opposite way. You are not me, how dare you!! So if you want me to do something, please don’t say anything, or steer me the opposite way instead lol. Sorry, Mario, you might have been one of the pushing hands here. A PhD is a once-in-a-lifetime opportunity to work hard on something I (might) believe in, and I’m not happy enough with the craft I’ve done to let my past become an anchor for my future. I was completely lost here… So I started talking with friends, sharing what I was doing with people in related areas, who introduced me to others in their networks. The tree kept expanding. I read what they recommended and sometimes reached out to the people who wrote it. Slowly I began to see possible answers to the question I kept asking myself: what could I do that my future self would be proud of?

One especially important connection led me to my collaborator Alan, through our mutual connection Tony. In my current work, I’m pursuing an idea that I sometimes feel I’m the only person who believes in. I have an intuition that this is how it should be, even though no one has done it this way before, and that there must be a way to make it work. Through discussions with Alan, I developed not only ideas for the current project, but also a small glimpse of what I might want to pursue throughout my PhD. Whenever something interests me, I think about it for days and connect everything I encounter back to it. “There Will Be a Scientific Theory of Deep Learning” came onto my radar during this time. At another point, I might not have paid much attention to it, to be honest. But it arrived when I had started asking a broader question: what does it actually mean to understand a learning system, rather than simply make it work? I’m glad I read it, because it gave that glimpse a more concrete shape.

It also left me with a set of questions I could not resolve by reading it again. Jamie Simon, the paper’s first author, has a public booking page, so I booked a slot. That is the whole story of how it happened, which I mention only because I had assumed for a long time that talking to the people whose papers you read required knowing someone. We talked for an hour, me in a 40-degree German summer with all the windows open, him somewhere considerably more reasonable. Our discussion ended when his laptop hit one percent battery. Bits of that conversation are folded into the rest of this post.

What kind of theory are we looking for?

What stayed with me was not simply the paper’s prediction that a scientific theory of deep learning will eventually exist. It was the kind of theory the authors seemed to be asking for. We already have mathematical descriptions of learning algorithms, and an ever-growing collection of empirical observations about trained models. But there remains a large gap between describing an algorithm, observing its outcomes, and explaining the mechanisms through which a particular learning process produces a particular internal structure and behavior.

This paper calls the emerging theory that might fill this gap learning mechanics. The name comes from an analogy with physics. A physical object moves through space under forces, whereas a neural network moves through parameter space under gradient updates. Its motion is shaped by the interaction among the architecture, data, objective, initialization, and learning rule. Unlike many natural systems, none of these ingredients is hidden from us. We can record every weight, activation, gradient, and loss value. The problem is not that the system is inaccessible. The problem is that everything interacts with everything else, at a scale where following every individual component becomes useless. So how do we build a science of something we can observe completely but still cannot understand? The paper does not present a finished theory.

There is a question underneath all of this that I did not have words for until I asked Jamie about it. I had been describing my own frustration: you get a faithful description of what a trained system is doing, and the description is just complicated. Not wrong. Complicated. My supervisor’s worry is that “the structure is genuinely this complex” is both true and unpublishable, because a reviewer will reasonably ask, so what? I expected him to take my side. He did not, and instead made the problem bigger:

Jamie: Maybe there doesn’t exist a simple explanation. It’s a very deep question, and one that has no consensus answer right now. When does something have a human-interpretable explanation? Everyone wants to explain AI, but what kinds of explanations exist, and when? Choosing a question that actually has an answer is the hard part.

This reframed the paper for me. I had been reading the title as a claim about effort. But whether a system admits a compact explanation at all is a property of the system, not of how hard we try. Thermodynamics exists because a gas happens to have a small number of variables that govern its behavior. Nobody arranged that. So the first question is not “what is the theory of deep learning” but “which parts of deep learning are the kind of thing that has a theory”. Some phenomena will compress. Some will turn out to be genuinely high-dimensional, and the honest result there is a negative one.

With that caveat in place, I read the paper as a map of how such a science might grow: start with systems we can solve, learn from what those simplifications leave out, identify macroscopic regularities, and ask which insights survive across different systems.

Start with the systems we can solve

The first answer is a familiar one: begin with a smaller system that mathematics can actually reach. One especially productive simplification is linearization. The paper discusses two forms of it, which sounded similar to me at first but simplify different parts of the problem.

$$ f(x;\theta)=W_LW_{L-1}\cdots W_1x. $$

The resulting model is linear in its input \(x\). However, because the parameters are multiplied together across layers, its learning dynamics remain nonlinear in \(\theta\). These models are obviously much simpler than realistic neural networks, but they still reproduce surprisingly rich learning behavior. For example, Saxe et al. (2013) showed that deep linear networks can learn different singular modes of a task sequentially, with modes associated with larger singular values emerging first. This gives a mathematical account of something that also appears in nonlinear networks: models often learn simple or dominant structure before weaker or more complicated structure. The paper describes this as a greedy low-rank bias. Small initializations, greater depth, stronger mini-batch noise, and explicit regularization can all strengthen this tendency.

$$ f_{\mathrm{lin}}(x;\theta)=f(x;\theta_0)+\nabla_\theta f(x;\theta_0)^\top(\theta-\theta_0). $$

This time, the model remains nonlinear in its input but becomes linear in its parameters around initialization. The resulting training dynamics can be described using the neural tangent kernel, or NTK (Jacot et al., 2018). The representation at initialization determines which functions are easier to learn, and the theory can make quantitative predictions about training dynamics and generalization. This is already useful. It turns observations such as “the model learns simpler patterns first” into something that can be derived and measured. Many ideas that now feel like intuitions or rules of thumb were once phenomena that had to be isolated in simple systems before anyone could explain them.

But linearization also removes something essential. In the NTK regime, the model’s hidden representations barely change during training. The model learns, but it does not really learn new features. That makes the theory tractable, but it also excludes one of the main reasons deep neural networks are interesting in the first place. This is where the story has to move beyond linearization.

From lazy learning to feature learning

The paper describes two qualitatively different ways a wide neural network can behave. In the lazy regime, the network’s internal representations remain close to their initial values. Its output changes, but its features are effectively frozen, so the NTK description works well. In the rich regime, the hidden representations evolve. Neurons can specialize, the geometry of the representation can change, and the model adapts its features to the structure of the data. The asymmetry is not just conceptual. The lazy side is mathematically tractable and largely understood; the rich side is where the interesting behavior lives and where we still have no unified theory.

What I found interesting is that the difference between these regimes does not require changing the entire architecture. It can emerge from how quantities such as initialization and output scale change with width. Scaling down the network output forces the parameters to move further during training, promoting feature learning. Increasing the output scale instead pushes the system toward linearized, lazy behavior (Chizat et al., 2019). The same broad family of models can therefore behave either like a fixed kernel or like a feature-learning system.

Lazy and rich training dynamics under different output multipliers

Figure 1: Lazy and rich training dynamics under different output multipliers. In the rich regime, weights move toward the teacher feature directions, whereas in the lazy regime, they remain close to initialization. Image source: Simon et al., 2026.

The distinction is therefore not simply whether learning succeeds. In both panels the loss goes down; only in the rich regime does learning reorganize the model’s internal representations. This is what makes learning mechanics feel like mechanics: the goal is not only to prove that optimization converges, but to identify the effective variables that determine which kind of learning dynamics we get.

The usefulness and danger of limits

The move to infinite width is part of a more general strategy. When a system has too many interacting components, take an appropriate limit and hope that its aggregate behavior becomes simple. Physics does this all the time. The ideal gas law is derived using an effectively infinite collection of particles but still describes finite amounts of gas remarkably well. Deep learning theory similarly studies limits of width, depth, learning rate, batch size, and initialization scale. These limits can replace an enormous discrete system with a cleaner continuous object. However, taking a limit is not enough by itself. We still need the limiting system to tell us something about the finite system we actually care about. The paper calls the expectation that finite systems behave like their infinite limits plus manageable corrections the discretization hypothesis. Under this view, finite width, finite depth, or finite learning rate introduce quantitative errors, but do not usually change the qualitative structure predicted by the limiting theory.

My reading: What if finiteness is not merely a correction to an infinite system, but part of the phenomenon itself?

Real systems have finite width, finite data, finite compute, and finite training time. These constraints may not only add small errors to an otherwise correct infinite-limit description. They might change the qualitative behavior of the system. If so, should we always begin with a system at infinity and work backward? Or should we sometimes study the constrained system directly? I do not think this makes limits useless. They give us solvable reference points and often reveal variables that we would not otherwise know to measure. The lazy-rich distinction itself is an important example of an insight exposed through infinite width analysis. But I think the transfer from the limiting system back to the real one should itself be treated as a scientific question, rather than an automatic final step. A clean solution at infinity is only useful if we understand what survives the return to finite systems.

I asked him a version of this and got an answer about research practice rather than about mathematics. His approach to choosing a simplification is to start from the complex system and work backwards, boiling things away, but never letting the complex system leave view. There is a chain of approximations all the way down, he said, and at each link you owe yourself the question of what this tells you back about the original.

Jamie: You’ve got to be able to see a path back to the original complex system you want to understand. That path isn’t just going to be there by magic. Your job is to find it, as a trailblazer.

I had been treating “does this simplification transfer?” as a property of the simplification, something to check at the end. He treats it as something you maintain the whole way through, which means that failing to see the path back is information about the model rather than a problem for later. That does not settle whether finiteness is fundamental, but it changes what I do about it: keep the finite system, the one with a real memory limit and a real data budget, in view the entire time, and treat every step away from it as a debt.

From empirical regularities to scientific laws

Not every scientific law begins with a complete microscopic explanation. Sometimes the regularity comes first. Neural scaling laws are one example. Across ranges of model size, dataset size, and compute, test loss often follows surprisingly predictable power laws. On a log-log plot, relationships involving systems with billions of interacting parameters can become approximately linear. These laws do not yet explain why a particular model learns a particular feature. But they show that the aggregate behavior of a complicated learning system can still be captured by a small mathematical relationship. In that sense, prediction can arrive before complete understanding.

scaling law

Figure 2: Neural network loss follows predictable power law relationships with compute, dataset, and parameter count. Image source: Kaplan et al., 2020

Another path toward theory begins not from a fully solvable model, but from regularities the practitioners already observe. Consider learning rate and batch size. Practitioners have long used scaling rules to adjust one when changing the other. For stochastic gradient descent, simultaneously increasing learning rate and batch size can preserve approximately similar training dynamics when the number of processed examples is held fixed. This is commonly known as the linear scaling rule (Goyal et al., 2017). Later theoretical work connected this heuristic to an underlying stochastic differential equation (Li et al., 2019), and to conditions under which the SDE approximation, and therefore the scaling rule itself, actually holds (Li et al., 2021). This does not completely solve neural network training, but it explains why a useful rule works and gives us a language for reasoning about when it might break. The same applies to the critical batch size (McCandlish et al., 2018), which captures a trade-off between two resources. A large batch size can reduce the number of sequential optimization steps but require more total computation. A small batch size can use computation more efficiently while requiring more sequential updates. There is no universally optimal batch size without first saying which resource we care about.

I like these examples because they show how something can move from engineering practice toward scientific explanation. We begin with a pattern that seems to work, isolate the relevant variables, and eventually derive a more general relationship.

A similar idea appears in maximal update parameterization or \(\mu P\). Under standard parameterization, the optimal learning rate can change as model width changes. \(\mu P\) chooses width dependent scaling rules so that features continue to evolve while hyperparameters remain relatively stable. This allows researchers to tune a smaller proxy model and transfer the resulting hyperparameters to a much wider model (Yang et al., 2021). This is theory making contact with real model design. It does not explain every feature learned by the final model, but it removes one source of trial and error.

Maybe a scientific theory of deep learning will not arrive as one grand equation. Maybe it will first appear as a growing collection of local theories that explain individual parts of the system, and only later become connected.

Universality is the real test

Still, understanding one model under one carefully chosen setting is not enough. Deep learning includes different architectures, datasets, objectives, optimizers, and modalities. If every system requires a completely separate explanation, then we have accumulated descriptions, but not yet a general science. This is why I found the paper’s discussion of universality especially important.

Different architectures can reach similar solutions. Different datasets share statistical structure. Representations learned by different models sometimes become more similar as the models become larger and better. For example, the Platonic Representation Hypothesis proposes that models trained on different modalities may converge toward a shared statistical representation of reality (Huh et al., 2024). If a phenomenon repeatedly appears across architectures, initializations, and datasets, then it probably does not depend on every microscopic detail of one particular model. The irrelevant details begin to wash out, leaving a smaller set of things that actually require explanation.

My reading: Universality is not simply another piece of evidence for learning mechanics. It determines whether understanding one model can teach us anything about another.

Of course, the difficult word here is “if”. Representational similarity depends on how representations are compared. Apparent convergence under one metric may disappear under another. Universality should not become an assumption that we impose because we want a clean theory. It is something that has to be demonstrated, together with the conditions under which it holds. But without some form of universality, it is hard to see how insights from a single system can accumulate into a general theory. We ultimately want to understand complex learning systems in general, not only one network trained once. Suppose we completely understand one model, but none of the resulting insight transfers to another architecture, dataset, or training run. That understanding may still be locally useful, but it tells us little about deep learning as a broader phenomenon. Universality is the bridge that might let us move from an explanation of one system to a science of many systems.

Where learning mechanics meets mechanistic interpretability

This brings me to the connection I care about most. Mechanistic interpretability asks what a trained model has learned: its features, circuits, representations, and internal algorithms. Learning mechanics asks how the training process produced them. Simon et al. describe this as a relationship between the biology and the physics of deep learning: interpretability catalogues the organisms, learning mechanics asks what forces produced them. The connection works in both directions.

Bidirectional relationship between lm and mi

Figure 3: The two directions of the symbiosis. Learning mechanics tells interpretability when its working assumptions are earned; interpretability hands learning mechanics the structures that a theory has to account for.

Learning mechanics -> mechanistic interpretability

Learning mechanics could give mechanistic interpretability firmer foundations. Many interpretability methods rely, either explicitly or implicitly, on assumptions (Simon et al., 2026) such as:

  • Linear representability: features correspond to meaningful directions in activation space.
  • Locality: features or circuits can be localized to particular model components.
  • Sparsity: a feature or circuit is relevant to only a relatively small set of inputs.
  • Compositionality: complex computations arise through combinations of simpler mechanisms.

A theory of learning could tell us when these assumptions should hold, when they fail, and how the corresponding structures arise from the interaction among data, architecture, and training dynamics. It could also help move mechanistic interpretability from describing what exists in a trained model toward explaining how it came to exist. Finding an induction head (Olsson et al., 2022) is one problem. Explaining why training produces induction heads, under which conditions they emerge, and whether other models must develop similar mechanisms is another.

Learning mechanics <- mechanistic interpretability

In the other direction, mechanistic interpretability gives learning mechanics concrete phenomena to explain. Induction heads, grokking circuits, sparse features, and learned geometric structures are not merely illustrations added after a theory is complete. They can be the empirical observations from which a theory begins. This matters because much of deep learning theory starts with deliberately simplified data distributions. Mechanistic interpretability instead encounters the structures models learn from realistic data. It can therefore expose phenomena that a theory built entirely from toy settings might never think to ask about.

My reading: Theory does not always have to come first. Sometimes an empirical mechanism gives us the object that a theory should explain.

This two way relationship is what makes the combination exciting to me. We can work forward from training dynamics and ask what mechanisms should emerge. Or we can work backward from an observed mechanism and ask what learning process could have produced it. One direction risks becoming mathematically clean but detached from the system we care about. The other risks producing compelling descriptions without explaining why those structures exist. Together, they might constrain each other.

When I asked where the field actually needs people, Jamie pointed straight at this intersection, and specifically at sparse autoencoders (Cunningham et al., 2023). SAEs are the workhorse of a lot of current interpretability work, and there is no learning-mechanics account of how they train, what they converge to, or when their sparsity assumption is earned rather than imposed. As he put it, mech interp has “been using the tool but hasn’t studied the tool”. The same goes for the more basic questions: what is a feature, what would a toy model of feature recovery look like, how does sparsity interact with training dynamics and generalization.

If you wanted to start here, the foundation he suggested is less exotic than I expected. Math basics and ML basics, beginning with dynamics and the scaling of parameters and width, which is more or less the vocabulary this paper runs on. Physics, not for the analogies but because a physics education teaches you what science is. Experimental basics that nobody teaches explicitly: run many trials, make good plots, keep good notes and records. And on the interpretability side, learn to train SAEs well, precisely because he thinks they are overhyped: you cannot build something better until you know what is there and why it does what it does. I like that order. Skepticism that skips the “learn it properly first” step is just taste. The last piece was a method rather than a subject: pick a few researchers whose work you would like to emulate and try to back out from their papers what they know well.

The part I did not plan to ask about

The technical questions were the ones I had prepared. The one I cared about was whether “understanding as a goal” justifies work whose payoff is unclear. My main supervisor is skeptical, not unkindly: he suggested an easier first project so I would get a win before taking bigger risks. I told him I had four years of not so successful research behind me and already knew what failure looked like. Still, I wanted to hear how someone more senior in the field defends the same bet on an ordinary Tuesday. He did not defend it. He reframed it:

Jamie: As an engineer, and I’m not one, I imagine you’d want a problem you can work on and reliably produce something from. But as a scientist, especially starting a PhD, you want a big mystery you might be able to say something deep about. And this is one.

I had been carrying science versus engineering around as a taxonomy for sorting papers. He uses it to describe what you want from the next few years of your life. Reliable output and a big mystery are both legitimate, and they are not the same job. Put that way, the question stops being “is this valuable” and becomes “which of those do I want”, which I can actually answer. The rest of his case was that this is high-risk and high-reward, that it is not what a big lab will do, and that the payoff really is unclear, so it is not the agenda for anyone who wants to reliably make a medium-sized impact. He reached for thermodynamics as a precedent and then immediately undercut it: history rhymes, but don’t take the analogy too seriously. I appreciated the caveat more than the analogy. The physics framing is seductive precisely because it implies the ending is already known.

The line that stayed with me was about the shape of a PhD itself. No department tells you this, he said, but what they should tell you on your first day is that you are here for four to seven years, that it is a marathon and that the job in the first weeks and months is to seek out the challenge you will accept. The universe has mysteries big and small, and your purpose is to accept a grand one. That is why a PhD is long, and why you are not hired paper by paper. I have never heard the beginning of a PhD described as having that particular content: not producing, not even learning, but choosing which mystery you will spend years failing at. It retroactively explains the months spent feeling lost. That was the work. Nobody tells you.

The other thing I asked about was more practical: how to read. New work appears faster than anyone can keep up with, and I had never found a way of reading that felt like it was building anything. His diagnosis was not about how much time I was spending.

Jamie: Looking for the thing that helps with the problem you’re working on is like trying to find a needle in a haystack. Even if you think you know what you’re looking for, you don’t actually know what a needle looks like yet.

That named the problem better than anything I had told myself. I had been searching, and search only works if you can recognize the target. What he suggested instead is much smaller in scope: one paper a week, searched by person rather than by topic, five minutes to decide whether it interests you at all. Write notes. Then write down the follow-up questions you would ask if the project were yours, take that hypothetical seriously even though you will not do it, and send the good ones to the author. Most weeks that gets you an exchange with a working scientist, and feedback of a kind your advisor cannot give you. He mentioned that he spent his first two years switching topics every fortnight and produced barely one paper, and that this was fine, because what he was accumulating was the ability to have ideas: by year five, everyone around him could only execute them. This is the most reassuring thing anyone has said to me since I started my PhD, and also the most demanding, because the metric it puts me on is one nobody else can see.

Which is where this post comes from. I read a paper, wrote down what I would do if it were mine and what I did not believe, and sent the questions to the author. That is his loop, run exactly once. I am publishing the notes because the next step is to do it again next week.

What I want to carry forward

I do not yet have a precise statement of what I will work on for the rest of my PhD, which would be suspiciously convenient after spending so much time feeling lost lol. But I now have a better sense of the kind of questions I want to ask.

I want to understand not only what is inside a trained model, but how it came to be there. I want empirical observations to suggest the objects worth explaining, and mathematical models to tell us which explanations can generalize beyond one system. I also want to remain suspicious of elegant theories when their assumptions remove the phenomenon we actually care about. That is still too broad to be a research agenda, but it is already more concrete than an empty future.

If I had to name the version of this I keep coming back to, it is about pretraining. Most of the current conversation about what models can do is really a conversation about post-training, but a capability that never takes shape during pretraining is not something post-training can conjure into existence. So: when does a capability become present at all, and what does that transition look like from the inside? It is a question about a training process producing a structure, which puts it on the learning mechanics side, and it needs interpretability’s vocabulary to even say what “present” means, which puts it on the other. I do not know yet whether it is tractable, or whether it is one question or a hundred. But it is the one I keep returning to.

That is what I want to ask. What I am taking from that hour is a standard for how to ask it. Judge the work by its simplicity rather than its performance, and when the simple thing fails, treat why it failed as the finding. Write as a member of a field rather than a competitor in it: Jamie’s version is that a failure paper nobody cites can be more useful than a successful one, and that the papers cited in the long run are the ones that were useful to somebody, even just to their thinking. Pay the debt I mentioned earlier: never let the complicated system out of view while simplifying it. And trust my own sense of what seems fundamental. That last one I already have, at least in its negative form. I opened this post admitting that when someone tells me what suits me, I go the other way. “Don’t let someone convince you something’s important without a fight” is the same reflex, pointed somewhere useful. The positive version, a sense for what actually is fundamental rather than a reflex against other people’s claims, is the part I do not have yet.

Jamie: You have to carry both an unreasonable belief in yourself and a willingness to adapt and change course. You’re trying things provisionally, and you might not follow that course forever. But you think it’s important, you think it’s interesting, and you want to see what’s at the end of the corridor.

I do not know what is at the end of mine yet. But the shape of the question is clearer than it was, and mechanistic interpretability and learning mechanics now feel like two sides of it: what would it take to turn deep learning from something we know how to build into something we scientifically understand?

References