The claim that a different architecture would simply be more legible
Localised architectures — lower bandwidth per node, higher expressivity — are argued to be fundamentally more interpretable than deep networks. If true, interpretability is a design choice we already made.
Research on localised machine learning architectures argues that networks with lower bandwidth but higher expressivity per node have the potential to be fundamentally more interpretable than deep neural networks.
The word carrying the argument is fundamentally. Most interpretability work treats opacity as a difficulty to be overcome with better tools — sparse autoencoders, circuit tracing, activation patching, all applied to an architecture taken as given. This says the opacity is a property of that architecture, and a different one would be legible without the tooling.
If that holds, a great deal of current effort is being spent recovering structure that a different design would never have destroyed. Distributed representations are hard to read precisely because meaning is smeared across many units by construction; concentrating it per node makes reading easier for the same reason it makes training harder.
Which is the objection. Distributed representation is not an accident — it is a large part of why these systems generalise, and the field has repeatedly found that architectures constrained for legibility give up capability. Nobody has yet shown a localised architecture reaching frontier performance, and until someone does, "more interpretable" is a claim about a class of models that does not compete.
It remains a useful counterweight to the assumption that we must interpret whatever we happen to have built. The honest position is that the trade has never been measured properly, only asserted from both sides — the same gap visible in the dense-versus-sparse argument playing out in shipped models.
arXiv — Enhancing AI Interpretability and Safety through Localised Architectures → · arXiv — Mechanistic Interpretability for AI Safety: A Review →