Circuits at scale, and the dictionary problem underneath

Circuit discovery just became a regression problem instead of a manual search. The result is only as trustworthy as the features it runs over, and those are still not stable.

CircuitLasso frames circuit learning as sparse linear regression over SAE features, recovering how human-interpretable semantic features propagate and influence predictions without the manual tracing that does not survive a large dictionary.

Why the arithmetic forced this

Sparse autoencoders decompose activations into dictionaries of interpretable features, and the number of possible interactions grows far faster than anyone can inspect. Manual circuit tracing produced the field's best results and cannot be scaled by adding researchers.

A sparsity penalty does the selection an analyst would otherwise do by eye. The output is a set of feature-to-feature relationships rather than a single hand-annotated pathway, which is less satisfying to read and vastly more of it.

The inherited caveat

A circuit learned over SAE features is only as sound as the dictionary. Feature consistency across SAE training runs remains an open problem, and there is a live position in the literature arguing it should be the field's priority rather than an implementation detail. If two runs on the same model yield different features, circuits built on either are describing the decomposition as much as the network.

Adjacent work is pushing on the decomposition itself — cosine-scored autoencoders attacking magnitude sensitivity, binary sparse coding, residual replacement models for vision transformers — alongside a consolidating survey covering circuits, sparse features and symbolic reasoning.

What consolidation signals

Survey papers appear when a field stops being a frontier and starts being a method. Residual stream, attention, induction heads, SAEs, transcoders — five years ago that was a list of open questions and now it reads as a curriculum.

The honest position is that interpretability now has reliable tools for describing what a network is doing and limited tools for guaranteeing it will not do something else. Those are different problems. Only one of them has a textbook.

arXiv — Scalable circuit learning for interpreting large language models → · arXiv — Mechanistic interpretability for neural networks: circuits, sparse features and symbolic reasoning → · arXiv — Position: mechanistic interpretability should prioritize feature consistency in SAEs →