Mechanistic interpretability gets a consolidating survey — circuits, sparse features, symbolic reasoning
A July overview covers how sparse autoencoders and transcoders decompose activations into interpretable features and works through transformer circuit analysis across the residual stream, attention and induction heads. Consolidation papers appear when a field stops being a frontier and starts being a method.
The scope tells you what has stabilised. Residual stream, attention mechanisms, induction heads, SAEs, transcoders — five years ago that list was a set of open questions. It now reads as a curriculum, which is what a survey is for.
Adjacent work is pushing on the decomposition itself rather than its applications. Cosine-scored sparse autoencoders attack the magnitude sensitivity in standard SAE scoring; binary sparse coding and residual replacement models for vision transformers extend the approach past text.
The honest read is that interpretability now has reliable tools for describing what a network is doing and still limited tools for guaranteeing it will not do something else. Those are different problems, and only one of them has a textbook.
arXiv — Mechanistic interpretability for neural networks: circuits, sparse features and symbolic reasoning → · arXiv — Size doesn't matter: cosine-scored sparse autoencoders → · arXiv — Interpreting vision transformers via residual replacement model →