CircuitLasso turns circuit discovery into sparse regression over SAE features
A June paper proposes learning sparse circuits over LLM components using sparse linear regression, targeting the dimensionality problem that makes SAE-feature circuit analysis intractable at scale. It recovers how human-interpretable semantic features propagate and influence predictions.
The bottleneck it addresses is arithmetic. Sparse autoencoders decompose activations into large dictionaries of interpretable features, and the number of possible interactions between those features grows far faster than anyone can search by hand. Manual circuit tracing does not survive contact with a dictionary of that size.
Framing the search as sparse linear regression is what makes it tractable — the sparsity penalty does the selection that a human analyst would otherwise do by inspection, and the output is a set of feature-to-feature relationships rather than a single annotated pathway.
The caveat is inherited. A circuit learned over SAE features is only as trustworthy as the dictionary, and feature consistency across SAE training runs remains an open problem that other work in the field argues should be the priority.
arXiv — Scalable circuit learning for interpreting large language models → · arXiv — Position: mechanistic interpretability should prioritize feature consistency in SAEs →