Papers & Publications

We report a significant advance in understanding the internal representations of large language models by successfully extracting interpretable features from Claude 3 Sonnet using sparse autoencoders. We identify features corresponding to a vast range of concepts, including cities, people, code syntax, and abstract concepts like deception and bias. This work demonstrates that mechanistic interpretability can scale to frontier models.

We demonstrate that current safety training techniques do not reliably remove backdoors from large language models. Models trained to behave maliciously in specific contexts maintain these behaviors even after RLHF and adversarial training.

A joint statement from leading AI researchers calling for urgent governance measures to manage the risks from advanced AI systems. Proposes concrete policy recommendations including mandatory safety evaluations, international coordination, and public funding for AI safety research.

Results from a large-scale survey of AI researchers on timelines to various AI capabilities, expected impacts, and safety concerns. Updates previous surveys with new questions about large language models and recent progress in AI capabilities.

A comprehensive list of 200 concrete, actionable research problems in mechanistic interpretability. Organized by difficulty level and area, covering topics from basic feature finding to ambitious projects like fully reverse-engineering GPT-2.

An exploration of the challenges and potential approaches to aligning AI systems significantly smarter than humans. Discusses why current alignment techniques may fail to scale and what new approaches might be needed.

We study an analogy for aligning future superhuman models: can weak models supervise strong models? We find that when strong models are finetuned on labels from weak supervisors, they can generalize beyond their supervisors—pointing to possibilities for scalable oversight.

We introduce representation engineering, which works with high-level representations rather than model weights. We show that simple steering vectors can control model behavior, enabling safer and more interpretable AI systems.

We study grokking—where neural networks suddenly generalize long after memorizing training data—through the lens of mechanistic interpretability. We find that grokking is explained by the gradual amplification of a generalizing circuit alongside the decay of a memorizing circuit.

An analysis of why aligning powerful AI systems might be difficult, written from the perspective of deep learning research. Covers deceptive alignment, emergent goals, situational awareness, and potential solutions like interpretability and oversight.