This paper introduces NPEFF, a novel interpretability method that decomposes per-example Fisher matrices to reveal the specific strategies large language models use during prediction generation. It provides tools to analyze, mitigate, and study complex model behaviors like unlearning.
Researchers introduce Non-Negative Per-Example Fisher Factorization (NPEFF), an interpretability method designed to uncover the internal processing strategies employed by models. NPEFF achieves this by decomposing per-example Fisher matrices into a set of learned rank-1 positive semi-definite components. Through human evaluation and automated analysis, the study demonstrates that these NPEFF components correspond directly to specific model processing strategies across various language models and text processing tasks.
The work further details how to leverage these components to construct parameter perturbations, allowing users to selectively disrupt a specific processing strategy within the model. The authors demonstrate the utility of NPEFF in analyzing and mitigating the collateral effects of unlearning and studying in-context learning. NPEFF is shown to outperform existing baselines such as gradient clustering and sparse autoencoders for dictionary learning over model activations. The code for this work is publicly released.