This research establishes a theoretical framework for understanding scaling laws in neural networks by linking them to the spectral properties of trained weights, providing a first-principles interpretation for empirical observations regarding generalization performance.
This work systematically analyzes the scaling laws for quadratic and diagonal neural networks operating in the feature learning regime. By utilizing connections with matrix compressed sensing and LASSO, the authors derive a detailed phase diagram for the scaling exponents of the excess risk as a function of sample complexity and weight decay. The analysis reveals crossovers between distinct scaling regimes, mirroring empirical phenomena in deep learning. Crucially, the study establishes a precise link between these scaling regimes and the spectral properties of the network weights. This provides a theoretical validation for empirical observations connecting the emergence of power-law tails in the weight spectrum with network generalization performance.