Why neural networks learn edges before textures

Neural networks exhibit a strong simplicity bias, learning simple features before complex ones. This work analyzes that bias through a Fourier lens, revealing that networks trained on images first exploit amplitude (pixel correlations) before phase information (edges and higher-order structure). Using a synthetic translation-invariant data model with controllable amplitudes and phases, the authors prove that SGD requires at least N³ log² N steps to learn phase-only classification, versus far fewer for amplitude-based tasks. Crucially, power-law spectra—characteristic of natural images—dramatically accelerate phase learning even though they don't directly improve accuracy. Experiments on textures, ImageNet, and CIFAR100 confirm this mechanism, providing mechanistic insight into how deep networks efficiently learn natural image distributions.