Do neural networks forget where they started?

Neural networks begin with a random prior baked into their weights, but training supposedly washes this away. This work measures how much initial bias survives by tracking initialization memory—how much the final predictor depends on initialization scale. On ResNet-9 trained with low-learning-rate SGD, test accuracy swings 26.5 percentage points across different initialization scales even with near-total training accuracy. Adam and larger learning rates with L2 regularization erase this memory entirely. The key insight: generalization-improving regularization works by erasing initialization dependence, not just by constraining models.