Why neural networks hide multiple copies of the same solution

Neural networks are fundamentally redundant: different weight configurations can compute identical functions. This paper completely characterizes that redundancy for three-layer bottleneck networks, showing which symmetries arise from layer composition and which ones actually constrain how gradients flow during training. The result includes an algorithm to check if two parameter sets are functionally equivalent in polynomial time.