Finding hidden low-dimensional structure before accurate prediction is possible

Many real datasets lie near a low-dimensional subspace, and finding that subspace efficiently is valuable even when full prediction is still out of reach. This work proves that fitting kernel ridge regression and then computing the Average Gradient Outer Product (AGOP) of the fitted model reliably identifies the central subspace of a multi-index polynomial, in a sample regime exponentially smaller than what accurate prediction requires. The result provides a theoretical explanation for why iterative kernel methods like Recursive Feature Machines work well in practice despite limited data. The contribution is primarily theoretical, establishing a formal separation between representation learning and prediction.