Does the order of training data change what language models learn?

Language models are typically trained on shuffled data, leaving their grasp of time-sensitive facts unclear. Researchers trained 6B-parameter models on temporally ordered Common Crawl snapshots versus standard shuffled pre-training, then evaluated them on 7,000+ time-grounded questions. Sequential training matched shuffled baselines on general language skills but consistently retrieved more recent and temporally accurate facts—while shuffled training peaked on older data. Code, checkpoints, and datasets are released.