Do language models really understand measurements or just guess?

Language models struggle to compare quantities across different measurement units, especially near decision boundaries where the answer flips with small changes. Researchers found LMs rely on simple numerical and unit-based shortcuts rather than converting both to a common scale. These errors are predictable and systematic, suggesting models lack genuine understanding of measurement equivalence.