Do vision models actually understand charts or just memorize them?

Chart question-answering benchmarks often let models cheat by recognizing familiar patterns rather than truly reading visual data. Chartographer reverses engineer charts into code, then generates variants with identical logic but different visuals—revealing whether models actually understand what they're looking at. Tests on GPT-4V and open-source VLMs show they frequently fail these counterfactual versions despite acing the original, exposing brittle reasoning masked by single-chart scores.