How to stop vision-language models from confidently lying?

Vision-language models confidently answer questions about images even when visual evidence is absent or irrelevant—a failure called mirage. Researchers propose TC-LIA, which inspects how question-relevant information flows through a CLIP vision encoder's layers to decide whether the model should answer or stay silent. Tested across five VQA domains and twelve models, the method reduces mirage rates from 22–67% down to below 3%, critical for medical and document-based applications where false confidence is dangerous.