How to stop diffusion models from making up fake details?

Diffusion models sometimes generate implausible details that don't exist in real data—hallucinations that undermine reliability. This work proves that overly smooth score functions cause hallucinations and proposes Variance-Guided Score Modulation to control the score's Jacobian, sharpening the model's learned distribution. On synthetic and real datasets, the method cuts hallucinations by ~25% without sacrificing fidelity or diversity. Code and new benchmark datasets released.