Why does offline feedback training beat expensive reinforcement learning for chatbots?

Training language models from user feedback in multi-turn conversations faces a costly trade-off: online RL handles conversation dynamics but requires expensive trajectory generation, while offline supervised fine-tuning is cheap but drifts from the training distribution. DRIFT sidesteps this by converting the RL objective into importance-weighted supervised learning—sampling trajectories offline, computing return-based weights, then training via standard fine-tuning. Empirically matches or beats RL baselines while maintaining supervised learning's efficiency and simplicity.