AI models are prone to generating false information, termed hallucinations, which stem from a structural feature of how they are optimized. Rather than simply being glitches, these hallucinations arise because AI systems, trained to predict likely words, prioritize fluency and confidence over accuracy. Similar to students guessing to avoid leaving questions blank in exams, AI completes gaps in knowledge with plausible-sounding falsehoods. OpenAI's researchers suggest a potential fix: adjusting scoring rules to encourage models to admit when they don't know something. They propose a simple rule: only provide answers if confident to a threshold, such as 90%; otherwise, the model should state it doesn’t know. While this isn’t perfectly feasible with current models, it could lead to improved performance by nudging them towards honesty. Until such adjustments are made, users can mitigate hallucinations by asking for sources, formulating precise queries, verifying answers across models, and maintaining a healthy skepticism regarding the responses. Users are advised to view AI output as draft material rather than final authority.

Source 🔗