OpenAI Reveals Causes Behind ChatGPT’s False Information
OpenAI has released new research addressing the reasons behind the phenomenon known as “hallucination” in its popular language model, ChatGPT. This term refers to instances when the model generates false but convincing information. The company identified that the core issue stems from the training and evaluation processes that prioritize guessing over acknowledging uncertainty.
Significance of the Findings
Large language models like ChatGPT are increasingly integrated into critical sectors such as education, healthcare, and customer service. When these models produce “hallucinated” outputs—statements that appear credible but are factually incorrect—they can erode trust and potentially lead to real-world consequences. OpenAI emphasized the importance of resolving this issue as more users rely on these technologies for accurate information.
Despite advancements in developing enhanced models, including the forthcoming GPT-5, hallucinations remain a significant challenge. Research by OpenAI scientists, including Adam Kalai and Santosh Vempala, indicates a need for structural adjustments in training incentives to effectively tackle the problem. The internal definition provided by OpenAI describes hallucinations as “plausible but false statements generated by language models.”
Understanding the Hallucination Issue
The research highlights that during the pretraining phase, language models learn to predict the next word in a sentence using vast amounts of text. However, they are not exposed to which statements are incorrect. This statistical method is adept at generating coherent sentences but struggles with less common facts, such as specific birth dates or publication titles.
When evaluating performance, accuracy often remains the primary focus, creating incentives akin to multiple-choice testing. As a result, models tend to benefit from guessing rather than admitting uncertainty. The researchers noted, “If the main scoreboards keep rewarding lucky guesses, models will keep learning to guess.”
To illustrate this, a comparison was made between two models on a straightforward evaluation test. The newer GPT-5 variant achieved a 52 percent abstention rate and a 26 percent error rate. In contrast, the older model, OpenAI o4-mini, reported a mere 1 percent abstention rate but a staggering 75 percent error rate.
OpenAI’s research paper stated, “At OpenAI, we’re working hard to make AI systems more useful and reliable. Even as language models become more capable, one challenge remains stubbornly hard to fully solve: hallucinations.” The company acknowledged that existing evaluation methods contribute to the issue by creating incentives that discourage honesty about uncertainty.
Looking ahead, OpenAI plans to revise evaluation benchmarks to prioritize uncertainty, aiming to improve the reliability of its models. This shift in focus could help mitigate the risk of hallucinations and enhance the overall performance of future iterations of ChatGPT and similar technologies.
