article contents:
- introduction: the fundamental challenge of hallucinations
- what are hallucinations and why do they happen?
- the problem with incentives: evaluations encourage guessing
- the proposed solution: modifying evaluation models
- beyond pre-training
- implications for the industry
- conclusion: towards a more honest ai
introduction: the fundamental challenge of hallucinations
a new research paper from openai questions why large language models like gpt-5 and chatbots like chatgpt continue to hallucinate, and whether anything can be done to reduce those hallucinations. openai defines hallucinations as “plausible but false statements generated by language models,” and acknowledges that despite improvements, hallucinations remain “a fundamental challenge for all large language models”—a challenge that will never be completely eliminated.
what are hallucinations and why do they happen?
to illustrate this point, the researchers say that when they asked a “widely used chatbot” about the title of the phd dissertation of adam toman kalai (one of the paper’s authors), they got three different answers, all of them wrong. they then asked for his date of birth and received three different dates, again, all of them wrong. the researchers suggest that hallucinations stem, in part, from the pre-training process, which focuses on making models correctly predict the next word without any right or wrong markers associated with the training data.
the problem with incentives: evaluations encourage guessing
however, the solution proposed in the paper focuses less on the initial pre-training process and more on how large language models are evaluated. the paper argues that current evaluation models do not cause hallucinations themselves, but they “set the wrong incentives.” the researchers compare these evaluations to multiple-choice tests where random guessing makes sense, because “you might get lucky and be right,” while leaving a blank answer “guarantees a zero.”
the proposed solution: modifying evaluation models
the proposed solution, then, is to modify how these evaluations are scored. openai says model evaluations need to “penalize confident errors more than uncertainty, and give partial credit for appropriate expressions of uncertainty.” the researchers argue that it is not enough to introduce “some new uncertainty-aware tests on the side.” instead, “widely used accuracy-based evaluations must be updated so that their scoring method discourages guessing.”
beyond pre-training
this approach is interesting because it shifts the focus from trying to “fix” the model during training to changing the environment in which it is evaluated. if key scoreboards reward lucky guesses, models will continue to learn to guess. by changing the rules of the game, openai hopes to incentivize the development of more cautious and honest models.
implications for the industry
if this approach is widely adopted, it could lead to a major shift in how ai models are measured and compared. instead of focusing solely on raw accuracy metrics, the industry might start evaluating models based on their ability to know when they don’t know something. this could lead to more reliable and safer chatbots that are less likely to mislead users with inaccurate but convincing information.
conclusion: towards a more honest ai
openai’s research paper offers a fresh perspective on the intractable problem of hallucination. by pointing out that bad incentives in the evaluation process may be a big part of the problem, it opens up a potential new path for improvement. building an ai that can confidently say “i don’t know” may be just as important as building an ai that can answer our questions.
