“Artificial intelligence capabilities” have become the center of widespread debate in the tech world today. In this article, we look at the story of French researcher François Chollet, who aims to uncover the flaws in intelligent models and demonstrate their limitations compared to human intelligence.
Article contents:
- Introduction
- Human mind and artificial intelligence
- ARC-AGI artificial intelligence test
- The debate surrounding artificial general intelligence
- Conclusion
- Frequently asked questions
Introduction
On April 4, 2025 at 12:32 PM ET, writer Matthew Wong published an article titled: “The Man Trying to Prove How Stupid AI Still Is.” The article discusses two of the most prominent figures in the field of artificial intelligence: Sam Altman and François Chollet. They share a single dream of building artificial intelligence models that reach what is called “Artificial General Intelligence” (AGI), meaning matching or exceeding the capabilities of the human mind. However, the two men differ in their views on current achievements, with Altman hinting that his company OpenAI has practically come close to achieving this, while Chollet, a French computer scientist and prominent critic, views this claim as “absolute nonsense.”
When will we get a ChatGPT-like model for the robotics world?
Human mind and artificial intelligence
When Matthew Wong interviewed Chollet earlier this year, Chollet pointed out that AI companies have grown intellectually lazy, claiming their systems are on the verge of technical consciousness or integrated knowledge without providing solid scientific proof. Many companies currently boast about their models’ ability to pass specific tests such as the LSAT, Advanced Placement Biology, or even introductory sommelier wine tests. While these models are impressive, for Chollet they lack the elements of true intelligence, as they are merely models that excel at specific tasks or passing known tests.
Chollet, like Altman and other tech giants, sees the path to true artificial intelligence extending toward grand horizons such as curing diseases, tackling climate change, fighting poverty, and even interstellar travel. However, success in these fields requires AI to demonstrate the ability to innovate, rather than just imitating simple tasks or compiling complex research reports.
Chollet is known for his critical approach, though he does not completely reject artificial intelligence. He contributed significantly to the development of deep learning research and worked as a software engineer at Google from 2015 until last November. Yet, he worries that the industry is straying off course.
ARC-AGI artificial intelligence test
In 2019, he created what he called the “Abstraction and Reasoning Corpus for Artificial General Intelligence,” or ARC-AGI for short, a test designed to expose the differences between intelligence based on memorization, retrieval, and repetition, and the “fluid intelligence” possessed by the human mind.
Chollet defines “fluid intelligence” as the ability to learn unfamiliar skills and solve problems based on fundamental principles, rather than accumulating and reproducing massive amounts of information. In 2022, he launched the “ARC Prize” with a $1 million reward to encourage researchers to advance their results to a higher level of creative thinking.
Tests showed that most humans score between 60 and 70 percent on ARC-AGI, while well-known models such as GPT-3 (which became the basis for ChatGPT) scored 0 percent. Later, the models improved somewhat, but none approached human levels until recently. Chollet believes that while these models are powerful in terms of computing and massive training, they lack the faculty of abstract thought possessed by a human child.
In the ARC-AGI test, models are asked to fill a grid with specific colors according to rules deduced by the tester from previous patterns, such as noticing that blue is always surrounded by orange in each example, and then applying the rule to new patterns. This type of task clearly tests creative deductive intelligence, not just the ability to recall data.
From Microsoft to NVIDIA: AI agents are coming in 2025
The debate surrounding artificial general intelligence
Major AI companies avoided this test because it seemed impervious to their models. Even GPT-4 achieved only a slight improvement when launched in 2023, and in 2024 a version called GPT-4o came out with slightly better capability, but percentages remained modest (around 5 percent). Meanwhile, Anthropic’s Claude 3.7 model did not exceed 14 percent.
However, the turning point came when OpenAI introduced a new model called “o1,” followed by an even more advanced version “o3” in late December, which surprised observers by reaching 87 percent on the ARC-AGI test—a level close to human performance. Chollet described this achievement as “real progress,” noting that the model began blending different modes of thinking in a way that suggests a degree of creativity.
Despite many being enthusiastic about claims that AGI is near, Chollet remains skeptical, pointing out that success on a single test does not mean achieving true intelligence. He resorted to launching a more difficult version of the test called ARC-AGI-2, where model scores dropped significantly, sending companies back to square one on the issue of true creative thinking.
Chollet explained that new models like OpenAI’s o3 took a long time (about 14 minutes per puzzle) and massive computational costs to reach solutions that look like “step-by-step thinking,” rather than the abstract thinking performed by the human brain in seconds. On the other hand, some company executives consider the high cost and lengthy thinking time to be no obstacle as long as it meets market goals.
Conclusion
Ultimately, the concept of “artificial general intelligence” remains controversial because human capabilities themselves are diverse and not precisely defined, let alone the ability to simulate them. Artificial intelligence still has a long way to go to learn how to invent new ideas and skills with flexibility resembling that of the human mind.
Frequently asked questions
1. What test did François Chollet create?
Chollet created a test known as ARC-AGI to measure the ability of models to engage in abstract and deductive thinking rather than simply reproducing stored information.
2. Did GPT-4 pass the test successfully?
GPT-4 scored modestly at first, and subsequent versions like GPT-4o showed slightly better results (around 5 percent), but they did not approach human levels.
3. Why does Chollet consider current models not truly intelligent?
Because he believes most models rely on memorizing and reproducing massive amounts of data without possessing the ability to invent creative solutions to new problems.
4. What is the importance of ARC-AGI-2?
ARC-AGI-2 is a more complex and difficult version designed to test how well a model can achieve true “fluid intelligence” rather than succeeding on narrow tasks.
5. Does passing current tests mean achieving artificial general intelligence?
Not necessarily, as many experts point out that achieving high performance on certain tests does not mean possessing “intelligence” close to human intelligence across various fields.
6. How do AI companies view these tests?
Views vary; some consider them a foundation for developing models toward more creative thinking, while others focus on practical application aspects that generate profits.
7. What is the future of artificial intelligence according to Chollet?
He believes artificial intelligence still has a long road ahead to develop its deductive and creative thinking capabilities, and he sees these tests as driving further progress.
