Revolutionizing AI Intelligence Testing: François Chollet’s New Nonprofit
Imagine a world where artificial intelligence can perform most tasks humans can. That’s the aim of François Chollet, a former Google engineer, who is co-founding a nonprofit to create benchmarks for testing AI’s “human-level” intelligence. This new venture, the ARC Prize Foundation, will be led by Greg Kamradt, an ex-Salesforce engineering director and founder of AI product studio Leverage.
Kamradt will act as the foundation’s president and board member. In a post on the nonprofit’s website, Chollet expressed their vision to steer artificial general intelligence (AGI) development. AGI is often seen as AI that can handle a wide array of human tasks.
“We’re growing into a proper nonprofit foundation to act as a useful north star toward artificial general intelligence,”
{François Chollet}
The ARC Prize Foundation plans to build on ARC-AGI, a test developed by Chollet to assess whether AI systems can learn new skills beyond their training data. This test uses puzzle-like problems that challenge AI to adapt and solve unforeseen issues.
- ARC-AGI stands for “Abstract and Reasoning Corpus for Artificial General Intelligence.”
- Many AI models struggle with this test, solving less than a third of the tasks.
In 2019, Chollet introduced ARC-AGI to identify gaps in AI’s ability to think like humans. Unlike other benchmarks that measure AI risk through superhuman questions, ARC-AGI focuses on closing the gap between human and AI capabilities.
“Future versions of the ARC-AGI benchmark will focus on shrinking [the human capability] gap towards zero.”
{François Chollet}
Chollet and Mike Knoop, Zapier’s co-founder, initiated a competition to encourage the development of AI that could excel at ARC-AGI. OpenAI’s unreleased o3 model was first to qualify but required substantial computing power.
“Early data points suggest that the upcoming [successor to the ARC-AGI] benchmark will still pose a significant challenge,”
{François Chollet}
This year aims to launch a second-gen ARC-AGI benchmark with new competitions, while preparing a third edition. Despite criticism of overselling ARC-AGI’s potential as an AGI milestone, the initiative continues its quest.
“You’ll know artificial general intelligence is here when creating tasks easy for humans but hard for AI becomes impossible.”
{François Chollet}
OpenAI CEO Sam Altman hinted at potential collaborations with the ARC-AGI team for future benchmarks. As debates over AGI definitions persist, it’s clear the journey towards truly intelligent AI continues with curiosity and ambition.
