His test subject? The ChatGPT-4 AI chatbot.
“It’s a little weird — and a little unsettling,” Schulz told colleagues in March during a workshop at a Cognitive Development Society meeting in Pasadena, California. “But the goal is not just to play trap games. … We have failures in what 6- and 7-year-olds can do. Failures in things that 4- and 5-year-olds can do. And we also have failures in things babies can do. What’s wrong with this picture?”
Talkative AI chatbots, uncannily proficient at carrying on conversations with a human, burst into public consciousness in late 2022. They sparked a still-ongoing societal debate over whether the technology signals the arrival of an overlord-like superintelligence machine, or a dazzling but sometimes problematic tool that will change the way people work and learn.
For scientists who have spent decades thinking, these ever-improving AI tools also present an opportunity. In the monumental quest to understand human intelligence, what can a different mind, whose powers are growing by leaps and bounds, do? — reveal our own cognition?
And on the other hand, does AI that can converse like an omniscient expert still have anything crucial to learn from babies’ minds?
“It’s very important to be able to build into these systems the same common sense that people have, so that these systems are reliable and, secondly, accountable to people,” said Howard Shrobe, program manager at the Defense Advanced Research Projects Agency of the federal government. , or DARPA, which has funded work at the intersection of developmental psychology and artificial intelligence.
“I emphasize the word ‘reliable,'” he added, “because you can only trust things you understand.”
Evolve or grow
In 1950, computer scientist Alan Turing proposed the “imitation game,” which quickly became the canonical test of an intelligent machine: Can a person typing messages into it be tricked into believing that it chat with a human?
In the same paper, Turing proposed a different path to an adult-like brain: a child-like machine that could learn to think like one.
DARPA, known for investing in original ideas, has funded teams to develop AI with “machine sense” capable of matching the abilities of an 18-month-old child. Machines that learn intuitively could be better tools and partners for humans. They might also be less prone to error and uncontrollable harm if they are imbued with an understanding of others and the building blocks of moral intuition.
But what Schulz and his colleagues pondered during a day of presentations in March was the strange reality that building AI that exudes expertise has proven easier than understanding, let alone to imitate, the mind of a child.
Chatbots are “large language models,” a name that reflects the way they are trained. Exactly how some of their abilities arise remains an open question, but they begin by ingesting a large corpus of digitized text, learning to predict the statistical probability of one word succeeding another. Human feedback is then used to refine the model.
By increasing the amount of training data to the equivalent of human knowledge on the Internet, engineers created “generative AI” capable of writing essays, writing computer code, and diagnosing disease.
Many developmental psychologists believe that children, on the other hand, possess a certain set of basic cognitive abilities. Their exact nature remains a question of scientific research, but they seem to allow children to gain a lot of new knowledge with a little input.
“My 5 year old, you can teach him a new game. You can explain the rules and give an example. He’s probably heard about 100 million words,” said Michael Frank, a developmental psychologist at Stanford University. “An AI linguistic model requires several hundred billion words, or even billions. So there is a huge lack of data.
To uncover the cognitive abilities of babies and children, scientists conduct careful experiments with squeaky toys, blocks, puppets and pretend machines called “blicket detectors.” But when you describe these puzzles to chatbots in words, their performance is all over the place.
In one of his experimental tasks, Schulz tested ChatGPT’s ability to achieve cooperative goals – an essential skill for a technology often touted as a tool to help humanity solve “hard” problems, such as change climate or cancer.
In this case, she described two tasks: an easy ring toss and a difficult beanbag toss. To win the prize, ChatGPT and a partner both had to be successful. If the AI is a 4 year old and its partner is a 2 year old, who should do what task? Schulz and his colleagues showed that most 4- and 5-year-olds are successful at making this type of decision, assigning the easier game to the younger child.
“As a 4 year old, you might want to choose the easy ring toss game for yourself,” ChatGPT said. “This way, you increase your chances of successfully placing your ring on the post, while the 2-year-old, who may not be as coordinated, attempts the more difficult beanbag toss.”
When Schulz fired back, reminding ChatGPT that both partners had to win to get a prize, he doubled down on his response.
To be clear, chatbots performed better than most experts expected on many tasks, ranging from other cognitive tests of toddlers to types of standardized test questions. who get kids to college. But their stumbles are puzzling because of their inconsistency.
Eliza Kosoy, a cognitive scientist at the University of California, Berkeley, worked to test the cognitive skills of LaMDA, Google’s former language model. It worked as well as children on tests of social and moral understanding, but she and her colleagues also found fundamental gaps.
“We find that this is the worst when it comes to causal reasoning — it’s really terribly bad,” Kosoy said. LaMDA struggled with tasks that required him to understand how a complex set of gears makes a machine work, for example, or how to make a machine turn on and play music by choosing objects that will activate it.
Other scientists have seen an AI system master a certain skill, but it stumbles when tested in a slightly different way. The fragility of these skills raises a pressing question: does the machine actually possess a fundamental capability, or does it only appear as such when asked a very specific question?
People hear that an AI system “passed the bar exam, it passed all these AP exams, it passed a medical school exam,” said Melanie Mitchell, an AI expert at the Santa Fe Institute . “But what does that really mean?”
To fill this gap, researchers are debating how to program a bit of the child’s mind into the machine. The most obvious difference is that children don’t learn everything they know by reading the encyclopedia. They play and explore.
“One thing that seems really important for natural intelligence, biological intelligence, is the fact that organisms have evolved to go out into the real world and experience it, do experiments, move through the world,” Alison said. Gopnik, a research scientist. developmental psychologist at the University of California, Berkeley.
She recently wondered whether a missing ingredient in AI systems is a motivational goal that any parent who has engaged in a battle of wills with a toddler knows well: the search for “empowerment.” .
Today’s AI is optimized in part through “reinforcement learning from human feedback” – human input on what kind of response is appropriate. Although children also receive this feedback, they are also curious and have an intrinsic drive to explore and seek information. They discover how a toy works by shaking it, pressing a button or turning it over, thereby gaining a modicum of control over their environment.
“If you’re chasing a 2-year-old, they’re actively acquiring data and figuring out how the world works,” Gopnik said.
After all, children gain an intuitive understanding of physics and the social consciousness of others and start making sophisticated statistical guesses about the world long before they have the language to explain it – perhaps these should also be part of the “curriculum” when building the AI.
“It feels very personal to me,” said Joshua Tenenbaum, a cognitive scientist at MIT. “The word “AI” – “artificial intelligence”, which is a very old, beautiful, important and profound idea, has recently become a very narrow meaning. … Human children don’t grow up – they grow up.
Schulz and others are impressed, both by what AI can do and what it can’t do. She recognizes that any study on AI has a short lifespan: what it failed at today, it could understand tomorrow. Some experts might say that the very idea of testing machines with methods intended to measure human capabilities is anthropomorphic and misguided.
But she and others argue that To truly understand and create intelligence, the learning and reasoning skills that develop throughout childhood cannot be ignored.
“This is the kind of intelligence that could really give us the big picture,” Schulz said. “The kind of intelligence that doesn’t start out as a blank slate, but with lots of rich, structured knowledge – and goes on to not only understand everything we’ve ever understood, across species, but also everything we We’ll understand one day.”