A recent study from researchers at MIT and Harvard suggests that while generative AI models can perform impressive tasks like generating text or navigating cities, they do not necessarily form a coherent understanding of the world. This was demonstrated through experiments showing that AI models, despite accurately providing driving directions in New York City or making valid moves in games like Othello, struggle when conditions change. For example, when streets were closed in New York City, the AI’s accuracy dropped significantly.
The research focused on transformers, the backbone of large language models (LLMs) like GPT-4, which predict the next word or token in a sequence. Despite their predictive power, these models can perform well without truly understanding the underlying rules of tasks, like how streets connect or how game strategies work. The researchers created two new metrics—sequence distinction and sequence compression—to test whether the models could form an accurate world model for tasks like navigating and playing games. Surprisingly, models trained on randomly generated data formed more accurate world models than those trained on data from strategic choices.
The study found that while these AI models can appear to understand tasks, they often rely on incomplete or incorrect internal models, as seen in the case of AI-generated city maps filled with nonexistent streets. The researchers argue that for AI to be useful in more complex, real-world scenarios, scientists need to develop better approaches to ensure that models not only make accurate predictions but also form coherent and reliable world models.
The research raises important concerns about deploying AI in real-world settings where unexpected changes, like detours or novel challenges, could lead to failures.
Here is a link to the study to find out more.
