The Limitations of Large Language Models
Large language models (LLMs) like GPT-4o and Claude have gained attention for their ability to write essays and solve equations quickly. These models can process vast amounts of data, but they also have limitations. Sometimes, they fail in understanding basic concepts like letters and syllables, leading to viral memes and a sense of relief that AI still has a long way to go before becoming all-powerful.
One reason for these failures is that LLMs are built on transformers, a type of deep learning architecture. Transformers break down text into tokens, which can be words, syllables, or even letters. However, this doesn’t mean the models truly understand the individual components. As AI researcher Matthew Guzdial explains, the transformers don’t know about the individual letters in a word like “strawberry,” but rather the contextualized representations of the tokens that make up the word.
This inability to comprehend letters and syllables is deeply embedded in the architecture of LLMs, making it a challenging issue to fix. Even if experts were to establish a perfect token vocabulary, models would likely still struggle with the fuzziness of language.
The complexity increases when LLMs learn multiple languages. Tokenization methods that assume spaces separate words can’t be applied to languages like Chinese, Japanese, Thai, Lao, Korean, and Khmer, where spaces are not used. It’s computationally infeasible for transformers to look at characters directly without imposing tokenization.
Diffusion models, used in image generators like Midjourney and DALL-E, offer an alternative to transformers. These models reconstruct images from noise and are trained on large image databases. While they perform well on objects like cars and faces, they struggle with smaller details like fingers and handwriting. However, these problems may be easier to address than the issues with transformers.
AI image generators may produce unexpected results, like misspelling words on a menu for a Mexican restaurant. However, OpenAI is developing a new AI product called Strawberry to enhance reasoning abilities. Strawberry can generate accurate synthetic data, improving the accuracy of OpenAI’s LLMs. Google DeepMind has also made strides in formal math reasoning with AlphaProof and AlphaGeometry 2, which performed well on math problems from the International Math Olympiad.
While memes about AI’s inability to spell “strawberry” circulate, OpenAI and Google DeepMind are actively working to overcome these limitations and improve the capabilities of AI systems.
