Exploring the Hypothesis: Could ChatGPT Be a Celestial Tour Guide? (Continuation)
In a fascinating experiment, researchers have delved into the intricate world of ChatGPT, exploring its relationship with a popular video game, Minecraft. This exploration suggests that ChatGPT may be more game-like than a simple tool like a hammer or a horse.
At the heart of this experiment lies the concept of embeddings. These are high-dimensional numerical vectors, typically with thousands of dimensions, that ChatGPT uses to convert words into a list of numbers for computers to understand their meanings. Embeddings consist of a list of numbers that represent various characteristics of a word, such as how it looks, tastes, and even its nutritional content, although it's important to note that these characteristics are abstract representations learned by the model during training.
The process begins with tokenization, where ChatGPT breaks text into tokens, which can be entire words, parts of words, or even single characters. Each token is then mapped to a unique integer ID based on a vocabulary of about 50,000 tokens. These IDs themselves do not carry meaning but serve as an index. The next step is the embedding lookup, where each token ID is converted into a dense vector—a list of thousands of floating-point numbers. These vectors, or embeddings, are learned representations that encode semantic features.
To visualise this, the researcher has built an 8 x 8 x 8 walled garden in Minecraft Classic to "plot" the words and phrases. The positions of the words and phrases in the Minecraft garden match those in the 3D graph. Interestingly, the embeddings of apple-ish things, crushed ice, lemons, and phrases like "given to teachers" and "hangs from a branch" are found to be neighbours in the 3D graph. On the other hand, the embeddings of the word "hammer" is found to be off in a corner, suggesting it might be less related to the other items in this three-dimensional context that includes intelligence.
Sam Altman, CEO of OpenAI, describes GPT-4 as a tool, not a creature. However, in this experiment, ChatGPT discerns the intent of a prompt and navigates through thousands of dimensions to lead to the right spot in the GPT universe, suggesting it might be somewhere between a horse and a hammer in a three-dimensional context that includes intelligence.
The author plans to continue this exploration, using an embedding engine provided by OpenAI to look up the embeddings of more words and phrases, including horse and pie crust, to further understand the high-dimensional universe of ChatGPT. This experiment is not just a fun way to visualise the complex world of language models, but it also offers insights into how these models understand and process language.
In this experimental setup, the high-dimensional numerical vectors (embeddings), used by ChatGPT to comprehend word meanings, are visualized in a Minecraft garden, where words and phrases are represented by positions in the 3D Minecraft environment. The embeddings of items like apples, crushed ice, lemons, and phrases related to teaching are found to be neighbors, while the embedding of the word 'hammer' is discovered to be distinct, possibly indicating its lesser relation to the other items in the context of artificial-intelligence. This departure from the conventional understanding of a simple tool like a hammer lends credence to the idea that ChatGPT, while not a creature, operates somewhere between an ordinary tool and a horse when considering its ability to discern intent and navigate complex language data, which is ascribed to intelligence.
){kind=link}