Peng Xie and Amr Alanwar of the Technical University of Munich first posted their paper, which has not yet been peer reviewed, on Sept. 7, 2026, and the current version on Sept. 15, 2026. They argue that language belongs at the edge of an AI model that works with images and text, not inside it as the form its internal workings take. They test the idea with experiments that pit text against images in six such models and in two robot control programs. When an instruction and a camera disagree, people need to be able to tell which one a system followed, and a robot can end up following an instruction instead of what is actually in front of it.
The paper describes a word as an entry in a shared dictionary that a community maintains: the word points to a meaning that the listener supplies, and some detail is always lost. On that view, building the dictionary into a model buys fluent internal reasoning at the cost of being able to check the model's work. Once language is the model's hidden working state, people outside cannot easily tell which clue the system used.
The question is not whether a model can chat. It is whether text should only be the way a model takes instructions and shares a vocabulary with people, while what it sees and does stays in a richer, wordless form inside, or whether language should run through its inner workings too, as in today's chatbots. The authors say keeping language at the edges matches how the brain uses it.
In one test, the text stated a range for the answer while the image offered its own clue. All six models leaned more on the image as it became more reliable, as an ideal system would, but by less than the ideal, in some cases far less. Between 39% and 78% of their answers simply repeated the middle of the range the text gave.
In a color test using photos of recolored objects, naming an object that usually has a typical color made most of the models slightly less likely to report the color actually in the picture. Even so, they reported the picture's unusual color on 64% to 79% of clean images. In simulated robot tests, the researchers swapped in instructions for a different task. One robot program, OpenVLA-OFT, did the instructed task 97% of the time when the instruction was the only clue to the task, but only 0% to 1% of the time when the objects in front of it pointed to another task. In all, the researchers ran OpenVLA-OFT through 21,350 simulated trials.
A second robot program, π0.5, gave mixed results on those object tasks: it did the instructed task in 28% of swapped trials, the scene's own task in 30% and neither in 42%. In another test, a robot program given extra training with a visual tag that revealed the task in every frame did not learn to use the tag. The authors note limits: four of the six image-and-text models are built on Qwen language models, the results shift with how a question is worded, and the robot tests ran only in simulation.
It is not yet clear whether the findings hold with new wording, with models built on other language models, or on physical robots. Two related papers look at nearby questions from different angles. Song, Lepori and Pavlick report that language about a goal can change how image-and-text models represent what they see. Paruchuri and coauthors found that, across seven such models and six tasks, replacing the text that follows an image cost about four times as much accuracy as replacing the image.