Back

OpenAI says it disrupted a campaign to extract its models' hidden reasoning

Illustration: a glass flask distilling glowing liquid from a network of neurons, its tube corked so the vial beside it stays empty
Illustration: FrameSignal (AI-generated)

OpenAI said on Wednesday, Sept. 30, that it disrupted a coordinated campaign in July to extract protected reasoning, the hidden record of how a model works through a task, from its models. It attributes a core cluster of the activity to people associated with Moonshot AI, the Beijing company behind the Kimi AI models. OpenAI says the operators broke no encryption or databases; they manipulated conversations with its models at scale, in ways that violated its terms of service.

OpenAI published a security post on Wednesday, Sept. 30, saying it had identified and disrupted a coordinated campaign to extract protected reasoning from its models. OpenAI says stolen reasoning can help train another model without the original's safeguards, and it frames that as a safety and national-security risk shared across the most advanced AI systems.

OpenAI calls the pattern adversarial distillation. Distillation means training one model to imitate another by learning from its answers; the adversarial kind is the unauthorized use of one model's outputs or reasoning to help train, reproduce or improve another model. Protected reasoning is the model's internal record of working through a task. Extracting it can reveal information left out of the final answer and help others reproduce what the model can do.

The company says the operators did not break encryption, compromise a database or gain direct access to stored user conversations. Instead, they manipulated their conversations with its models so protected reasoning could be reproduced in visible form at scale, in ways that violated OpenAI's terms of service. OpenAI says this weakness is not unique to its models and that it has shared what it knows with other AI companies. Independent security researchers had also told OpenAI about related weaknesses, which it says it confirmed were real. One new method copied encrypted reasoning from one conversation and asked a model in another conversation to decrypt and transcribe it. OpenAI says it closed that pathway and strengthened its protections for hidden reasoning across users, workspaces, organizations and model families.

The activity began on July 1 at low volume, OpenAI says, then spiked on July 24 and 25 with 16,000 requests using an extraction pattern from more than 4,000 users. Related activity across a cluster of more than 15,000 users was fully disrupted by July 28. OpenAI notes that those figures count attempts, not necessarily successful extractions.

OpenAI says it is unclear whether all the operators came from a single actor. It attributes a core cluster of the activity to individuals associated with Moonshot AI, a Beijing company that makes the Kimi AI models. OpenAI says it responded by banning or restricting fraudulent accounts, tightening sign-up and infrastructure controls, widening its monitoring, and sharing its findings through the Frontier Model Forum and government information-sharing channels.

More on OpenAI