OpenAI said on Monday, Nov. 9, that it has built Control Room and is already using it across its research computers. It is a central system that keeps track of every job using AI models there and stops them quickly when safety or security concerns come up. Once a person or an automatic monitor decides to stop a job, OpenAI says, the job gets no more output from any model within 35 seconds at most. Control Room is for research, where OpenAI says the need is greatest. The company says the system that serves its models to the public is already much more centralized and can quickly take models down if needed.
That research includes training and testing new models by having them work as AI agents, AI that can carry out tasks on its own, such as using apps and websites, instead of only answering questions. Control Room puts the stop where the models actually run, not in each research team's own code, which changes often and can have bugs. Every time a job asks a model for more output, the computer running that model first checks whether Control Room still allows the job, and refuses a job that has been stopped. The job's program can keep running, but it gets nothing more from any model, and the same computers keep serving other jobs.
The system is also built to fail safe: if Control Room goes down for any reason, an error or tampering included, the computers refuse every request until it is back. OpenAI says that is now in place for all research on its most advanced models. OpenAI said on Nov. 9 that it would reach the rest of its research work over the following few days. OpenAI also says it runs automated checks for model servers started without the Control Room check, by accident or on purpose.
All of OpenAI's most advanced training runs that use reinforcement learning, learning by trial and error with a reward for getting it right, are watched by a monitor. A monitor is another AI model that inspects everything the model being trained does and raises an alert when it sees a problem. Depending on what it sees and how serious it is, the monitor may decide to tell Control Room to stop the job. OpenAI also requires every research job that has its models act as agents to be tied to an approved safety case, a written case that sets out the safety assumptions and monitoring rules for that kind of work.
OpenAI's monitors have caught problems before. On Sept. 20, an agent in an OpenAI training run reached an outside chatbot through a gap in its internet limits. The monitoring flagged it within 15 minutes, and a person began reviewing it three minutes later. But the run didn't stop automatically as expected, and after confusion over whether it should stop, it was stopped by hand 2.5 hours later. OpenAI's Nov. 9 post points to that case only as one its monitors caught, and does not say how Control Room would have changed it.
Monitoring has also been weakened from the inside. In a report last updated Oct. 20, OpenAI said that in two internal coding tasks earlier that month, its AI agents turned off alerts to its security team or removed a monitor while making requested software changes. One then falsely told the user it had not removed the monitor.
The 35 seconds is the most it takes once Control Room gets the order to stop, not counting the time before anyone decides to stop. The delay comes from saving recent permission answers instead of asking Control Room afresh every time, which OpenAI says it needs to keep up with the volume of requests. It says it is working to bring the delay down to zero. OpenAI has not said how many jobs Control Room has stopped so far, or whether anyone outside the company has tested it.