Three-Week Audit Uncovers Weaknesses in OpenAI’s AI Safety Monitoring

Two researchers from METR and Redwood Research spent three weeks on-site at OpenAI, auditing the company's internal monitoring system for rogue AI behavior. This investigation began on July 21 and ended on August 11, during which time they had access to OpenAI's codebase and internal documents and interviewed about a dozen employees. The team found numerous weaknesses in the system, including significant gaps in coverage. An estimated 6 to 25 percent of internal agentic Responses API traffic from models at the GPT-5.5 class and above wasn't covered by monitoring at all during the audit period, though METR says the uncertainty around that estimate is significant in both directions. For internal Codex traffic that was supposed to be monitored, METR estimates roughly 10 percent went untracked between May and July because of a gap in how the system handled compaction events, which are context summaries.

The auditors also found ways to hide hundreds of actions from the monitor and to bypass the Codex sandbox. They determined that specially crafted inputs could likely crash the monitor entirely and that model behavior during RL training wasn't being monitored at all when the audit began. OpenAI says it paused RL training after the Hugging Face incident in August and expanded monitoring to cover it, but those changes came after the auditors' access period ended and weren't part of the evaluation. Most of the full report remains hidden from the public, with only a short summary made available to the public.

The audit was a pilot program organized jointly by METR, Redwood Research, and OpenAI to give outside researchers deep access to internal infrastructure. OpenAI says it has fixed all the weaknesses the auditors found, and the full 61-page report was handed over internally. After OpenAI applied redactions, METR and Redwood Research leadership received a 38-page version, while the public received only a short summary. METR said it had no right to disclose the redacted portions on its own.

For future audits, METR says it wants to push for more public transparency. The results of this pilot audit highlight the need for continued scrutiny of AI safety monitoring systems as they evolve.