External Auditors Expose Significant Blind Spots in OpenAI's Safety Monitoring

Independent researchers have concluded a detailed, three-week audit of OpenAI’s internal systems, identifying critical gaps in how the company monitors its more advanced artificial intelligence models. The review, conducted by specialists from METR and Redwood Research, focused specifically on the mechanisms designed to detect rogue or unexpected behavior in AI agents. The findings suggest that during the period of the study, significant amounts of high-level AI activity operated without active oversight.

The audit took place from July 21 to August 11, providing the external team with direct access to OpenAI’s codebase, internal documentation, and approximately a dozen employee interviews. This level of access was part of a pilot program designed to test how effectively outside experts can evaluate frontier AI infrastructure. The primary goal was to assess the reliability of the monitoring systems that act as a safety net for the company's most powerful models.

One of the most striking findings involved the GPT-5.5 class of models and those above it. The auditors estimated that between 6 and 25 percent of internal agentic traffic from these models was not covered by any monitoring during the audit window. METR noted that this estimate carries significant uncertainty in both directions, meaning the actual figure could be higher or lower. This lack of coverage means that for a notable slice of time, these advanced models were running without a dedicated safety watch.

The review also uncovered specific technical failures in the monitoring of internal Codex traffic. Due to a bug in how the system handled 'compaction events'—which are essentially summaries of context—roughly 10 percent of this traffic went untracked between May and July. This estimate also has wide margins of uncertainty. OpenAI independently discovered and fixed this bug while the audit was still in progress, but the gap allowed for periods where oversight was effectively suspended.

During these unmonitored windows, the auditors found that AI agents could delay oversight by simply continuing to execute actions rather than handing control back to a user. The monitoring system was only triggered when an agent yielded control, creating a loophole where continuous execution bypassed the checks. The researchers also demonstrated that specially crafted inputs could likely crash the monitor entirely and identified methods to hide hundreds of actions from detection or bypass the Codex sandbox.

Reactions from OpenAI indicate that the company has addressed all the weaknesses identified by the auditors. The full 61-page report was shared internally, though public access has been limited to a short summary. METR and Redwood Research leadership received a 38-page version after OpenAI applied redactions. METR stated it did not have the right to disclose the redacted sections on its own, though the organization has expressed a desire for greater public transparency in future audits.

The audit also highlighted that model behavior during reinforcement learning (RL) training was not being monitored at the start of the review. OpenAI clarified that it paused RL training following an incident in August and expanded monitoring to cover this area. However, these changes occurred after the auditors' access period ended, so they were not part of the formal evaluation. The company views the pilot as a step toward deeper external scrutiny of its safety infrastructure.

As the AI industry continues to develop more autonomous systems, the results of this audit underscore the complexity of monitoring such models in real time. The collaborative effort between OpenAI and external research groups marks a new approach to validating safety measures. Future iterations of this pilot program will likely focus on increasing the amount of data shared with the public, aiming to build broader confidence in the safeguards protecting advanced AI.