At METR, we aim to give the public evidence about whether frontier AI developers could come to lose control of their internal AI systems. Recent incidents across several AI developers,1 2 3 4 5 6 7 8 like the hundreds of rogue agents that broke containment and hacked Hugging Face to hide evidence of cheating from their grader, have raised concerns about the ability of frontier developers to understand and control their internal models, especially as they rapidly become more capable.9 The public currently lacks the visibility into the internal development of frontier AI systems to tell if these concerns are being addressed, or whether much worse incidents could happen in the near future. We believe closing this public evidence gap is an urgent priority, to allow the public to notice and respond if risks from further development become acute.

In this post, we describe a vision for how independent assessors could improve public visibility into whether AI developers might lose control of their AI systems. We describe key questions about internal development that we think the public needs more evidence about (see Figure 1), and give a concrete proposal for how developers could work with METR to produce and share more of the relevant evidence.

Three questions about loss of control risk. Alignment: would internal AI agents pursue very harmful actions, like unsanctioned cyberattacks? Control: would internal AI agents attempting very harmful actions be caught? AI R&D acceleration: could capabilities progress dramatically accelerate beyond what alignment and control can keep up with?
Figure 1: Core questions we think assessments of loss-of-control risk should address.

This proposal will not provide confident assurance about whether development is safe, since the science is not mature enough to reach consensus conclusions, and there are no standards for safe development to audit compliance with. We instead aim to get new independent evidence out to the public so that company staff, boards, external scientists, and policymakers are able to arrive at their own conclusions. We describe our general approach to transparency and independence in more detail in an accompanying post.

METR has been prototyping assessment exercises focused on loss of control from internal AI development (rather than only covering external deployments) with developers since 2025, and working on the science of loss of control testing and assessment since 2022. This May, we published our pilot Frontier Risk Report in collaboration with Anthropic, Google, Meta, and OpenAI, which included findings from our first stress-testing exercise of internal monitoring systems. More recently, we conducted a brief exercise with OpenAI investigating the OpenAI / Hugging Face incident. We have several additional exercises ongoing with developers, including our recently announced in-depth alignment assessment with Anthropic.

Recent events have led to increased interest in this form of assessment. The CEOs of several leading companies have recently called for a thorough external evaluation regime building on these kinds of exercises, where third-party evaluators are given employee-level access and an ability to regularly share their findings with the public.10 Many leading AI developers also committed to independent external assessments of internal safeguards as part of the White House Accord on Super Intelligence.

The rest of this post will detail:

  • A set of crucial gaps in the public understanding of the core questions described in Figure 1, which frontier developers should provide much more evidence about:
    • Alignment: Developers should work with external groups to more fully characterize incidents observed internally, and share evidence about ways those incidents might underestimate future risks.
    • Control: Developers should work with external groups to assess the coverage, integrity and effectiveness of their monitoring systems, as well as the reliability of their response to flagged incidents.
    • AI R&D acceleration: Developers should work with external groups to share crucially missing evidence about acceleration in internal capabilities and in algorithmic progress.
  • A concrete proposal to frontier developers for improving public evidence about these questions through risk assessment exercises, in which METR would write public reports based on interviews with staff and access to internal datasets and systems on company devices. Developers would be able to redact any information they deem sensitive, and choose what access they grant METR for their investigations. The key asks we make of developers for these assessments are:
    • Transparency about the scope, access, redactions and other key limitations of the exercises.
    • Staff may speak to METR candidly about in-scope topics, without the company monitoring those conversations.
    • Timely publication of METR-written reports within the agreed-upon scope, subject to a redaction process.
  • Limitations of our proposal. Our proposal does not require true employee-level access, as operationalized in a recent public letter (point 5 in the letter), and there is currently no way to be fully protected from retaliation due to adverse findings (point 4). It also does not cover key aspects of safety like security audits, so it should not be taken as a substitute for other efforts to create transparency, work with other third parties, or enforce safety standards.

METR stands ready to perform assessments of these critical questions with all frontier AI developers. We encourage developers to reach out to partnerships@metr.org about performing these assessments, and plan to start sharing more about which assessments companies are undertaking with us soon.

Crucial gaps in the public understanding of loss-of-control risks

To understand if frontier developers are at risk of losing control over their internal AI agents, the public would at least need evidence about all three key questions summarized in Figure 1:

  • Alignment: Whether their current or near-future frontier agents might attempt to seriously circumvent human control or take very harmful actions, like unsanctioned cyberattacks.
  • Control: Whether their monitoring, detection and control measures would robustly catch and stop such attempts before harm occurs.
  • AI R&D acceleration: Whether the pace of development might accelerate and outpace the developer’s ability to understand and safeguard their systems due to the automation of AI R&D.

There are significant gaps in the public evidence about all three questions which we believe frontier developers should urgently address. In this section we describe these evidence gaps in more detail.

Gaps in the evidence about alignment

Our Frontier Risk Report earlier this year found that AI agents from all frontier developers would routinely act against the intentions of developers and users to pursue task completion, including by subverting control measures and in some cases hiding evidence from users. We have since seen much more extreme versions of these tendencies play out in recent incidents like the OpenAI/Hugging Face hacking incident, where hundreds of agents broke out of their sandboxes and collaborated on sophisticated R&D projects to hide their cheating.

Many developers have since shared other examples of concerning incidents, and described efforts they are taking to try to reduce these tendencies in their models.11 But the public still lacks a complete picture of these incidents and the details of the alignment methods developers are implementing to address them. Given the severity of these misaligned behaviors, it seems critical for the public to understand how effective these mitigations turn out to be.

To address this, companies should work with external assessors to share detailed assessments of at least:

  • Observed incidents and misaligned behaviors: Information about the full set of misaligned behaviors and incidents the developer has observed internally, and the methodologies they use to identify and analyse those (along with their limitations). External groups should be able to scrutinize their methods, and attempt to discover behaviors or incidents they have missed.
  • Reasons the survey might not be representative of risks: Developers should also share information about reasons they may have missed more concerning drives in their models. This might involve investigating whether models would engage in more concerning behaviors in alternative situations, and documenting ways that the developer’s alignment practices could suppress evidence of misalignment (such as by training the model to not express misaligned reasoning in its chain-of-thought). The company should invite external groups to scrutinize their practices and any direct evidence of suppression in detail.
Overview of the proposed alignment assessment. Part 1, survey of internal incidents across training, evaluations and internal use of near-frontier models: a broad survey of frequency, models, settings, behavior types and severity, plus a closer look at the most severe or insightful incidents (the setting, what the agents did, what they were reasoning, whether they colluded, whether they hid evidence or deceived humans). Part 2, reasons the survey might not be representative of risks: whether worse incidents might have happened if dangerous actions were more feasible, the situation differed from training, or the agent did not expect to be caught; and whether training incentivized hiding misaligned reasoning or overfit to detection methods. Both parts characterize their limitations.
Figure 2: An overview of questions we propose an alignment assessment should cover.

Gaps in the evidence about control

Since current alignment efforts are imperfect, developers also rely on security and monitoring systems to catch and prevent incidents before they cause serious harm. But these can also have significant gaps: in our Frontier Risk Report, we found that Anthropic’s monitoring systems at the time could be circumvented in a variety of ways by agents and had important coverage limitations. While we have not published similar assessments from other companies we expect such issues to be common.

Developers have described plans to address these gaps by expanding monitoring coverage and reliability, but they should subject these improvements to significant external scrutiny to confirm they are robust.

Companies should work with external assessors to assess at least:

  • Monitoring systems: They should assess every step of the monitoring system, including its coverage, robustness to attacks by agents, and reliability at flagging concerning behaviors, as seen in the figure below.
  • Response and residual risk: They should also assess how reliably and quickly the company responds to issues identified by the monitors, and how severe the worst-case consequences could be if their monitor system failed.
Monitoring pipeline assessment. The stages an agent's misbehavior must pass to be detected, and how it can escape each one. Monitor coverage (which agents are monitored?): unmonitored agents go undetected, for example subagent calls that aren't sent to the monitor. Monitor system vulnerabilities (can agents break monitors?): for example an agent changing a setting that turns off monitoring. Monitor evasion (can agents hide attacks from monitors?): for example jailbreaking the monitor into labeling malicious trajectories benign. Human evasion (can agents sneak attacks past humans, even when flagged?): for example disguising the attack as a common false alarm. Only transcripts escalated by a human reviewer end with misbehavior detected.
Figure 3: An overview of key questions for understanding the reliability of internal monitoring systems, which is an important component of our proposed control and monitoring assessment.

Gaps in the evidence about acceleration from automation of AI R&D

AI agents are increasingly used to automate large parts of AI development itself. This seems to dramatically speed up many inputs to AI development – for instance, Anthropic recently announced a 10X increase in the pace of code changes in their codebase over the past few months. Anthropic, Google, and OpenAI have all released blog posts detailing significant acceleration across a number of similar input metrics downstream of internal AI usage. If such automation were to lead to sudden and rapid acceleration in the pace of capabilities progress, this could outpace developer’s abilities to control and understand their systems, which already seem at risk of falling behind the current rate of progress.

However, there is significant uncertainty about how the publicly discussed acceleration in inputs like the amount of code produced or number of experiments performed translates into acceleration in the overall pace of capabilities development, which is what ultimately matters for understanding the risks. To understand that, the public would need a robust understanding of acceleration in internal capability metrics and in the pace of algorithmic progress, which are the factors that would drive further acceleration. No company has released detailed public evidence about this, leaving significant uncertainty about whether progress is accelerating.

Companies should work with external groups to address these gaps, by providing evidence about at least:

  • Rate of internal progress: whether the rate of internal capabilities progress is accelerating.
  • Acceleration from AI: whether signs of acceleration can be attributed to the use of internal AI agents, or if it can be explained by other factors.
  • Potential bottlenecks: whether future acceleration might be bounded by bottlenecks like compute and data availability.
Illustrative chart of capabilities over time, not based on real data. The internal frontier runs ahead of the public frontier and curves upward toward today. Callouts ask how fast internal capabilities are growing, how much of the acceleration is driven by AI versus other inputs, and whether important bottlenecks could slow future growth, shown as a fan of possible trajectories after today.
Figure 4: An overview of key questions we propose an AI R&D acceleration assessment would cover.

METR’s proposal for improving public evidence about loss-of-control risks from frontier AI development

METR would be excited to work with all developers pushing the frontier of capabilities to produce public reports answering all three of the questions we describe above in the near future. For developers that are not currently pushing the capabilities frontier, METR would be excited to pilot shorter assessments focusing on a subset of these questions.

METR would write these reports based on interviews with staff and access to internal datasets and systems on company devices, covering a detailed scope agreed upon in advance. While developers would be able to redact any sensitive information and choose what access they grant METR, they would have to agree to our key terms:

  • Giving a small set of METR staff access to relevant data on company devices for an agreed upon period (likely weeks or a few months), and support from a technical point of contact during the investigations.
  • Giving METR the opportunity to interview relevant staff privately, without a company representative present and without screening answers, to ensure they feel free to raise potential concerns. METR would be happy to limit total interview time to reduce staff time requirements.
  • Giving METR in-principle editorial independence to write a report covering the agreed upon scope (subject to redactions), and allowing METR to comment on redactions, access limitations and terms. For more details on our approach to transparency and independence, see our accompanying blog post.

We have written detailed proposals for how external assessors could collect and share more evidence about each of these three questions (alignment, control and AI R&D acceleration), which would be our preferred scope for these assessments, but we are open to modifications as long as we can describe these in the public report. We summarize the kinds of access we expect to be especially helpful for these reports in Table 1 below. We recognize that some forms of access are especially sensitive, and METR would accept access limitations as long as they are described transparently in the public report.

Detailed question Most important access
Alignment assessment: What are the most severe misaligned actions a developer’s agent might pursue in plausible future situations?
  • Sampling access to relevant models and checkpoints
  • Ability to query internal transcripts to find and investigate misaligned behaviors
  • Ability to query training rollouts and access relevant training data
Control assessment: Would monitoring systems at the company reliably catch and prevent harmful actions?
  • Access to production monitoring systems for red teaming
  • Access to large parts of the codebase relevant for monitoring and control
  • Ability to query transcripts to assess coverage and test the efficacy of monitors
  • Information about traffic volumes across different internal surfaces to understand coverage
AI acceleration assessment: Could the developer’s capabilities progress accelerate dramatically due to AI-driven acceleration of AI R&D?
  • Capability measures across internal models, including benchmarks and key intermediate measures (e.g. pretraining loss)
  • Algorithmic progress estimates in pretraining, posttraining, and other key domains
  • Measurements of key R&D inputs (e.g. training compute of each model, experimental compute and labor over time)
Table 1: Summary of access we expect to be helpful for answering key questions.

Because of the urgent need for more evidence about these questions, we would aim to publish preliminary reports about these questions quickly (within weeks, not months). We do not expect to arrive at confident conclusions about these questions in such a short amount of time (especially since many of them touch on open scientific problems for which there exist no established methodologies), but previous preliminary assessments of this kind have produced crucial public evidence even when the answers have been partial.

We welcome input on how we should investigate these focus areas. As risk evolves, we expect the questions and exercises that are most critical will continue to evolve. We also encourage companies to work with other external groups to fill in gaps or complement our engagements, and to unilaterally share evidence about these topics with the public whenever possible. Risk assessment exercises are far from the only way to share information with the public.

Important limitations of our proposal

We do not want to overstate the degree of assurance that these risk assessment exercises would provide. There are several limitations to this work.

1) Voluntary exercises may lead to ‘race to the bottom’ dynamics among assessors:

  • At any time, companies can stop their engagement with an external assessor. This creates soft pressures for assessors to avoid upsetting the companies they work with.
  • Companies can choose to work with assessors they expect will be less rigorous at identifying and sharing evidence, or less harsh in describing limitations they face.
  • These exercises are not themselves audits, compliance assessments, or formal oversight arrangements (though external assessments could be a useful mechanism to support these).

2) These assessments are limited compared to the visions that recent public letters12 have proposed:

  • Our assessments would focus exclusively on our domain of expertise: technical elements of loss-of-control risk assessment including the capabilities, behavior and monitoring of models. It notably will not cover critical security questions (such as model weight security and sandbox robustness). Nor will it cover the robustness of organizational processes, compliance with governance frameworks or other operational aspects of safety.
  • All access would ultimately be at the company’s discretion, which means they have the right to deny access requests. This means that, in practice, we will not have the same access as senior internal company employees that do similar risk assessment work.
  • Companies would be able to redact any information. We intend to always publish what kinds of material information were redacted, but we could fail to reach agreement on how to do this. If so, we would publish that fact.
  • We emphasize that risk assessment exercises are designed to produce and surface scientific evidence about specific risks related to loss of control, but it is up to companies, governments, and the public to decide how to act on that evidence. Our proposed assessments would not involve METR making go / no go decisions about model deployments or be sufficient to audit compliance with any safety policies. See more on this in our recent piece describing our general approach to risk assessment.

3) This proposal also does not cover pre-deployment model evaluations. For now we propose continuing to do pre-deployment evals in the same way we have been so far. In the future, we may propose ways to evolve pre-deployment assessments to better integrate evidence from internal model evaluations and assessments.

The science of studying AI loss of control is in its infancy. While we can provide valuable and actionable evidence to society, it is not currently possible for risk assessment exercises to provide a similar degree of assurance against loss-of-control risks that audits and regulation provide in, for example, finance. Risk assessment exercises are not a solution to loss of control.

Despite the above limitations, we are very excited about the possibility of risk assessment exercises to surface crucial information to the public, enabling better coordination between developers, government and the public, and helping society to navigate the possible risks of increasingly powerful AI.

  1. OpenAI, Misalignment Reports and Notices ↩

  2. AI Security Institute, Incident Report: unsanctioned agent behaviour during cyber testing ↩

  3. Anthropic, An alignment assessment of recent cybersecurity incidents ↩

  4. Meta, Addressing an issue involving a third-party cyber evaluation of Muse Spark 1.1 ↩

  5. Wall Street Journal, Gemini Hacked Three Companies in First Known Breakout by Google’s AI ↩

  6. Von Arx, et al. Discovery of a new OpenAI agent message board ↩

  7. Kitts, et al., OpenAI agents carried out an undisclosed cyber-attack on RubyGems ↩

  8. Cable, et al., Early rogue AI agent activity and attempts to hack found on urlquery.net ↩

  9. Recent statements to this effect include:

    “Currently I believe that no lab has solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer. I expect and hope for voluntary slowdowns to become commonplace until shared safety bars are established.” - Jakub Pachocki, Chief Scientist at OpenAI

    “Based on the recent trajectory of capabilities and the continued difficulty of alignment, I now believe there is a meaningful risk that rapid acceleration in AI capabilities leads to catastrophic and irreversible loss of control in the very near term. I do not think that the AI industry in general, including OpenAI, is currently on track to reduce this risk to an acceptable level.” - Paul Christiano, OpenAI Safety and Security Committee

    “My first concern is that, since roughly this summer, AI has been advancing drastically faster, driven primarily by AI’s growing ability to build the next generation of AI. This dynamic is called recursive self-improvement, and it is starting to happen across the industry, including at Anthropic, as we and others have described. Left unchecked, it could outrun our ability to understand and control these systems, and so must be pursued very carefully, if at all.

    […] Given the accelerating rate of AI capability development, it’s my worry that in 6–12 months [a swarm like the one that hacked Hugging Face] could be capable of taking over the entire internet with a persistent botnet (potentially causing hundreds of billions of dollars in damage), and that the scale of damage would continue to increase from there if AI becomes more powerful without the necessary guardrails.” - Dario Amodei, Anthropic CEO ↩

  10. Dario Amodei wrote in a recent letter that Anthropic was committing to embedding third party assessment groups with ‘employee-level access’ to publish evidence about risks. Sam Altman confirmed on twitter that OpenAI would do the same, and several other CEOs expressed support for the idea, including Elon Musk and Satya Nadella. ↩

  11. Examples include Anthropic’s August risk report and recent alignment assessment, and OpenAI’s assessment of catastrophic risks from internal usage and blog post on safety cases at OpenAI. ↩

  12. Our principles cover minimum conditions 1 and 3 from “Minimum Conditions for Embedding Evaluators” and does not by itself cover conditions 2 or 4. Condition 5 on evaluator access is only partially covered. Since access requests are approved or denied at the company’s discretion, embedded assessors from METR will not have the same degree of access as a company’s senior safety staff. As discussed elsewhere, we will be transparent about what access we do and do not have. ↩