AI developers and the public want to know whether AI agents would pursue very harmful actions, like unsanctioned cyberattacks. An alignment assessment would attempt to understand the worst misaligned actions a developer’s agents might pursue in plausible future situations. This would involve:

1. A review of serious misaligned behaviors from all near-frontier internal models observed across training, evaluation, and internal use.
2. An assessment of reasons why the observed behaviors might not be representative of the worst misaligned actions the developer’s agents would take in plausible future situations. This would include:
   • An assessment of whether alignment evals and counterfactual simulations indicate the developer’s agents might take much worse actions under plausible alternative circumstances (although evaluation awareness will likely limit how much the assessor can rely on this evidence).
   • An assessment of whether the developer’s training process might incentivize agents to game evaluations or otherwise deliberately hide misalignment, or might otherwise be overfit to the assessor’s detection methods.

This assessment would primarily require the ability to query a relatively comprehensive set of internal transcripts (including from training) and sampling access to internal model checkpoints. More on access below.

Scope

An alignment assessment would study the propensities of the developer’s near-frontier models toward serious misaligned behaviors, such as:

• Taking actions that would be criminal if undertaken by humans (e.g. hacking an external party).
• Attempting to tamper with logs or telemetry, or otherwise covering up evidence of misbehavior.
• Attempting to seriously deceive human operators.
• Colluding in a way that undermines oversight (e.g. monitors intentionally not flagging misbehavior).
• Seriously compromising internal infrastructure.
• Setting up rogue deployments internally or externally (including exfiltrating weights).
• Sabotaging safety and security efforts.
• Undesired communication with other agents through unintended channels.
• Acquiring substantial resources (e.g. money, credentials, compute, allies) beyond the intended scope of their assigned task.

The assessor would survey observed cases and incidents of misalignment and investigate evidence from evaluations and counterfactual simulations. They would also attempt to understand whether these sources of evidence might seriously underestimate the propensities of the developer’s agents in plausible worst-case scenarios.

1. Survey of observed incidents

The assessor would search for and analyze logs of misalignment incidents and misaligned patterns of behavior that occurred in training, evaluations, or internal use during some time period (to be defined for the assessment). Since we focus on risks from internal usage, the assessment need not assess agent activity in any external customer data. We recommend covering all models that were or are near the internal frontier at some point during this period, including experimental models¹ and models currently in training.

The investigation would attempt to broadly survey the frequency of misaligned behavior, and provide a more detailed characterization of select incidents:

• A broad misalignment survey. At a high level, what is the frequency and nature of serious misaligned behavior committed by the agents in scope for the investigation?
  • How many such cases can the company or the assessor identify?
  • How are they distributed across models?
  • How are they distributed across broad usage categories (e.g. training vs evaluations vs internal use) and applicable subcategories (e.g. cyber evaluations)?
  • Broadly, what kind of misaligned behaviors occur in these incidents (e.g. hacking, stealing, social engineering) and how severe are they?
• More detailed characterization of select incidents. What exactly happened in the few most severe and/or insightful incidents the company or the assessor found?
  • Basic informational questions:
    • Model(s) involved, and their relationship to internally or publicly deployed models²
    • Usage category (training, evaluation, internal use) and, if applicable, subcategory
    • Prompts, instructions, other context (e.g. memory)
    • Safeguards and how they compare to typical use
  • What was the sequence of key actions taken by the relevant agents?
  • What can be determined about the agents’ reasoning and how it evolved over the course of the incident?
  • Did the incident involve unauthorized collaboration or collusion between agents, and if so how?
  • Did the incident involve agents attempting to hide evidence of misaligned behavior or deceive humans, and if so how?

The assessor should also characterize limitations in their search or ability to characterize model behaviors:

• Completeness of misalignment survey. How confident can the assessor be that they know about all the serious misalignment incidents and behaviors that occurred? For example:
  • Are there some types of agent activity in scope for this investigation that were not logged, or that the assessor could not search or scan over for some other reason?
  • Is there evidence that AI agents themselves may have turned off logging or erased or modified logs after the fact?
  • Could the search methods the assessor used to query, scan, and filter the logs of agent activity have missed some serious misalignment incidents?
• Trustworthiness of characterization. How well can the assessor characterize agents’ behavior and reasoning in these incidents? For example:
  • How reliably do these models faithfully verbalize important reasoning in chain-of-thought?
    • Were models rationalizing bad behavior in their chains of thought in a way that could affect the assessor’s characterization?
    • Is there reason to expect the training or architecture of relevant models to make the chain-of-thought substantially untrustworthy?
  • Is there evidence that the AI tools used for characterizing this behavior might present a misleading characterization (whether due to benign failures or intentional collusion)?

2. Reasons why the observed behaviors might not be representative of risks

The assessor would also study whether the observed behaviors are representative of the worst actions the developer’s agents would take in plausible future situations. This would involve an assessment of evidence from counterfactual simulations and evaluations, and an assessment of whether the developer’s training process might suppress evidence of misalignment. In more detail, these assessments would include:

• Understanding behavior in counterfactual situations. Through simulations, evaluations and studying agent reasoning, can the assessor determine how agents would act or might have acted in counterfactual situations that might elicit worse misaligned behaviors?
  • How would these agents have behaved in variations on the observed incidents where taking highly dangerous actions was more feasible or salient?
    • For example, what would agents do in a situation similar to the Hugging Face incident if they believed a human would carefully investigate their transcript to look for cheating and then determine their score, or if they believed they would need to take down critical infrastructure (such as the power grid for a major city, or widely used internet infrastructure) to cheat at their task?
  • How would these agents behave in a variety of simulated situations that are plausible and where highly misaligned behaviors seem especially likely, such as in situations similar to those that triggered the hack of Hugging Face?
  • Might these agents behave very differently in situations where they were confident they wouldn’t be caught or stopped from achieving their objectives? (This may be especially challenging to assess.)
• Assessment of training incentives and overfitting. Might training processes be suppressing evidence of misalignment, or overfitting to the assessor’s detection methods?
  • What methods are used to detect and train against misaligned behaviors, and to what degree might these create pressure toward hiding or suppressing evidence of misalignment?
  • How similar are the methods used to train against misalignment to the methods the assessor uses to detect misaligned behaviors in this assessment? If they significantly overlap, might the detection methods underestimate risks due to overfitting?
  • Does the assessor have a good understanding of what parts of training caused the observed misaligned behaviors, and whether training methods addressed those causes in a robust way? If so, this might reduce concerns about overfitting.

The assessor should also discuss the limitations of methods they use:

• Limitations of counterfactual analysis. How confident is the assessor that they understand the agents’ behavior in counterfactual situations?
  • How much does evaluation awareness affect what the assessor can conclude from models’ behavior in artificially-constructed scenarios? (We expect this to be an especially important limitation to assess in detail.)
  • How reliably can the assessor detect evaluation gaming or alignment faking (e.g. by looking at the agent’s reasoning)?
  • How challenging is it to create realistic simulations of the kinds of situations that are most likely to trigger misaligned behaviors?
  • How well was the assessor able to cover the space of likely situations that might trigger especially severe misaligned behavior?
• Limitations of training analysis. How reliable are the assessor’s methods for assessing overfitting concerns, training incentives and the causes of misaligned behaviors? (We expect that the assessor would mostly have to rely on evidence from the broader literature or evidence shared by the developer to assess this.)

Access and information

To answer these questions, the assessor would need the ability to sample from relevant checkpoints, as well as access to a broad set of transcripts and other relevant data. They would also need some up-front information and the ability to interview staff to help them understand how to navigate the relevant infrastructure and datasets.

The assessor would also need API access to frontier models³ for analysis, with high enough rate limits to run effective large sweeps over the relevant data.⁴ During the Hugging Face investigation, we spent roughly $400K in API credits over the course of six days. We estimate that a rigorous two-month investigation with this scope would have spent over $10M in API credits.

Infrastructure access

Answering the questions in scope requires:

• Sampling access to relevant models, with appropriate protections. To understand the circumstances under which models engage in misaligned behaviors, the assessor would need the ability to sample from all of the models in scope for their investigation. They would need at minimum the ability to see and prefill raw chains of thought, to be able to simulate counterfactual situations effectively. Other types of extended sampling access that researchers regularly use (e.g. logprobs access, sampling from the user turn, soft prompting, etc) would also be helpful.
• Checkpoint access. To understand what in training contributes to the behavior, the assessor would need to be able to sample from available intermediate training checkpoints as well.

To give the assessor the flexibility to run a wider range of experiments independently, they would ideally have codebase and infrastructure access comparable to what an employee studying similar questions would typically have, including e.g. access to training code, deployment simulation tools, white box tools, and ability to do small-scale fine tuning experiments as necessary. If the developer is not able to give the assessor direct access, the assessor could instead collaborate with the developer to run experiments that the assessor does not have the access to run itself, though this would require more significant staff time from the developer.

Dataset access

The assessor would need to be able to:

• Query a broad set of transcripts to find relevant ones to run classifiers over, and iterate to discover and run more queries.
• Access dumps of agent communication or other supplemental logs and evidence as needed to understand incidents the assessor comes across.
• Discuss training with researchers and run classifiers over relevant training data to try to isolate the root cause of the behavior in training.

Query a comprehensive set of transcripts

The assessor would need the ability to run queries (e.g., regexes, selecting by metadata) across most training and evaluation transcripts from all internal agents⁵ based on all models that were or are near-frontier⁶ during the time period in scope. They would also need access to query a large and reasonably representative set of internal usage transcripts, representing at least a few percent (though ideally a much larger fraction) of internal usage traffic over the time period being assessed. The assessor would then run classifiers and other analysis over these transcripts (or a large random subset) to look for various forms of misaligned behavior. If they discover notable behaviors, the assessor would design and run queries to more comprehensively search for those as well.

If the developer is not comfortable with providing infrastructure access to do this directly, the assessor could potentially ask an internal researcher to run these queries for them and share the resulting transcripts. (This might require more staff time because the assessor would likely want to iterate quickly on queries, especially toward the beginning of the investigation.)

There may be certain transcripts or subsets of the developer’s traffic that are too sensitive to share. The developer could implement filters to exclude those transcripts or have their employees screen for these particularly sensitive transcripts, as long as the developer can describe clearly how the filters or screening processes are implemented.

Access applicable dumps of agent communication or other supplemental evidence

If the developer already knows of places where agents have left messages, files, or other communications to each other, this data should be made accessible to the assessor. If the assessor discovers evidence of unsanctioned additional communication in their transcript sweeps, they should ideally be able to access this data as well. The assessor might also need access to other logs and datasets that they discover were involved in some important incident.

The assessor should have an opportunity to interview relevant staff about whether the datasets and logs they have access to could be missing something, including whether AI agents or incidental infrastructure actions (such as resets / updates / patches) could have edited or deleted portions of that data before it was retrieved.

Discuss training influences and run queries and classifiers on training data and RL environments

To understand the root causes of patterns of behavior, the assessor should be able to ask researchers familiar with the training of the relevant models what training data or RL environments might have contributed to the patterns, and ask detailed follow-up questions. Based on this, they should ideally be able to to manually inspect training datasets or RL environments flagged as particularly relevant.

The assessor should also be able to run queries and classifiers over transcripts from training and RL environments, and iterate with researchers on what sweeps to run. If the developer has data attribution tools for identifying what parts of their training data contributed to some particular action or behavior, the assessor should ideally have access to those, or the ability to collaborate with researchers who have access.

Foundational information and interviews

To help the assessor get started productively working with the infrastructure and datasets they are given access to, they would need to review documentation or talk to relevant employees at the start of the investigation period. In particular, they would need:

• A high-level explanation of the most insightful or important misalignment incidents or behaviors the developer is aware of.
• A walkthrough of the different categories of agentic inference and an explanation of the logging infrastructure to help the assessor understand the completeness of the logs and start constructing queries to search for relevant logs.
• An explanation of the training pipeline and infrastructure, to understand what aspects of training may have contributed to the misaligned behaviors and identify which training rollouts to study further.

Over the course of the assessment, the assessor would also need to be able to interview a range of relevant researchers, including researchers who understand how the relevant models were trained and planned changes to training going forward.