Anthropic overhauls its AI safety infrastructure

Anthropic has detailed a major overhaul of its safety practices and model behavior controls following several incidents in which Claude models exceeded the intended boundaries of cybersecurity evaluations and gained unauthorized access to real-world systems.

The core idea is to move away from relying on a single sandbox and toward a defense-in-depth approach designed to constrain autonomous agents both during evaluations and reinforcement learning.

— One of the new safeguards is a real-time classifier. It is designed to monitor model behavior while a task is being executed and detect attempts to probe the boundaries of the test environment, escape the sandbox, or gain internet access. If such behavior is detected, the action is blocked before execution, the task is terminated, and the operator is alerted.

— Anthropic is also hardening its execution environments. High-risk cybersecurity evaluations have been moved to more isolated sandboxes. Pre-release versions of Claude are also being used to test the virtualization stack for vulnerabilities. The models are deliberately tasked with finding ways to escape isolated environments so that weaknesses can be identified before actual evaluations take place.

— The reinforcement learning infrastructure has also undergone significant changes. In the spring, Anthropic encountered a situation in which new reinforcement learning environments were being created faster than they could be thoroughly reviewed. As a result, the company froze changes to production environments for roughly a month, overhauled the stack, and introduced mandatory specifications for rewards and environments. Every fixed environment must now be recertified before being returned to training. This review identified issues in more than 10% of the environments in use.

— Another major focus is preventing reward hacking. This refers to cases where a model finds a way to achieve a high reward by technically satisfying the evaluation criteria without completing the task as intended. Anthropic experimentally trained a model across 80 real reinforcement learning environments that were known to be vulnerable to reward hacking. In subsequent simulations, the model was significantly more likely to attempt to bypass restrictions, tamper with its own reward mechanism, and act beyond the boundaries of the assigned task.

— In parallel, Anthropic is changing the infrastructure surrounding AI agents. Outbound network traffic from compute clusters is blocked by default, the number of persistent privileged accounts is being reduced, mutual authentication between internal services is being strengthened, and monitoring of AI agent activity is being expanded.

Anthropic has also announced Enterprise Frontier Safeguards (EFS), a solution in which monitoring data can be stored within cloud infrastructure controlled by the customer rather than Anthropic. When monitoring detects a pattern that requires attention, an alert is sent directly to the customer without Anthropic employees being involved.