When Guardrails Go Wrong

We don’t need hamstrung AI.

By Mike Loukides

The latest round of restrictions and safeguards for frontier models are overly fussy and limiting. A Claude skill that I created demonstrates what happens when guardrails go astray. My skill helps me to find articles and blog posts that go into O’Reilly Radar’s monthly Trends to Watch. It reads roughly a dozen well-known sites like The New Stack, The Next Web, and Hacker News, plus any other sources that it finds useful. After reading the sites, it produces a digest of the most important articles published in the last day. I use it as a sanity check on my own reading: Did I miss anything important? Am I on the fence about something that might be an important leading indicator?

I’ve used the skill daily for a couple of months now. It suddenly stopped working with the following message:

API Error: Sonnet 5’s safeguards flagged this message. Our intentionally broad safeguards allow us to deliver more capabilities faster, but can sometimes flag legitimate cybersecurity work. Apply to the Cyber Verification Program to reduce these interruptions. Send feedback with /feedback or learn more: https://support.claude.com/en/articles/14604842-real-time-cyber-safeguards-on-claude

When I started a new Claude Code session with Haiku, the skill worked without problems. (I didn’t try Opus or Fable; if Sonnet found the skill dangerous, I’m sure Opus and Fable would draw the same conclusion.) GPT 5.6 with “high” reasoning was able to execute a very similar skill without problems. So what happened to Sonnet?

The best approach to debugging AI is often to ask the AI itself, so I pasted the message into another Claude Code session and asked it what was happening. The response came down to the descriptions of Hacker News, Bleeping Computer, and The Register. The phrase “vulnerabilities, exploits, threat reporting” in the description of Hacker News triggered Sonnet’s guardrails. Ironically, that description is both incorrect and Claude generated. (Reminder to self: Be more careful when asking Claude to develop a skill from a task.) Sonnet came up with three solutions, the first of which was to let it rewrite the skill with more neutral descriptions like “security industry news.” Fair enough, but I did the editing myself.

Then I went back to the original Claude Code session. It still didn’t work. I expected that I’d need to do something to reload the skill, but the problem was worse. Regardless of the prompt, the original session wouldn’t do anything except repeat the error message. It wouldn’t even commit the modified skill to my GitHub repo. However, Sonnet executed my skill correctly in a new Claude Code instance.

So I returned to Sonnet to find out what’s going on. The answer was interesting: The error may have been triggered by the skill, but when evaluating security threats, the models base their decisions on the entire conversation, not just the specific skill that was called. If a model needs to call a skill that it thinks is problematic, that call is part of the conversation, part of the context. The entire conversation is then forever dead and lost.

What can we learn from this? First, it’s a problem for a program to stop working because of a change over which you have no control. If anything, the industry has erred on the other side; we’re all familiar with “we don’t really understand why this works, so don’t touch it, don’t update the compiler, don’t update the libraries, and run it on emulators of computers that haven’t been built in 40 years.” That’s not just a problem for COBOL code from the 1970s; we see the same thing with C, C++, Java, JavaScript, and just about every language that ever went into production. Legacy code is everywhere. The “don’t change anything” approach isn’t necessarily a bad thing; it certainly beats “here’s a new library, you’re going to love it, you can’t use the old version any more, and wow, look at all the things it broke, guess you’ll have to fix them.” AI where working code breaks at random is a lot less useful than AI that works day in and day out. Stability is a virtue. It’s impossible to work effectively when the environment changes from day to day and isn’t under your control.

But that’s not really what bothers me. It’s rather bizarre that reading well-known sources is treated as a security risk, especially when the “risk” seems to come from an AI-generated description. Of course, we know about hallucinations, errors, and prompt injections. The possibility of a Hacker News post that injects a hostile prompt isn’t zero, and it’s also possible that a model might mistakenly interpret an example of a hostile action as a prompt. I also don’t expect any model to reason that a skill must be safe because it’s been in use for months (though files have time stamps). Artificial intelligence always coexists with artificial stupidity, as does natural intelligence.

Guardrails may keep you from going off a cliff, but they may also prevent you from going where you need to go. And that’s a problem. There’s a basic concept from signal processing and data science called the receiver operating characteristic (ROC). In any binary classification system, you can never achieve perfect classification. The only way to guarantee that no true positives (dangerous things) slip through the classifier is to reject everything. The opposite is equally true: The only way to eliminate false positives (things that look dangerous but aren’t) is to let everything through, including dangerous actions. In theory, it’s possible to get arbitrarily close to perfect classification, but you know how that goes: “The difference between theory and practice is bigger in practice than in theory.”

The ROC curve. (This figure is from Wikimedia Commons and licensed under Creative Commons Attribution-Share Alike 4.0 International.)

We know how to make AI “safe”: Go back to 2022 and models that can only tell the difference between cats and dogs. The model might mislabel a few things, but the consequences of an error are small. Safety comes with limitations, and none of us who use AI for real work want to return to the days of dogs, cats, and bananas. And while I don’t want the ability to use Claude to generate hostile attacks against unsuspecting victims, and while I understand the danger of interpreting any input text as a command (for example, an article describing the Morris worm), I have a problem with an AI that refuses to perform reasonable tasks. The ROC tells us that we can’t have perfect guardrails, but there’s no rule against overly fussy ones. What’s allowed, and what’s forbidden? What are the limits? We don’t know. And that’s the situation we’re in now. We can’t know in advance what is and isn’t acceptable, and the rules can change at any time. A tool with unknown limitations is much less useful than a tool that tells you what it can and can’t do. I’ve enjoyed using Claude to write programs that play with prime numbers and infinite series, and fortunately I don’t rely on any of those programs for my job. But what if tomorrow (or a month from now or a year from now) Claude decides that testing whether large numbers are prime signals an attack against cryptography?

I’m not completely unsympathetic to scoring an entire conversation rather than individual actions. A series of steps, each of which appears innocuous by itself, is more likely to lead an agent to a hostile action than a single prompt. But again, given how valuable context is, do we really want the penalty to be losing all the context for an innocuous project? There are risks on either side, including the possibility that a model will ignore its guardrails; after all, rules that a harness adds to the context are at best advisory.

Guardrails always have unintended consequences. We need to learn what the ROC is teaching us: that it’s impossible to get to the upper left corner of the diagram, where we have perfect rejection of true positives (dangers) and no rejection of false positives. But we also need to get as close to that upper left corner as possible if we want our classifiers to have consistently useful output. An engineering team needs to balance risk against usefulness, and they’re clearly out of balance now. Risks will never go away, but guardrails whose boundaries are unclear and overly strict lead to models and agents that are less useful, rather than more. The bad guys will always figure out how to do bad stuff. Hamstrung AI for the rest of us is not a solution.