Have you ever told an AI 'never do this' and watched it do it anyway?

I have. And the worst part is, the AI thought it was being helpful. I'd been worried about this for a while. Whenever you give an AI agent a tool that can write to something — your billing system, your customer database, whatever — the safety story usually goes: “I told it the rules in the system prompt. I gave it 16 rules. It'll be fine.” It's mostly fine. Until it isn't. The thing that bothers me about prompt rules — the rules you write into the system prompt — is that they're aspirational. Th