Someone on your team is about to sign an AI vendor contract. Pricing, SLA, data residency, SOC 2 report, done. Nobody in the room asked whether the model underneath the product does what the vendor claims when a stranger talks to it.
I keep seeing this. Engineering leaders who would grill a database vendor for two hours about replication lag will wave through an AI feature because the demo looked slick. The demo is the one environment where the thing behaves.
Jailbroken means something specific
People use "jailbroken" loosely, so let me be precise about what worries me.
A model is jailbroken when an attacker gets it to ignore its instructions. Your careful system prompt says "never reveal other users' data, never call the refund tool without approval." An attacker writes something clever into a support ticket, a CV, a calendar invite, a web page your agent reads... and your instructions lose.
The OWASP Top 10 for LLM Applications puts prompt injection at number one. LLM01. Above sensitive information disclosure, above supply chain, above data poisoning. The industry's own risk list says the first thing to worry about is your model taking orders from someone who is not you.
Number six on the same list is Excessive Agency. Give a model tools, a database connection, an email account, and a jailbreak stops being embarrassing output and starts being an action with consequences.
Two years ago your AI feature wrote text. Now it writes to systems. The blast radius changed. Most procurement checklists did not.
The numbers worth knowing
Here is the one statistic I hand to founders who think this is hypothetical.
Anthropic published results on its Constitutional Classifiers work. Against an unguarded model, the jailbreak success rate was 86 percent. With the first generation of classifiers in front of it, 4.4 percent. Same model. The difference is entirely the defensive layer wrapped around it.
Sit with 86 percent for a second. A raw model with no guard layer loses to a determined attacker almost every time. Your vendor's product sits somewhere on the line between 86 and 4.4, and you have no idea where, because you never asked.
The cost side got worse too. IBM's 2026 Cost of a Data Breach report puts the global average breach at 4.99 million dollars, up 12 percent and a record. It logs a 56 percent rise in AI-driven attacks. Model inversion attacks, where someone pulls training data back out of a model, average 6 million.
Your AI feature is not a feature any more. It is an attack surface with a monthly invoice.
The lesson from the number plate cameras
Something landed on Reddit this week worth borrowing.
404 Media and the EFF went through the audit logs of Flock, the number plate camera network used by US police forces. Officers have to type a reason for each search. The logged reasons include "LMAO", "IDK", "Hehe" and "asdfg". Hundreds of thousands of searches, across 2023 to late 2025, on a system marketed on its accountability controls.
The control existed. The field was mandatory. Nobody read the output.
I have sat in vendor calls where the answer to every security question was "everything is logged." Logging is not a control. Logging plus someone whose job includes reading the log is a control. Ask which one you are buying.
Five questions before you sign
Take these into the next vendor call. None of them need a security background to ask, and the quality of the answer tells you more than the SOC 2 PDF does.
1. Which model is under the product, and what happens when it changes?
If the vendor will not name the model, walk. If they name it, ask what their process is when the provider ships a new version. Model updates move behaviour. A prompt hardened against last quarter's model is an assumption, not a defence.
2. What sits between the user input and the model?
You are asking whether a guard layer exists. Input filtering, output classification, a separate moderation pass. "The model is safe by design" is a marketing answer. The 86 percent figure is what "safe by design" looks like when someone attacks it.
3. What tools does the model hold, and who approves the dangerous ones?
Read access, write access, payments, emails, deletions. The right answer names specific tools and names a human approval step on the destructive ones. A vague "it works autonomously" is a vendor telling you they built the blast radius and hoped.
4. When did you last red team this, and who did it?
Internal testing is worth something. External testing is worth more. If nobody has run garak or a comparable scanner against the product, you are their red team and your customers' data is the test corpus.
5. If prompt injection happens tomorrow, how do I find out?
This is the Flock question. Not "do you log" but "what alert reaches which human, in what timeframe." If the answer is a dashboard nobody has opened since onboarding, you have bought the mandatory reason field.
Why this lands on leadership, not on the security person
Every one of those five questions is answerable by a non-technical leader. None of them need you to read a research paper. So why do they go unasked?
Because asking feels like admitting ignorance, and most leaders would rather look confident in a vendor meeting than look uninformed in front of their own team.
I have written before about how bad bosses get made, and the pattern repeats here. My own research found 99.5 percent of respondents had worked for one or more types of bad boss. A recurring thread is a leader who stops asking questions in public because they are frightened of the answer being over their head.
You do not need to be an AI expert. You need to be the person in the room willing to say "I do not understand how this fails... walk me through it." The team watching you ask learns it is safe to ask too. Do the opposite and you get a procurement process where everyone nods at a demo and nobody owns the risk.
Engineers spot this instantly. They know when a decision got made on vibes. What follows is quiet disengagement from the people best placed to catch the problem before your customers do.
What I do
I build on these APIs, so I am not pointing from the sidelines.
My rules are boring. Name the model version and pin it. Assume any text reaching the model is hostile, including text from my own database, because someone else put it there. Give agents the narrowest tool access the job needs. Keep a human in front of anything destructive or expensive. Read the logs on a schedule, not after an incident.
None of this is clever. It is the same defensive posture we learned for user input twenty years ago, applied to a component with better manners and worse boundaries.
The vendors selling you AI features are moving fast because their investors demand it. Speed is fine. Speed with nobody asking the five questions is how a support chatbot ends up refunding a stranger's order, or reciting an employee's grievance file back to the person it was about.
So, next vendor call. Who in your organisation is going to ask question three... and what happens to them if the answer is bad?