The Four Questions I Ask Every AI Vendor Before Signing
I am not the engineer. I am the one who signs the vendor contract, and the one who has to answer for it later. These are the four questions I ask before my signature goes on anything.
In my last post I wrote that my job is to defend the AI we use, not to build it. A few people asked the obvious follow-up: so what do you actually ask a vendor? Here is the honest answer. Four questions. None of them are technical. All of them are about one thing — when someone with authority asks what your model did, can we prove it, or are we repeating what you told us.
This stopped being abstract this year. In the United States, insurance regulators in twelve states have begun examining how carriers use AI, and the framework they are using makes no distinction between AI a company builds and AI it buys. Accountability stays with the company either way. In Canada, the same logic is arriving through OSFI and Quebec's own rules on automated decisions. The pattern is the same everywhere: the buyer answers for the vendor's model. So the contract is where I do my real risk management, and these four questions are how I get there.
The four questions
Can you tell me which version of your model produced a specific result, months after it happened? Not "do you log things." Everyone logs things. I mean: if a regulator points at one decision from last quarter, can you tell me exactly which model version generated it. If the answer involves "we would have to check," the answer is no — and I now know what I am signing up to defend.
If I have to reproduce a decision for an examiner, what exactly can you hand me? A responsible-AI summary is not an answer. A SOC 2 report is not an answer to this question either; it tells me your controls exist, not what your model did on my data. I want to know what concrete artifact lands on my desk when I have to show my work. If the only artifact is the vendor's own assurance that everything was fine, I am defending your word, not evidence.
Is your record of what the model did something I could independently check, or do I have to trust your copy of it? This is the one most vendors have not thought about. If the only record of what happened is the one the vendor keeps and can edit, then in an audit I am asking a regulator to trust a log written by the party with the most reason to write it favorably. An independently checkable record is a different category of thing, and the difference is the whole point.
Will you put audit rights, model documentation access, and regulator cooperation in the contract, in writing? Goodwill is not a clause. If a vendor is willing to cooperate with an examiner, they should be willing to write it down. The conversation about adding these terms takes months, and an examiner's timeline does not pause while my legal team negotiates. So I would rather have the uncomfortable conversation before signing than discover the gap when an inquiry is already on my desk.
What I have learned asking these is that the answers sort vendors very quickly. The strong ones engage with question 3 and understand immediately why "trust our log" is not the same as "here is proof." The ones who get defensive, or who keep redirecting me to their accuracy benchmarks, are telling me something useful too.
You do not need to be technical to do this well
None of this requires me to read the model's code. That is the part I want other operators to hear. You do not need to be technical to do this part of the diligence well. You need to know the difference between a claim and a proof, and you need to be willing to make the vendor put the proof in the contract. That is an operations skill, not an engineering one, and right now it is one of the most valuable things a compliance or procurement lead can bring to the table.
If you only adopt one of these, make it question 3. Everything else follows from it.