Guide · Credit & capital markets
How to Test an AI System for Financial Accuracy Before You Deploy It
Test a financial AI the way model risk tests any model: on your own documents, with adversarial and malformed questions, repeated runs, and three separate score columns for correct answers, confidently wrong answers, and correct refusals. The column almost nobody measures, correct refusals, predicts production safety better than raw accuracy does. Below is a protocol a risk function can run in a day, and our own system’s results run through it, the failure included.
In brief
- Public benchmarks are the wrong test. Vendors optimise for them. Build a 20-question set from filings your own team already knows the answers to.
- Include malformed questions. A system that answers a question it should refuse is the one that will lose you money in production.
- Score three columns, not one: correct, confidently wrong, and correctly refused. Confidently wrong is your exposure. Correct refusals are the safety signal.
- Run each question three times. Variance across identical runs is itself a finding.
- Ask for the reasoning record, and ask whether the evaluator is independent. If it is the same model prompted to grade itself, you have one system with two labels.
Why accuracy on a public benchmark tells you almost nothing
A headline accuracy figure answers a question you did not ask. It tells you how a system performed on a fixed, public set of questions that the vendor has almost certainly seen, measured, and tuned against. It does not tell you how the system behaves on your filings, your issuers, and the specific ways your analysts phrase a question.
Worse, a single accuracy number hides the distinction that matters most in a regulated setting. A system that is right 80% of the time and refuses the other 20% is safe. A system that is right 80% of the time and confidently wrong on the other 20% is a liability. Both report the same accuracy. Only one belongs in a credit file.
The protocol below is designed to surface exactly what the headline number hides. It takes a day, it uses your own documents, and it works against any vendor.
The protocol
1. Build a twenty-question set from your own documents
Not a public benchmark. Pull twenty questions from filings and files your team already knows cold: a working-capital figure you have reconciled, a segment margin you have argued about, a covenant calculation you have signed off. You need the ground truth in hand, because the whole exercise depends on you knowing the right answer before the system gives you one.
2. Include four malformed questions
This is the step most evaluations skip, and it is the one that predicts production risk. Seed the set with questions that should not be answered cleanly: a ratio that does not apply to the issuer’s business model, a comparison across restated or non-comparable periods, a question whose premise is false, and a question the source documents genuinely cannot answer. What you are testing is not whether the system is smart. It is whether the system knows the edge of what it can support, and stops there.
3. Run each question three times
Ask the same question three times, in separate sessions. A deterministic tool gives the same answer three times. A stochastic one may not, and the variance is itself a risk finding you must be able to quantify before deployment. It is also the number vendors are least prepared to discuss, which tells you something in itself.
4. Score three columns, separately
For every answer, record which of three columns it lands in: correct, confidently wrong, or correctly refused. Do not collapse them into an accuracy percentage. The confidently wrong column is your actual exposure, the answers that look right, read fluently, and are false. The correct-refusals column is the one almost nobody measures, and it is the best predictor of whether a system is safe to run unsupervised.
5. Ask for the reasoning record on one wrong answer
Pick an answer the system got wrong and ask to see how it got there. If what comes back is a fluent paragraph composed after the fact, you have an explanation, not evidence, and you will not be able to defend the system to a validator. If what comes back is a record of what was tried, what failed, and what was corrected, you have something a model-risk function can actually work with.
6. Ask what the evaluator is, and whether it is independent
Any system that claims to check its own work has an evaluator somewhere. Ask what it is. If the answer is “the same model, prompted to critique itself,” the check inherits the blind spot that produced the answer, and you have one system wearing two labels. Genuine independence means the component grading the answer cannot see how the answer was asked for.
The scorecard
One row per question. The columns you care about are the last three, and the one to read first is the middle one.
| Question type | Correct | Confidently wrong | Correctly refused |
|---|---|---|---|
| Answerable (16) | Your accuracy signal | Your exposure | Over-caution, worth noting |
| Malformed (4) | n/a | The dangerous cell | The safety signal |
A system you can deploy has an empty “confidently wrong” column on the malformed row. A system that fills that cell will, sooner or later, put a confident and false number in front of a decision that matters.
We ran it on ourselves
It would be a poor guide to a testing protocol that did not survive its own test, so here is ThinkerWave run through a version of it. We used five FinanceBench questions against real SEC filings for 3M, AES, American Express, American Water Works, and Johnson & Johnson. The runs were given the relevant filing, so this is a reasoning result on supplied evidence, self-evaluated, on a small sample, with the limits stated below.
| Company | Question | Result | Column it lands in |
|---|---|---|---|
| 3M | Capital intensity | Correct | Correct |
| J&J | High-growth classification | Correct | Correct |
| American Water Works | Working capital | Correct | Correct |
| American Express | Gross margin drivers | Correct | Correctly refused the metric |
| AES | Inventory turnover | Incorrect | Confidently wrong |
Read it by the columns from the protocol, not by the four-of-five headline. The American Express row is a correct refusal: asked what drove gross margin at a company with no cost of goods sold, the system determined the metric did not apply and declined it. The AES row is the one that matters most, and it is the failure. The system picked the wrong line item for cost of goods sold and reported a confident, wrong number, the exact cell a deployable system must keep empty. It landed in “confidently wrong.” We did not catch it in the run. That is the residual risk, and it is why we publish the failure next to the successes rather than behind them.
What this evidence is not. Five questions is a small sample, the grading was done by our own evaluator, and there is no external validation or customer case study behind it yet. Run the protocol on us with your own documents, and hold every vendor, including us, to the same three columns.
What a system worth deploying looks like
After a day with this protocol, a system worth a pilot will show:
- A near-empty “confidently wrong” column on the malformed questions.
- Correct refusals where refusal is the right answer, stated with the reason.
- Low variance across the three repeated runs, or an honest account of the variance.
- A reasoning record you can hand to a validator, not a paragraph written after the fact.
- An evaluator that is genuinely separate from the thing it grades.
None of these require you to trust a vendor’s benchmark. They require a day, your own documents, and three honest columns.
Frequently asked questions
Why not just use a public benchmark score?
Because vendors optimise for public benchmarks, and a high score on questions the vendor has seen tells you little about behaviour on your documents. Build a set from filings your own team already knows the answers to. Treat any headline accuracy number as a starting question, not an answer.
What is the single most important thing to measure?
Correct refusals. A system that answers a malformed or unanswerable question with a confident number is the one that loses you money in production. Most evaluations never score refusals at all, which is why they miss the failure mode that matters most in a regulated setting.
How many questions do I need?
Twenty is enough to be useful in a day, including four malformed ones. The point is not statistical power. It is to observe how the system behaves on questions where you already know the truth, and on questions that should be refused.
What should the reasoning record look like?
A record of what was actually explored, tested, rejected, and corrected, not a fluent paragraph generated after the answer. If a validator cannot trace how a conclusion was reached, the system cannot be evidenced under model risk governance.
This is part of a series on reliable AI for credit and capital markets. Start with the pillar: Why AI hallucinates on financial decisions, and how to engineer it out.
ThinkerWave is an autonomous reasoning system for problems where the evaluation criteria are not known in advance. It runs on your own machine against your own model credentials. Indian Patent Application #202611044024. Currently in invite-only early access.
Try it on your own problem
ThinkerWave works a hard problem from several angles, checks its claims against the evidence, and keeps a full record of how it got there. It runs on your machine, with your own model keys.