Guide · Credit & capital markets

Why AI Hallucinates on Financial Decisions, and How to Engineer It Out

AI systems hallucinate on financial analysis because they commit to one line of reasoning and then judge that reasoning against criteria they chose themselves. Retrieval reduces the problem but does not remove it: the model still decides what the retrieved numbers mean. The fix is structural. It takes independent verification, evolving evaluation criteria, and a preserved reasoning record an auditor can inspect.

In brief

A hallucination in finance is rarely an invented company. It is a real number used in the wrong place, a ratio applied to a business model it does not fit, or a confident answer to a malformed question.
Retrieval-augmented systems still fail this way. On FinanceBench, using real questions against real SEC filings, a retrieval-augmented baseline (GPT-4-Turbo with a shared vector store) answered roughly 19% correctly (Islam et al., 2023).
FINRA’s 2026 Annual Regulatory Oversight Report added a dedicated generative-AI section and named hallucination explicitly as a risk firms must test for and manage. It signals examination priorities. It does not create new rules.
The controls that work are the ones that separate the thing producing an answer from the thing judging it, and that keep a record of both.
No control reaches zero. Any vendor telling you otherwise is selling you the failure mode.

What a hallucination actually looks like in a credit or trading context

The public conversation about hallucination is dominated by the wrong example: the fabricated legal citation, the invented quotation. Those are easy to catch, because they do not exist and a search proves it.

Financial hallucinations are worse because every component of them is real.

A model reads a 10-K correctly, extracts a genuine figure, and applies it to the wrong line item. Every number in the output is a real number from a real filing. The arithmetic is correct. The answer is wrong, and nothing in the output is falsifiable by inspection. You have to redo the work to find it.

A subtler version: the model answers a question that should not have been answered. Ask what drove the change in gross margin at a financial services company and a single-pass system will compute something, because it was asked to. Financial services firms have no cost of goods sold, so gross margin is not a meaningful figure for that business model. The correct answer is that the question is malformed, and a system optimising for helpfulness will almost never give it.

FINRA’s 2026 report frames the general case in terms a compliance officer recognises: output that is inaccurate or misleading but presented as fact. In a credit file or a desk memo, “presented as fact” is the entire problem. The output arrives in the same format, the same tone, and the same confidence as a correct one.

The operational definition worth adopting: a financial hallucination is any output whose confidence is not earned by its evidence. That definition covers fabrication, misapplication, and unwarranted certainty, and it is the only one that maps cleanly onto what a model-risk function is actually required to control.

Why it happens: three mechanisms, not one

Mechanism	What it is	Why prompts and bigger models don’t fix it
Trajectory lock	Every token is conditioned on the framing chosen in the first sentence.	A bigger model commits more fluently, producing a more persuasive wrong answer.
Self-graded evaluation	Asked to check itself, the model grades against criteria from the same framing.	It passes its own arithmetic and consistency checks while missing the framing error.
Unknown criteria	The dimension that matters is often the one nobody thought to list.	A rubric fixed in advance cannot contain a criterion no one knew to write.

1. Trajectory lock

A language model produces an answer by committing, early, to a framing, and every subsequent token is conditioned on that commitment. If the first move treats a diversified industrial as a single business, everything downstream inherits the aggregation error. The model cannot step outside the framing it chose in its first sentence, because the framing is now its context.

This is not a prompting problem. Better prompts move the starting point; they do not create the ability to abandon it mid-flight. Nor is it fixed by a larger model. A more capable model commits to its framing more fluently, and produces a more persuasive wrong answer.

2. Self-graded evaluation

Ask a model to check its own work and it checks against criteria it generated. Those criteria come from the same framing that produced the error. A system that aggregated a diversified business will evaluate its answer for internal consistency, arithmetic, and completeness, and pass itself on all three, because segmentation was never on the list.

This is the mechanism most reliability tooling misses. Guardrails, groundedness scoring, and output validators all check the answer. Very few check whether the question of what counts as a good answer was framed correctly in the first place.

3. The criteria you didn’t know to specify

Every evaluation rubric is written by someone who already knows what to look for. In well-understood tasks, that is fine. In financial analysis, the dimension that matters most is frequently the one nobody thought to list: Simpson’s paradox in a segmented business, a reconciliation break, a metric that does not apply to this issuer’s model at all.

A system that can only be graded on criteria specified in advance is structurally incapable of catching those. It will score highly and be wrong.

Why retrieval doesn’t fix it

Retrieval-augmented generation is the standard answer to hallucination, and it genuinely helps. Grounding a model in source documents removes a large class of fabrication.

It does not remove the three mechanisms above. Retrieval governs which numbers the model sees. It does not govern which number the model chooses, what it decides the number means, or whether it should have been used at all. The model still selects the line item. The model still frames the comparison. The model still grades itself.

FinanceBench is the cleanest public evidence that RAG alone is not enough. It tests models on real questions against real SEC filings, exactly the workload where retrieval should be most effective, and a retrieval-augmented baseline (GPT-4-Turbo, shared vector store) answered only about 19% correctly. Part of that is retrieval itself: finding the right page in a shared store is hard, and the paper identifies retrieval as a primary bottleneck. But retrieval is only the first half of the job. The second half, choosing the right figure, framing the comparison, and grading the result, is left entirely to the model.

The load-bearing conclusion: if your reliability strategy is “we use RAG,” you have addressed the easiest failure mode and none of the expensive ones.

What regulators are now asking for

Three things have converged, and together they are the reason this question moved from research to procurement in 2026.

Framework	What it asks of you	New rules?	In effect
FINRA 2026 Oversight Report	Assess obligations before deploying generative AI; governance; hallucination and bias controls; supervise AI agents.	No. Signals exam priorities.	Dec 2025
SR 11-7 (model risk)	Sound development, effective validation, governance, applied to a stochastic model.	Already in force.	Pre-existing
EU AI Act (high-risk)	Data governance, technical documentation, human oversight, record-keeping. Credit scoring is a named high-risk application.	Yes, though the deadline is moving.	2 Aug 2026 → 2 Dec 2027 (proposed)

On FINRA: the 2026 report, published on 9 December 2025, added a standalone generative-AI section for the first time. It expects firms to assess their compliance obligations before deploying generative AI, to establish governance frameworks, and to build controls addressing hallucination and bias specifically. The nuance most vendor content gets wrong: the report does not create new binding rules. It tells you what examiners will prioritise.

On SR 11-7: the US supervisory guidance on model risk management already applies. It predates all of this and was written for quantitative models, which is exactly the difficulty. Its three pillars assume a model whose behaviour can be specified and re-tested. Applying that framework to a stochastic, general-purpose system is the open problem model-risk functions are working through. “It’s an LLM, SR 11-7 doesn’t apply” is not a position that survives an examination.

On the EU AI Act: it names credit scoring as a high-risk application, carrying obligations for data governance, documentation, human oversight, and record-keeping, with penalties up to €15 million or 3% of global turnover. The original 2 August 2026 deadline for standalone high-risk systems is being pushed to 2 December 2027 under the EU’s Digital Omnibus package. The date is moving; the obligations are not.

The practical consequence is the same in all three: you must be able to show your work. Not a confidence score. Not a post-hoc explanation generated after the fact by the same system that produced the answer. A record of what was considered, what was rejected, and on what basis.

Six controls that measurably reduce hallucination

Ordered by how much they change outcomes, not by how easy they are to buy.

1. Separate the producer from the judge

The single highest-value structural change. The component evaluating an answer must not have access to the instructions, persona, or framing that produced it. If it does, it inherits the blind spot.

This is harder than it sounds and easy to fake. Running a second prompt on the same model with “now critique this” does not achieve it, because the critique is conditioned on the answer’s own framing. Genuine independence means the evaluator cannot see how the answer was asked for.

2. Let the evaluation criteria change

Fixed rubrics catch known failure modes. In financial analysis, the expensive failures are the unknown ones. A system that can add a criterion it discovers mid-analysis, such as “check whether this metric applies to this business model,” catches errors that no static rubric would have.

In practice this means evaluation is not a gate at the end of the pipeline. It is a thing that develops as the analysis develops.

3. Force the same question through different framings

If one framing produces one answer, run the analysis again from a genuinely different analytical position (liquidity, correlation structure, tail risk, regime change) and compare. Agreement across independent framings is weak evidence of correctness. Disagreement is strong evidence that something needs a human.

The output of this is not a number. It is a range and a set of conditions under which each framing holds.

4. Preserve the reasoning record, not the explanation

An explanation is generated after the fact and is subject to the same failure modes as the answer. A reasoning record is what was actually explored, including the paths abandoned and why. That distinction matters enormously for audit.

Post-hoc explanation	Preserved reasoning record
Generated after the answer, often by the same system.	The trail captured as the work happened.
Inherits the answer’s blind spots.	Includes the paths tested, failed, and corrected.
Fluent, but nothing a validator can verify.	Evidence a validator or auditor can inspect.

5. Penalise unearned confidence explicitly

Most systems are optimised, directly or indirectly, to sound useful. In a credit context that optimisation is actively dangerous. The correct output is often “insufficient evidence to conclude,” and nothing in a standard training or evaluation loop rewards that. Whatever scores your outputs must be able to mark an answer down for asserting more than its evidence supports, not just for being wrong.

6. Test with adversarial and malformed inputs

Your evaluation set should include questions that should not be answered: a ratio that does not apply to the issuer’s model, a comparison across incompatible periods, a question whose premise is false. A system’s behaviour on malformed questions tells you more about its production risk than its accuracy on clean ones. Most vendor benchmarks contain no malformed questions at all, and that is the gap where your losses will come from.

What we found running this ourselves

We built ThinkerWave around the first four controls above, and ran it on FinanceBench, five questions against real SEC filings for 3M, AES, American Express, American Water Works, and Johnson & Johnson. Our runs were given the relevant filing, so read this as a reasoning result on supplied evidence, not a like-for-like retrieval comparison.

Result: four of five correct, self-evaluated, on five questions, with the limits stated below.

Company	Question	Result	What happened
3M	Capital intensity	Correct	Computed the ratios in a sandbox rather than estimating
J&J	High-growth classification	Correct	Took five generations of evolving financial judgment
American Water Works	Working capital	Correct	One pass, 163 seconds, every line item cited
AES	Inventory turnover	Incorrect	Used the wrong COGS line item
American Express	Gross margin drivers	Correct	Determined the metric was inapplicable and said so

The two results worth your attention are the last two, and one of them is the failure.

The AmEx question asked what drove changes in gross margin. American Express is a financial services company. It has no cost of goods sold, which makes gross margin meaningless for it. The system worked that out, after several generations of developing financial judgment, and declined the metric. No seed criterion, prompt, or instruction told it to check metric applicability. It arrived at that check by accumulating failures.

The AES failure is the honest counterweight. It selected the wrong line item for cost of goods sold. This is exactly the class of error described at the top of this page: real numbers, correct arithmetic, wrong answer, invisible without redoing the work. Our system did not catch it. That is the residual risk, and it is why no control list on this page claims to reach zero.

Starting from three generic seed criteria, the system evolved ten finance-specific ones, including these three: investigate root causes of performance changes rather than describing outcomes; segment diversified business performance rather than relying on aggregates; and validate data integrity before using numbers. Nobody specified them. The first is analytical discipline, the second is Simpson’s paradox, the third is audit practice. A system that starts without them and ends with them is doing something a fixed rubric cannot.

What this evidence is not. Five questions is a small sample. The scoring was done by our own evaluator, which is a language model of the same family and may share its biases. There is no external expert validation of these results, and no customer case study behind them yet. We publish them with those limits stated, because a benchmark result without its sample size and its grader is a marketing number, not evidence. This is a field where you should be asking every vendor, including us, for both.

How to test any AI system for financial reliability before you deploy it

A protocol a risk function can run in a day, against any vendor.

#	Do this	What it reveals
1	Build a 20-question set from your own documents, not public benchmarks.	Whether accuracy holds outside what the vendor optimised for.
2	Include four malformed questions (a ratio that doesn’t apply, a false premise, an unanswerable one).	Whether the system refuses or invents.
3	Run each question three times.	Variance across identical runs, itself a risk finding.
4	Score three columns separately: correct, confidently wrong, correctly refused.	Confidently wrong is your exposure; correct refusals predict production safety.
5	Ask for the reasoning record on one wrong answer.	Whether you can evidence the system to a validator, or only get a fluent paragraph.
6	Ask what the evaluator is and whether it is independent.	Whether “producer” and “judge” are two systems or one with two labels.

Run this before procurement, not after. It takes a day and it disqualifies more vendors than a six-week pilot.

Frequently asked questions

Can AI hallucination be eliminated entirely?

No. There is formal work arguing hallucination cannot be fully eliminated from large language models, and the practical evidence agrees. The achievable goal is to make errors detectable and the reasoning auditable, not to reach zero. Treat any claim of hallucination-free AI as a disqualifying signal.

Does FINRA’s 2026 report create new obligations for firms using generative AI?

No. It is an annual report of observations and effective practices, and it states that it does not establish new legal or regulatory requirements. What it does is signal examination priorities for 2026, including hallucination controls, testing and monitoring, governance frameworks, and supervision of AI agents.

Does SR 11-7 apply to large language models?

Model risk management guidance was written for quantitative models, but supervisors have not carved out generative systems. Firms should assume the three pillars, sound development, effective validation, and governance, all apply, and should expect validation to be the hardest of the three to evidence for a stochastic system.

Is retrieval-augmented generation enough for regulated financial use?

It addresses fabrication and leaves reasoning errors intact. Benchmark evidence on real SEC filings shows retrieval-augmented systems still failing a large share of questions even when the source documents were available to them. RAG is a necessary component, not a reliability strategy.

What’s the difference between explainable AI and an auditable reasoning record?

An explanation is generated after the answer, often by the same system, and inherits its blind spots. A reasoning record is the preserved trail of what was actually explored, tested, rejected, and corrected. Validators and auditors need the second; most tooling produces the first.

ThinkerWave is an autonomous reasoning system for problems where the evaluation criteria are not known in advance. It runs on your own machine against your own model credentials. Indian Patent Application #202611044024. Currently in invite-only early access.