AI Answers Without Error Bars

On September 18, 2026, CNN reported, citing four sources familiar with the episode, that a false intelligence report circulated through the US military this spring, during the war with Iran. The report claimed that a Chinese ship in the Middle East was carrying components for a nuclear weapons programme. Forces moved to intercept the vessel. According to two of the sources, armed personnel were preparing to board it and military aircraft were in the air.¹

CNN reports that it was “only just before the planned operation” that officials dug deeper into the report, which had been put together by a special operations analyst, and found it had been generated with the help of AI. A chatbot the analyst had used “inaccurately identified the material the ship was carrying” after fusing open-source information with classified signals intelligence. One source called the report “entirely false” and said it “almost started a war.” Another offered a sentence that deserves a place in every AI governance briefing: “AI allows you to get to a bad idea faster.”¹

What is not known matters as much. The account rests on anonymous sources and has not been officially confirmed; CNN reported that Special Operations Command Pacific and the Pentagon did not respond to its request for comment. The chatbot has not been identified, and CNN could not establish whether it was a commercial product or a government one. The reporting does not describe what the chatbot’s answer looked like, whether it carried caveats or cited sources, whether the finished report was marked as AI-assisted, or where the error arose: in the model, in the underlying data, in the query, or in the review that followed. CNN also reported that there is no single set of standards for how the US military verifies information generated by these tools.¹ Nothing in this Insight attributes the incident to any named company or model.

This Insight therefore offers no diagnosis of the incident. It uses the incident to pose a question whose answer does not depend on the missing details: when an AI system hands an analyst a claim, what should accompany it? On that question the public record is clear. In their default form, today’s general-purpose AI assistants deliver a claim as prose and attach no quantified statement of how likely it is to be true.

A claim without a probability is an assertion. A probability with no statement of its own reliability is an incomplete measurement. Frontier AI assistants from OpenAI, Anthropic, Google, and Meta deliver the first. One new entrant delivers the second, for a narrower class of question. No one yet delivers what a scientist would call a result, and high-stakes decisions should be governed accordingly.

What should accompany a claim

Intelligence tradecraft already separates the questions involved. The US Intelligence Community’s analytic standards, set out in Intelligence Community Directive 203, require analysts to express the likelihood of a judgment on a defined scale running from “almost no chance” to “almost certain,” and to describe separately their confidence in the sources and reasoning behind it. The directive keeps the two apart on purpose: products must not combine a confidence level and a degree of likelihood in the same sentence.² Confidence in this sense is a qualitative judgment about the quality of sourcing and analysis. It is not a statistical quantity, and this Insight does not treat it as one. The principle carries over all the same: how likely a claim is, and how solid the basis for that estimate is, are different questions that need different answers.

We do not know what the chatbot in the reported incident displayed. We do know what general-purpose chat assistants produce by default: fluent prose in which a well-supported statement, a weak inference, and a fabrication read alike. Once such a sentence is pasted into a report, even its machine origin can disappear unless someone records it.

“Keep a human in the loop” is the usual remedy. It is incomplete. A human reviewer who sees only a confident sentence has little to review. Oversight needs an input: some quantity that tells the reviewer where to spend scepticism.

Four quantities that should not be confused

Discussion of AI “uncertainty” often moves between several different quantities as if they were one. They answer different questions, they are measured in different ways, and a system can do well on one while failing another. The rest of this Insight keeps them apart.

Metrology’s distinction between precision and trueness applies. Quantity 2 measures precision: whether the instrument gives the same reading twice. Quantities 3 and 4 bear on trueness: whether the readings are right, here. A system can be stable and wrong. Several things that are sometimes offered in place of these quantities are none of them: the probability of the next token, a confidence the model states when asked, a benchmark accuracy rate, and agreement among repeated samples. Each is discussed below.

What frontier models provide today

As of this writing, and to our knowledge, none of the general-purpose assistants from OpenAI, Anthropic, Google, or Meta attaches a probability of truth to the statements it makes in its default interface, and none documents such a feature. What exists falls into three categories. None supplies quantity 1, and so none reaches quantities 2 to 4.

• Token probabilities. Some developer interfaces expose the log-probability of each generated token, and openly released model weights allow anyone to compute them. Other interfaces expose nothing of the kind. In every case this is the probability of the next piece of text given the preceding text. It is a statement about wording. The same true claim can be worded in many ways, each with a low token probability, and a false claim can be worded in a highly probable way. It is not quantity 1.
• Confidence on request. A user can ask a model how sure it is, and it will answer. Xiong and colleagues evaluated this practice across several models and found that verbalised confidence tends to be overconfident, clustering at high values in a way that resembles how people talk about confidence rather than how often the model is right.⁵
• Aggregate benchmark rates. Developers publish hallucination and accuracy rates on test sets. These are population statistics. A model that is wrong 3% of the time on a benchmark says nothing about whether the sentence in front of the analyst belongs to the 3%, and nothing at all if the analyst’s question lies outside what the benchmark covered.

The research record shows that the raw material exists. Kadavath and colleagues at Anthropic found in 2022 that large language models are, in many settings, well calibrated when asked to assess whether their own proposed answers are true, while noting weaker performance on tasks unlike their training distribution.⁶ OpenAI’s GPT-4 technical report showed a pre-trained model whose stated probabilities closely tracked its accuracy, and then showed that the post-training process used to make the model a helpful assistant reduced that calibration.⁷ Farquhar and colleagues demonstrated in Nature that sampling a model repeatedly and measuring the spread of meanings in its answers can flag a class of fabrications: answers that vary arbitrarily from one sample to the next.⁸ The method has a limit the authors note, and it is the precision-and-trueness point again. A model that is consistently wrong agrees with itself and passes the test.

The incentives point the other way. In a 2025 paper, Kalai and colleagues at OpenAI argued that hallucinations persist because most evaluations grade models like students on an exam where a guess can earn marks and “I don’t know” earns none. Models optimised for leaderboards learn to guess. The authors called for changing how mainstream benchmarks are scored.⁹ A year later, public release announcements from frontier developers still lead with capability, autonomy, context length, and speed. We are not aware of any frontier developer that has committed publicly to delivering per-statement probabilities of truth with stated uncertainty. Research on the subject continues inside those organisations, and the problem is hard: a free-form answer contains many claims, and assigning a well-calibrated probability to each remains unsolved at production quality. That difficulty is a reason to restrict how such output is used. It is not a reason to present the output as though the problem did not exist.

One step forward: a probability

On September 15, 2026, a start-up named TypeSafe AI released Jev, which it describes as a “System One” model. Jev does not write text. An application sends it data and typed questions, and it returns structured answers: for a yes/no question, a probability that the statement is true; for a multiple-choice or scale question, a probability distribution over the options. TypeSafe says the model is trained with a method it calls Reinforcement Learning for Calibrated Decisions, aimed at “epistemically honest probabilities,” so that answers given 90% should be right about 90% of the time.¹⁰

This is a meaningful change of direction. It treats the probability as the product. The scope is narrower than a chat assistant’s: Jev answers bounded, typed questions put to it by an application, and it does not attempt the harder task of attaching probabilities to claims inside free-form text. Within that scope it aims at quantities 1 and 3.

It remains one rung short. Jev’s yes/no answer is a single number. The materials we reviewed describe no standard deviation, interval, or other statement of quantity 2. The “confidence” value attached to multiple-choice and scale answers is computed from the shape of the same probability distribution, so it restates the model’s output rather than independently assessing it. TypeSafe’s launch material reports workflow speed and cost comparisons; we found no published calibration curves or third-party calibration assessment as of September 19, 2026.^{10, 11} Those may follow. Until they do, quantity 3 rests on the vendor’s description of its training objective, and quantity 4 is for each buyer to establish in its own domain.

Why a probability needs a statement of its reliability

The Guide to the Expression of Uncertainty in Measurement, the reference document for metrology worldwide, states that a measurement result “is complete only when accompanied by a statement of the uncertainty” of the estimate.¹² The Guide concerns measured physical quantities, and we apply it here by analogy. The analogy is close. Physics journals do not publish a mass without an uncertainty. Polling firms report 52% with a margin of error of three points, conventionally a 95% interval, because the 52% is an estimate from a finite sample and a particular method.

A statistician may object that a probability already is a statement of uncertainty, and that asking for the uncertainty of an uncertainty is one step too many. For an ideal reasoner facing a single bet, the objection holds. A deployed model is not that reasoner. Its number is the output of an estimator, produced from one phrasing of the question, one sampling run, one training history, and one body of evidence. Change any of these, with the world held fixed, and the number moves. How far it moves is quantity 2, and it separates outputs that would otherwise look equal: 0.90 with an interval of 0.87 to 0.93 is a stable estimate, while 0.90 with an interval of 0.55 to 0.98 is a number that could easily have been something else. The Guide’s two routes to evaluating uncertainty both have counterparts here. Repeated observation corresponds to the first three items below; other knowledge about the instrument corresponds to the fourth.

• Rephrasing. Whether the probability survives when the same question is asked in different words.
• Resampling and ensembles. Whether repeated runs, or independently trained models, agree.
• Evidence perturbation. Whether the number depends on a single document or intercept whose removal would reverse the conclusion.
• Range of validity, stated separately. A model can be well calibrated on business classification tasks and carry no guarantee for wartime cargo identification from fragmentary signals. Calibration measured in one domain and relied on in another has the same standing as a laboratory instrument used outside its calibrated range. No interval from the first three items can reveal this, because a system outside its range can be perfectly stable.

These items support a report in the form physics uses for an experimental result: a value, followed by two uncertainty terms of different origin. The first term belongs to the question. It is the spread of the answer to this question, in this context, under the perturbations above, and it corresponds to quantity 2. The second term belongs to the instrument. It is the aggregated uncertainty of the model, established across many tested questions, and it expresses quantity 3 as a number. The Guide’s approach is to correct a result for every recognised systematic effect and then carry the uncertainty of that correction.¹² Applied here, known miscalibration is removed by recalibrating the model, and the model term is what remains: the finite-sample uncertainty of the calibration curve at that probability, together with the variation in calibration across the domains tested. If the two terms are independent they combine in quadrature, so that 0.15 and 0.06 give about 0.16. Reporting them separately is the better practice because they have different remedies. More evidence about this ship narrows the first. Only further testing of the model narrows the second. Quantity 4 governs whether the second term applies at all: outside the tested domain the model term is not larger, it is unknown, and no number should be printed in its place.

The two kinds of number play different roles in a decision, and they should not be merged. Decision theory sets an action threshold from the costs of error: act when the probability exceeds the cost of a false action divided by the combined costs of a false action and a missed one. If wrongly boarding a foreign state’s vessel is judged a hundred times more costly than missing an interdiction, the threshold is about 0.99. That comparison uses quantity 1, and it presumes quantities 3 and 4: a threshold applied to an uncalibrated number is arithmetic on noise. The uncertainty terms then answer a second question, which is whether the decision is robust. If the whole interval implied by both terms lies on one side of the threshold, the choice does not depend on accidents of phrasing or sampling. If the interval straddles the threshold, it does, and the defensible action is to verify before acting. This robustness rule is a governance convention of the kind used in engineering safety margins, and it is offered as such. These figures are illustrative. Nothing in the public reporting indicates whether any explicit probability or threshold figured in the incident CNN describes.

What high-stakes users should do now

For a high-stakes decision, no current AI model’s unsupported statement deserves trust on its own, and a bare point probability deserves only a little more. That is a statement about the form of the output, and it applies however capable the model appears. Until providers close the gap, the organisations deploying these systems have to build the discipline around them.

• Classify unquantified model output as a lead. It can direct collection and verification. It should never enter a decision product as a finding.
• Carry provenance with the claim. Any AI-derived content should remain labelled as such through every report that reuses it, so that reviewers know when they are reading a machine’s sentence.
• Write uncertainty into procurement. Require all four quantities by name: claim-level probabilities, a question-level interval or standard deviation on them with the perturbations used to produce it, a separate model-level term from calibration testing, calibration evidence from the buyer’s own operating domain, and demonstrated abstention outside that domain. Reliability diagrams, expected calibration error, and the Brier score are established tools for checking the third.¹³
• Test stability before deployment. Measure how outputs change under rephrasing, resampling, and removal of individual sources. Instability found in testing costs little. Treat stability as necessary and never as sufficient: a stable answer still has to be checked against ground truth.
• Set thresholds in advance. Decide what probability, with what uncertainty, a given class of action requires, and decide it before the system is in use.

Scientific values as a design requirement

Science earned its authority through a small set of habits: state how likely a result is, state how uncertain that estimate is, show the evidence, make the result reproducible, and say clearly when the instrument is outside its range. Frontier AI has been optimised for fluency, helpfulness, and benchmark performance. Those are real achievements. They do not substitute for the habits above, and the market has so far placed little pressure on developers to adopt them.

That pressure has to come from buyers, regulators, standards bodies, and independent evaluators. It is the reason Maple Quanta exists, and the reason organisations like it are needed: to insist that AI systems used for consequential decisions meet the standards already expected of a laboratory instrument, and to test whether they do. Probability, uncertainty, calibration, and reproducibility belong in the specification.

The public needs AI whose output carries the norms of science: a value, the probability that the value is true, the deviation of that probability, and evidence that such probabilities have held up in comparable cases. A system that delivers these has earned a measure of trust that fluency cannot confer.

By CNN’s account, the episode ended without a shot fired because officials dug deeper before the operation began. A system fit for that setting would tell them, in numbers, how much digging its answer needs.

An AI system fit for high-stakes use reports what it concludes, how likely that conclusion is to be true, how reliable that likelihood is, and whether it has been tested on questions of this kind.

Disclosure: This Insight is based entirely on public information and does not represent the views of the Government of Canada. Maple Quanta Inc. provides AI governance, evaluation, and assurance services, including the kind of calibration and uncertainty assessment discussed here; readers should consider that commercial context. This article is for general informational purposes and does not constitute legal, defence, intelligence, cybersecurity, or regulatory advice.