Research & Engineering Updates
Findings, including
the inconvenient ones.
Measurement audit30 August 2026
The instrument was bent, and one stubborn number was the only tell.
What we investigated
A single claim in our frozen evaluation set had never reproduced its own earlier verdict. The result had sat in our reports as 34 out of 35 for three weeks. We had explanations on file — marginal evidence, a hard case near a decision boundary — and had recorded it as an acceptable characteristic of the set rather than a defect.
What we found
The explanations were wrong. The evaluation harness was intended to freeze everything about a claim so that reasoning could be measured in isolation. It was instead rebuilding three of the four values that determine how much each piece of evidence counts. Every score drifted by roughly a tenth of a point in a consistent direction, on every claim, in every run we had ever recorded. Only one claim in the set sat close enough to a decision boundary for that drift to change its outcome.
Four further measurement defects surfaced the same day: a scoring path that never resolved fourteen results, a cache returning verdicts for claims that had not actually been run, and evaluation records that omitted the feature configuration, leaving us unable to prove which version of the system had produced which result.
What changed
After correcting the instrumentation, accuracy on evidence-based claims measured 0.935, up from 0.895. No change was made to how the system reasons. The figure moved because the measurement was corrected, not because the system improved.
Why it matters
Thirty-four out of thirty-five had the shape of a healthy result. It was one claim away from being indistinguishable from a healthy result — and every conclusion drawn from that instrument over the preceding month rested on values that were wrong in a direction nothing in the reporting could reveal.
What it does not mean. The corrected figure has not been independently validated, and it is measured on an internal set. It also does not mean the instrumentation is now known to be correct — only that five specific defects in it have been found and fixed. Previously published figures produced by the faulty instrument are withdrawn rather than restated.
Decisions recorded30 August 2026
Three decisions we wrote down, including one that cost us accuracy.
A feature we built, measured, and switched off
We implemented a capability, tested it thoroughly, and measured its effect against the evaluation set. The measurement showed no material effect, so it is switched off. Building something is not evidence that it helps.
A disagreement we logged rather than deferred to
On one claim, an external fact-check database entry conflicted with the conclusion our system reached on unanimous evidence from primary scientific sources. The system did not defer to the database, and on review the disagreement was recorded rather than resolved in favour of the more authoritative-looking source. Provenance means recording which sources disagreed, not selecting the one that is easiest to defend.
A change that would have improved the number, which we rejected
We measured what would happen if we adjusted our decision thresholds. It would have gained two claims. It would also have removed the constraint that prevents the system asserting conclusions from thin evidence. We did not make the change, and we recorded why — because tuning thresholds against your own test set produces a better score and a worse system.
What it does not mean. These are internal engineering decisions on an internal evaluation set, published for transparency rather than as validated methodology. The third-party fact-check entry referred to above is described in general terms and is not identified.
Controlled experiment18 August 2026
Our own system was systematically under-confident.
What we tested
Whether a ceiling on confidence scores persisted when evidence was held frozen, removing retrieval and model variation as explanations.
What we observed
Across two live evaluation runs and one frozen-evidence replay of the same claim set, no claim reached 0.70 confidence. The cause localised to a single factor binding on 30 of 35 claims — it counted retrieved material that took no position on the claim, and such material averaged half of everything retrieved. In one case a claim where every source that bore on it agreed was reported at 27% confidence.
Why it matters
Confidence is user-facing. For someone deciding whether a claim is safe to publish, a well-evidenced claim shown at 27% is not caution — it is a signal carrying no information, because the system says much the same thing about everything.
What it does not mean. A correction has been specified and its predicted effects registered in advance, but the confirming run is not complete. No post-correction confidence figures are published here.
Controlled experiment18 August 2026
Neutral evidence is not contradiction.
What we tested
Three candidate definitions of “conflicting evidence,” measured on the same frozen set, to establish when a claim genuinely warrants escalation.
What we observed
Treating any spread of stances as conflict flagged all 35 of 35 claims. Requiring genuine contradiction — evidence both supporting and opposing — flagged 3. Requiring a material minority flagged none. Eighteen claims contained no contradicting evidence at all, yet every one would have been reported as conflicted.
Why it matters
A system that reports everything as contested is as unhelpful as one that reports nothing as contested. This is now recorded as a standing invariant in our design.
What it does not mean. This rules out two escalation designs. It does not establish the correct one; that remains open work.
Updates describe internal engineering experiments and measurement audits on controlled evaluation sets. They are not peer-reviewed findings and have not been independently validated.
