Which annotations deserve a second look?

Average precision describes the complete ordering of the review queue. Recall at the primary 5% budget describes the operational slice an expert would actually inspect. The random-review baseline tells us whether that prioritisation is better than spending the same review effort without a model.

Restoration utility is deliberately separate. It asks what happens after simulated review when found injected changes are restored and the downstream classifier is retrained. A ranking can retrieve controlled inconsistencies efficiently while still failing to improve the later classifier.

Intervals and adjusted tests also answer different questions. Ranking-comparison confidence intervals use paired whole-group bootstrap resampling; registered Holm-adjusted values are one-sided. The H4 range instead contains the central 95% of differences across 100 random-review repetitions evaluated on the same fixed final reference set. It does not resample the reference groups. At 0% corruption, average precision is not applicable because the controlled positive event does not exist.

The H4 experiment restored exactly the same 5% review budget under audit-guided and random selection, then trained the same downstream model and evaluated it on the untouched final reference fold. Audit-guided restoration was not favoured.

Subgroup average precision is reported only when the preregistered support rule is met: at least 100 samples and at least 10 injected corruptions. Otherwise the study retains counts without inventing an unstable estimate.

The ranges expose heterogeneity that would disappear inside one aggregate score. They are descriptive, not a biological ranking: the preregistration did not define an omnibus test that could support a causal explanation for tissue, class, mechanism or corruption-rate differences.

Every value in the article is read from the accepted sealed run and carried into evidence.json. The table below traces each statement to its hypothesis, raw identifier, seed, point difference, confidence interval, adjusted p-value and bootstrap count.

The timestamped July freeze records a null commit, a dirty tree and untracked files; the first public Git commit followed on 19 August 2026. Internal hashes preserve file identity, but the later public history is not independent evidence that outcomes were unseen.

The public primary evidence release contains all completed-cell OOF predictions and rankings, the full group bootstrap, subgroup table and H4 restoration arrays. A standalone evidence recalculator that does not import the primary analysis package recomputes the saved H1-H7 comparison statistics; this is not third-party validation. Fold checkpoints were not retained, and a second image-to-result execution still requires a lawful PanNuke copy.

The earlier NuCLS aggregate NP/P experiment did not satisfy its frozen ranking gate and guided intervention made downstream macro-F1 worse. A separate leave-one-pathologist-out analysis then used raw JP.1 annotations as input, removed JP.1 from the hidden reference and compared the frozen global risk ranking with exact matched random review. Its five-patient ranking gate passed. Only this junior pathologist had sufficient public raw geometry and patient groups, so the result is one rotation rather than a pooled pathologist estimate.

The next public replication separated scoring from reference evaluation and published the sealed pre-reference scores first. It then passed both frozen gates on RIVA cervical cytology and MIDOG++ mitosis annotations, with positive aggregate whole-group intervals and non-negative point differences in all six rotations. MoNuSAC showed strong controlled retrieval but no statistically supported downstream gain or complete class safety. The selected AANCA candidate was later recorded internally as frozen before PUMA outcomes and passed all seven internally pre-specified gates under controlled corruption.

Public-history limit. The RIVA x MIDOG++ pre-reference scores were committed and pushed before reference evaluation, providing a public ordering record for this replication. By contrast, the PUMA protocol, configuration and result first appeared together in public commit c5bd44193b2abd67bc7e7f1bd9384aa87435d500. The NuCLS LOO protocol and execution likewise share one repository change. The PUMA readback does not retrain all 44 models. These are project-maintained verification paths, not third-party validation.

Before any primary split is frozen, the inspector verifies folds, shapes, channels, class order, instance identifiers and representative overlays. Cross-class overlaps, void pixels and affected instances are recorded rather than silently corrected.

The overlay is evidence about ingestion and representation, not improved segmentation. Eligibility flags may follow the frozen policy, but the raw masks and source annotations remain unchanged.

Source groups stay together, each audit score is out of fold, the final reference fold is withheld from selection, and every label state remains separate. The matrix retained all 36 comparison entries; unavailable optional pathology cells were not replaced with estimates.

Outcome exposure during technical recovery does not change the stored measurements, but it narrows what can responsibly be claimed. The accepted primary analysis is therefore permanently described as amended and exploratory.

The controlled positive event is an injected label change. Performance against it measures retrieval of that injected process, not naturally occurring annotation inconsistency or biological truth. The NuCLS and RIVA LOO positive events are strict-consensus disagreement after removing the input pathologist; MIDOG++ uses the other independent expert label. These references measure disagreement enrichment, not adjudicated error. The positive PUMA result is a new-source controlled transfer result, while its later stress and observed-label sensitivity analyses are exploratory because outcomes were already open.

Grouping strength differs by source: the primary PanNuke study guarantees source-patch separation, both NuCLS evaluations and MoNuSAC use patient groups, RIVA uses source-image groups, MIDOG++ uses image groups, and PUMA uses one ROI per case. These safeguards reduce leakage but do not transform a consensus or expert reference label into guaranteed biological truth.

Broader natural and operational validity still requires newly recruited blinded reviewers, ambiguity and abstention labels, a prospectively frozen policy evaluated on untouched patients or whole slides, and a multi-site comparison of work with and without AANCA.

The current project has reached PRIMARY_STUDY_COMPLETE, EXTERNAL_VALIDATION_COMPLETE and DEMO_COMPLETE. It has not reached CONFIRMATORY_COMPLETE. The binding natural-data action is retain_uncorrected: AANCA may prioritise cases for qualified review, but it may not silently exclude, relabel or overwrite them.

The next research phase is provisionally named AANCA v2. This is a prospective evidence programme for the same core auditing system, not a retroactive rename of the existing results. Opened PanNuke, NuCLS, MoNuSAC, PUMA, RIVA and MIDOG++ outcomes may inform diagnosis of weaknesses, but they cannot serve as the untouched final confirmation for the next claim.