Nine Methods Graduated. Zero Survived.

An experience report on a gate that could not tell skill from recall, and on the retraction machinery that made the mistake survivable.

By Korhan Temocin, founder of Core Method. 21 August 2026. Written as a practitioner experience report; every count in it reproduces from the archives it describes.

Horizon Core, a private research system I operate, is an attempt to distil the way different models reason about the future into named, reusable forecasting methods, and to let none of them be trusted on charisma: a method earns standing only by passing five statistical tests over a frozen prediction ledger. The experience in this report arrived in the middle of that build. There is a difference between knowing something and remembering it, and the gate I had built to guard that standard could not tell the two apart. This report is about the day I found that out, and about the four hours that followed: every credential the confusion had produced was revoked, and nothing about the mistake was erased.

On 22 June 2026, nine methods passed the gate. The same day, I retracted all nine. The gate had measured whether their answers were right, corrected, calibrated and consistent. The one question that mattered, derived or merely remembered, it could ask only through a hand-written recall tag, and the tag lied. The evaluation cases were famous episodes of technology adoption; to a well-read model, a famous case is an exam it has already seen the answers to. What follows is the autopsy: the probe that exposed the confound, the masking fix that failed, the append-only retraction machinery, and the three explanations my records offer for the zero, only one ever tested. What this report offers is not the contamination finding, which belongs to a known class, but the operational pattern for the day that class turns up inside your own credentials: invalidate the evidence, revoke the credential, preserve the history, block forward use. The pattern earned its keep: the same day, the corpus gained its first recall-resistant batch, and twenty days later the successor graduated thirty-three methods under the rebuilt gate, reality's first grades due January 2027. It closes with what an adversarial audit of its own numbers removed, because the report would be dishonest by its own standard if it had not been through one.

1. The deployment

Horizon Core is a multi-model agent system I run as the research arm of a small studio. The subsystem this report concerns is a gate that graduates forecasting methods. A method, here, is a reusable recipe for answering a held-out question about a historical technology adoption episode: colour television, household radio, microwave ovens and their relatives, posed as boards whose outcome the solver is not shown. Two separate closed-book lanes answer each board, and the system's own published rule is that pooled lanes count once: a pooled trial hits only if every lane that scored it hit.

The point of the gate is institutional rather than statistical: methods that graduate become things the system is allowed to trust and build on, so a false graduation is not a bad number in a table but a false credential inside the system's own decision-making, and everything downstream inherits it.

The deployment surface should be stated exactly, because it bounds every claim that follows. The gate ran as live internal infrastructure inside the studio's own research operation, not as a benchmark exercise: a graduated piece became eligible to speak forward, to back the system's dated claims about the future, and retraction removed exactly that eligibility, with the store excluding a retracted piece from every active listing, the forward bridge re-checking against the store itself, and the triggers sealing the permanence of the record. No client work and no external decision ever consumed these credentials. The episode is a complete containment cycle on that surface: a credential issued on evidence that could not carry it, revoked before anything downstream spoke with it.

2. The gate

Graduation required passing five tests conjunctively:

1. a sample floor: at least eight pooled blind trials before a method is even eligible,
2. a lucky-student guard: an exact binomial test against the corpus no-skill base rate, corrected with Benjamini-Hochberg across every method ever tested, not only the current batch,
3. a calibration ceiling,
4. a cross-pool requirement: the trials must span at least three distinct eras, and at least one must defeat recognition,
5. a posterior-mean floor.

A method that cleared all five had cleared a sample floor, beaten chance under multiple-testing correction, stayed calibrated, replicated across eras, and held a posterior. I considered this a hard gate, and by the standards I knew to apply, it was.

3. Graduation day

On 22 June 2026, the system recorded nine graduations under the then-current gate, minted as nine pieces which, in that batch, carried one method each, nine distinct content hashes. I write recorded rather than proven deliberately: the per-method verdicts were never persisted, a gap section 8 returns to. The mint rows sit in the archive with their original timestamps.

I then ran a recall probe panel against the corpus, asking one blunt question per (case, question) cell: can a pretrained model produce the correct answer from memory alone, without deriving anything?

All nine pieces were retracted before the day was out: 3 hours, 43 minutes and 51 seconds after the mint, the nine retraction rows landing within 116 milliseconds of each other with one distinct reason string, naming recall contamination. Two details matter beyond narrative. The 116 milliseconds says the nine retractions were one decision, not nine reluctant ones. And the commit that shipped the empirical recall gate is timestamped 4 minutes and 38 seconds before the retraction ran, its message pre-announcing the retraction of the recall-contaminated nine as the follow-on step: a second record under the same operator's control whose chronology agrees with the archive's. The pair establishes consistency, not independent witness.

The reason string is worth quoting verbatim; it names the mechanism, the detector, and the planned repair in one breath:

-- piece_retractions.reason, identical across all nine rows, verbatim:
recall-contaminated: minted on recall-leaky (case,qid) evidence; under the
empirical recall-resistance gate (migration 22) the method graduates on
0 recall-resistant trials. Superseded pending an obscure-answer corpus (Slice C).

4. What the gate had actually certified

The probe panel covered forty (case, question) cells across twenty cases. Twenty-three came back not recall-resistant; on sixteen, every prober produced the correct answer from memory alone. Each row carries a JSON evidence blob with each prober's answer, confidence and claimed basis, inspectable one row at a time.

The panel was small, and I want that on the record: thirty-eight cells were probed by a two-model panel, two by a three-lane panel. Nothing bigger ever ran, whatever my drafts remembered.

The mechanism is easy to state. The cases were famous, and for a famous episode the answer to a held-out question is not something a strong pretrained model needs to derive; it can remember it. My gate measured whether answers were right, corrected, calibrated and consistent, and recalled answers are all of those things. The sharpest fact: the gate did carry an anti-recognition clause, asked through a hand-written editorial tag, and the corpus had tagged colour television itself low-recall. The clause was satisfied by a label that lied. The precise statement of the failure is a confound: the gate could not distinguish derived performance from access to historical outcome knowledge, so it could not exclude recall from what it certified. The probe does not prove that each of the nine answered from memory; it proves the evidence behind their credentials could no longer be read as proof of skill, which was enough to revoke them. The lanes were sealed closed-book before graduation day, so the probe measured memory, not lookup. The phenomenon class is well documented [1, 3, 4]; this report claims no discovery in it.

5. The fix that failed

The obvious repair was to strip identity from the boards, abstracting regime and statement until a famous adoption curve stops being recognisable. I masked thirteen boards and ran my own re-identification check.

My notes from that evening say every one came back identified, at high self-reported confidence, off the figures and dates that had to survive the masking. The per-board results were never persisted, so that claim survives only as my own assertions, in the notes and in two same-author commit messages; this report treats all of it as testimony, not measurement. What is verifiable sits in the commit itself, verbatim: "three content suppliers + fall-1965 compatible lineup" is color TV. That is prose, not a number. An episode's identity does not live only in its figures; it soaks the whole board, and the rebuilt schema says so in its own words: the specific figures the method must keep to reason are the identity fingerprint.

I abandoned that branch, twenty minutes before the recall gate shipped. It was never merged, but it is preserved and inspectable: the thirteen masked boards, the masking engine and its leak validator still sit on it. What cannot be handed to a reviewer is the re-identification harness itself, which was never committed, so the probe's per-board results survive as testimony alone. That is the hole in this episode, and section 8 returns to it.

The wall I hit that evening has since been described independently. Lopez-Lira, Tang and Zhu show that language models recall pre-cutoff economic and financial values exactly, and that masking fails because the models reconstruct entities and dates from minimal context [2]. Their thoroughly neutered earnings call that still named the firm and the quarter is my colour television in another costume: identity survives in the prose. My notes remain notes; the failure mode they describe is replicated literature.

6. Retraction by append

The part I would build the same way again is the retraction machinery, and it is younger than it reads: the pieces store was born append-only the day before the mint, but the retraction table and its seal shipped in the same commit as the recall gate, 4 minutes and 38 seconds before their first use. The honest un-mint was built inside the four-hour window.

A piece in this store is never deleted and never rewritten; the schema forbids delete, update and replace on pieces, and delete on retractions, at the trigger level. A retracted piece is barred from speaking forward, but its row remains: the nine mint rows keep their original timestamps beside nine retraction rows in a separate table, naming one reason.

The enforcement is a handful of short triggers whose error strings state the store's position better than my prose. Two, verbatim:

CREATE TRIGGER pieces_no_delete
BEFORE DELETE ON pieces
BEGIN
    SELECT RAISE(ABORT, 'pieces are append-only (an earned piece is permanent)');
END

CREATE TRIGGER piece_retractions_no_delete
BEFORE DELETE ON piece_retractions
BEGIN
    SELECT RAISE(ABORT, 'a retraction is a permanent record (append-only)');
END

The correction did not overwrite the mistake. It was appended underneath it. This is what makes the episode auditable by someone who was not there: the store cannot be made to forget the graduation happened, and the retraction row itself refuses deletion. Candor here is not a narrative choice. It is a schema property, and where the schema is thinner than the prose, section 10 names it. Appendix B carries the cycle as a schematic.

7. Three threats, one acted on

My records name three candidate explanations for the zero:

1. the held-out sample was too small,
2. a recall confound contaminated the evaluation,
3. the corpus base rate had drifted to 0.625, far enough to raise the correction bar sharply.

These are not three parallel stories, and an earlier draft of this report made exactly that category error. They are threats to validity with different evidentiary status. The recall confound is the one that was tested and machine-recorded: it is the reason string in all nine retraction rows, written at the moment of retraction. The other two were never tested to a persisted result. One of them is presented, in a document whose own header still reads "Status: DESIGN", as a live-run finding that sixteen methods would have graduated at a balanced base rate; no output of that run was ever persisted, so this report holds it as an open threat rather than a finding. I retracted on the tested threat. What I cannot offer is a reconciled causal account of the zero: it is overdetermined in my records, and only one of its candidate causes was ever put to a test.

8. Auditing this report's own numbers

An experience report about a contaminated evaluation has an obvious failure mode: contaminated numbers of its own. On 1 August 2026 I put the claims of this episode through an operator-controlled adversarial claim-verification pass, querying the frozen archive directly rather than my notes about it. The protocol, so the word audit does no hidden work: inventory every countable claim, classify its source, recompute it from the archive, attack the survivors with counter-queries, remove or downgrade what fails, record every correction with a date. The pass ran on model agents; I do not report their number, because an agent is not an independent reviewer. What matters is which counter-queries killed which claims.

Two headline figures survived hostile re-query, and came back stronger than claimed: nine distinct methods, one member per piece, nine distinct content hashes; and nine retractions, none ever re-earned. Eight supporting figures did not survive, and they are not in this report. The pattern of what died is the useful part:

The same pass caught the archive's manifest contradicting its own contents: the summary line said 19 recall-resistant cells where the table returns 17. The manifest now carries the verified count and a dated correction note; the database itself is untouched and its checksum unchanged.

One gap survives everything: the per-method gate verdicts were never persisted. The archive proves the outcome of the run, nine minted and nine retracted, and not the run itself. That gap is mine, and this report does not paper over it.

9. What survives as method

Four lessons, each paid for above:

1. Score the evaluation, not only the system. Until something has tried to break the gate, a pass tells you about the gate. Mine was statistically strict and epistemically blind, and the strictness made the blindness more convincing. Conjunctive statistical tests are not a contamination defence: no correction, calibration or pooling downstream can recover what the evidence itself cannot carry.
2. A famous benchmark is a recall benchmark. If the evaluation cases exist in a pretrained model's memory, correctness stops being evidence of derivation, and identity masking fails on narrative residue, not only on numbers [2].
3. Keep the branch you threw away. The record of a failed fix is worth more than the fix would have been. The masking branch survives unmerged with its thirteen boards; the probe harness that condemned it was never committed, and that absence is the weakest point of this report.
4. Retract by append, and enforce it in the store. Habits produce quiet corrections; triggers produce trustworthy ones. Persist the run, not only the outcome: my missing gate verdicts are the outcome-only version of the same sin.

10. The repair, and what it cannot yet claim

The retraction's reason string did not only name the failure; it promised the repair, and the repair was built. The same day, the corpus gained its first obscure-answer batch: eight adoption cases probed obscure at the question level, fourteen of sixteen questions recall-resistant, credit gated per question, adopted as the only lever left once identity masking fell. Four of twelve candidate cases were dropped in review, two for unverifiable sources, one for a wrong answer, one for a recall leak: a defective case teaches a false method. Three days later recall resistance became mechanical for new admissions: a recall-scored case travels with a committed screen or the validator refuses it; June-era cases stay gated per question by the runtime table. The gate had always excluded probe rows from the skill count; what the rebuild changed is who answers the recognition question. The editorial tag was retired as untrustworthy, no method now graduates without at least one trial proven recall-resistant by measurement, and the fail-closed reading became the default the same evening. The probe programme grew from forty cells to three hundred and fifty-five.

On 12 July 2026, under that harder gate, the successor universe graduated thirty-three methods. The bar is one recognition-defeating trial per method; measured against the live store, each of the thirty-three carries at least seventy empirically recall-resistant trials, on 272 to 284 blind derivation rows. Whether the July run took the strict path is not persisted; the measured margin is the evidence that matters, and the mint doorway now refuses the relaxed flag outright. None of the thirty-three has been retracted, and none of the original nine is among them: as of 17 August 2026, zero of the nine revoked hashes has been re-earned, both figures re-checked read-only against the live store for this report. Two repairs are still owed, named rather than implied closed: per-method gate verdicts are still not persisted, and the retraction table carries only the delete seal, not the full update-and-replace seal its pieces enjoy.

What the thirty-three cannot claim is what the nine could not: reality's verdict. They are the system's own certification, issued by a better instrument; nothing in this project has yet been graded by reality, and that sentence calibrates everything above. Zero held from 22 June to 12 July 2026, and the title is not a present-tense claim. The first two forward claims stand open until January 2027 under a grading rule frozen when they were posted, and I disclose their shape before anyone computes it: the two lanes disagree on the same question, 0.55 for the event against 0.58 for its absence, so the first grading is guaranteed to hand the system at least one scored miss. The outcomes will be appended to the studio's research pages whichever way they fall; that future measurement is a calendar fact, not evidence for anything here.

Every June count reproduces from four tables extracted read-only from the frozen archive, distributed as CSV with the archive's SHA-256 checksum; the four queries and their expected results are in Appendix A. The July and August figures are read from the live store and carry their as-of dates. What the tables cannot show, the report has said where it bites.

References

1. Kapoor, S., and Narayanan, A. 2023. Leakage and the Reproducibility Crisis in Machine-Learning-Based Science. Patterns 4(9).
2. Lopez-Lira, A.; Tang, Y.; and Zhu, M. 2025. The Memorization Problem: Can We Trust LLMs' Economic Forecasts? arXiv:2504.14765.
3. Oren, Y.; Meister, N.; Chatterji, N.; Ladhak, F.; and Hashimoto, T. 2024. Proving Test Set Contamination in Black-Box Language Models. In Proceedings of ICLR 2024.
4. White, C., et al. 2024. LiveBench: A Challenging, Contamination-Free LLM Benchmark. arXiv:2406.19314.

Appendix A: the evidence extract

Four tables travel with this report as CSV, extracted read-only from the frozen archive (SQLite, 3,379,200 bytes, SHA-256 888440396a4220b1cb66ff8b806f79b5f4af8c007907c432f67965885ea82a03) by a script that opens the store with mode=ro and recomputes every figure rather than restating it.

The four queries, with their expected results:

1. SELECT COUNT(*), COUNT(DISTINCT piece_id), COUNT(DISTINCT method_hash)
   FROM piece_members;
   -- 9, 9, 9: nine pieces, one member each, nine distinct methods

2. SELECT MIN(created_at) FROM pieces;
   -- 2026-06-22T12:16:15.848502+00:00
   SELECT MIN(created_at), MAX(created_at) FROM piece_retractions;
   -- 2026-06-22T16:00:07.023205+00:00 to ...07.138945+00:00

3. SELECT COUNT(DISTINCT reason) FROM piece_retractions;
   -- 1: every retraction names the same cause

4. SELECT resistant, COUNT(*) FROM recall_verdict GROUP BY resistant;
   -- 0: 23 cells, 1: 17 cells
   SELECT COUNT(*) FROM recall_verdict WHERE recalled_correct = panel_size;
   -- 16: every prober correct from memory

What the extract does not show, stated plainly: there is no per-method gate-verdict table; the probe panel size is 2 on thirty-eight of the forty cells and 3 on two; the June re-identification harness was never committed; and the append-only triggers live in the archive's schema, checkable in the archive itself, not in a CSV.

Appendix B: the containment cycle, as a schematic

Korhan Temocin, founder, Core Method. 21 August 2026.