ThesisAnalysisOS

Research software · Justin Bennett

What still holds?

Research keeps moving under its own results. A check that passed last month is a fact about last month. ThesisAnalysisOS keeps every result an agent records together with what it stood on, so it can be asked later whether it still stands. It answers holds, fails, stale or unknown, and says why.

Step through a real run Read the code on GitHub

Python standard library only. No network and no model calls. It runs in a minute.

holds    check fit: passed 2026-11-22T01:29:09Z; nothing it tested has changedstale    check fit: analysis changed: data/swings.csv
Real output. The same check on the same task, before and after one input was measured again.

01 / A recorded run

Seven moments from one small measurement

Six pendulum swings, a fit for g, and a check that the fit is sane. Two agents work on it. Everything in the dark panels is output from ./taos demo, captured when this page was built. Nothing was typed in.

 taos gate run --task PEN-1 --agent codexok    fit          exit 0 taos task transition PEN-1 --to done --agent codex --reason "g = 9.82 +/- 0.02 from six swings" --evidence analysis/fit.pyPEN-1 -> done taos why PEN-1PEN-1  done  Measure g from the six swingsdone when: the fit passes its check and g is quoted with its errorholds    closure: codex moved it to done on 2026-11-22; the same instructions governholds    check fit: passed 2026-11-22T01:29:09Z; nothing it tested has changedPEN-1 stands as recorded. No person has accepted it yet.

1 of 7

A result is recorded

An agent runs the check and closes the task. The pass is stored with a fingerprint of the files the check depends on, and the closure with the instructions in force. Asked now, both hold. Read the last line too. A check passing is not a person accepting.

holds taos_core/gates.py

The plot style is edited. The check never read it. taos why PEN-1holds    closure: codex moved it to done on 2026-11-22; the same instructions governholds    check fit: passed 2026-11-22T01:29:09Z; nothing it tested has changedPEN-1 stands as recorded. No person has accepted it yet.

2 of 7

Something else changes

The plot style is edited. The check never read it, so nothing is rerun and nothing is doubted. A rule that trusts results by age would have had to guess here.

holds taos_core/basis.py

One swing is measured again, and data/swings.csv changes. taos why PEN-1holds    closure: codex moved it to done on 2026-11-22; the same instructions governstale    check fit: analysis changed: data/swings.csvPEN-1 does not stand as recorded: stale.

3 of 7

An input changes

One swing is measured again. The old pass is still true about the old data. It no longer answers for the new data, so the answer is stale, and it names the file that moved.

stale taos_core/basis.py

 taos gate run --task PEN-1 --agent codexok    fit          exit 0Another task needs the same check on the same files. It is not run twice. taos gate run --task PEN-2 --agent claudeheld  fit          not rerun: nothing it tested has changed since 2026-11-22T01:29:09Z

4 of 7

A second task asks the same question

The check is run again on the new data. Then another agent's task needs the same check on the same files. A result belongs to the files it tested, not to the task that asked, so it is held and not run twice.

holds taos_core/gates.py

 taos rule add --quote "Use the tape-measured lengths" --said-on 2026-01-05 --scope PEN-1added rule_c7f00301eb for PEN-1 taos rule add --quote "Use the laser-measured lengths" --said-on 2026-03-01 --scope PEN-1 --supersedes rule_c7f00301ebadded rule_ffa178c891 for PEN-1 taos rule governing PEN-1 --on 2026-02-10rule_c7f00301eb  said 2026-01-05, no expiry, scope PEN-1"Use the tape-measured lengths" taos rule governing PEN-1 --on 2026-03-02rule_ffa178c891  said 2026-03-01, no expiry, scope PEN-1"Use the laser-measured lengths"PEN-1 was closed before either instruction was recorded, and its closure says so. taos why PEN-1stale    closure: codex moved it to done on 2026-11-22; the instructions have changed since: rule_ffa178c891 governs now and is not one this was recorded under ("Use the laser-measured lengths", said 2026-03-01)holds    check fit: passed 2026-11-22T01:29:09Z; nothing it tested has changedPEN-1 does not stand as recorded: stale.

5 of 7

An instruction keeps its date

A person replaces an instruction. Both are kept, each with the day it was said, so what governed on any date can still be answered. Work closed under the old words is marked stale against the new ones, and the new ones are quoted as the person said them.

stale taos_core/rules.py

 taos task accept PEN-1 --note "read the fit and the residuals"PEN-1 accepted by human, for the files and instructions as they are nowA comment is added to analysis/fit.py after the person looked. taos why PEN-1stale    acceptance: human accepted it on 2026-11-22; since then analysis changed: analysis/fit.py

6 of 7

Accepted means this version

A person reads the result and accepts it. That acceptance is for these exact files and instructions. A comment added afterwards does not inherit it.

stale taos_core/tasks.py

A second check reads figures/style.txt, but says it depends only on the analysis. It is markedisolated, so it runs in an empty folder that holds only the files it declared. taos gate run --task PEN-2 --agent claudeFAIL  style        exit 2grep: figures/style.txt: No such file or directoryisolated: it could read only analysis. A file it could not find is one it did not declare.The declaration is corrected to name the figures, and the same check passes. taos gate run --task PEN-2 --agent claudeok    style        exit 0

7 of 7

A check passes only on what it declared

A second check says it depends only on the analysis and quietly reads the figure style. Marked isolated, it runs in an empty folder that holds only what it declared, so the hidden input is not there to be read. It fails until the declaration is corrected.

fails, then holds taos_core/gates.py

02 / The idea

One comparison, used for everything

A record is a statement, who made it, and what it stood on. Asking is one comparison between that basis and the present. There are four answers.

  • holdsWhat it stood on is the same.
  • failsChecked as things are now, and it does not pass.
  • staleSomething it stood on changed, and the change is named.
  • unknownIt cannot be established. It is never rounded to anything else.

Three rules follow, and none of them depends on which model is doing the work.

  1. Validity is the basis, not the age. A pass from last month holds if nothing it tested has changed. A pass from a minute ago does not if something has.
  2. The original words outrank any summary of them. An instruction is kept as the person said it, with its date.
  3. An observation has a kind and a source. Shown is not repeated. Recommended is not executed. A check passing is not a person accepting.

Where the same comparison is used

03 / Evidence

Measured, with the limits beside the numbers

  • 22 of 22known failure cases pass here. 22 of them fail at the first release, 071a348.
  • 20 of 20protections, removed one at a time, are noticed by the tests.
  • 42 of 42claims hold when read back against the code. A build of this page stops if one does not.
When is a recorded result safe to reuse? 60 edits to a small tree, and after each one all 6 checks are relied on: 360 results in all. A policy can spend executions, or rely on a stale result.
PolicyChecks executedStale results relied on
Rerun everything, every time3600
Rerun when older than 2 asks18030
Rerun when older than 4 asks9083
Rerun when older than 8 asks48161
Reuse by basis, inputs declared600
An oracle that knows what each check reads600
Reuse by basis, every check hiding one input29150

Reusing by basis executed what an oracle would and relied on nothing stale. An age limit buys fewer executions with stale results. The last row is the limit of the design in the open: a check that reads what it did not declare is reused wrongly. Run isolated, the same checks ran 366 times and never passed, so nothing was reused. Isolating one check added 14.0 ms, the median of 40 paired runs. 7 of 7 predictions were written down before the runs, and all of them held.

What this does not show

  • The evidence is mechanical. Nothing here shows that research comes out better.
  • One synthetic workload and one laptop. The counts are counts, not a speed-up.
  • In the open, what a check says it reads is trusted. Isolation closes that, at a cost per run.
  • Isolation closes the network and the rest of the workspace on macOS only. Elsewhere the record says they were open.

What asking costs

Keeping a record is not free. Its basis is fingerprinted each time it is asked. On a 48-file tree that took 15 ms, so reuse paid for any check slower than 3 ms. On a 4,800-file tree it took 305 ms, and reuse paid only for checks slower than 61 ms.

Every claim, with the code and the test behind it

04 / Where it comes from

Old ideas, applied to what research rests on

Nothing here is a new primitive.

  • Truth maintenance. A belief is kept with its justification and withdrawn when the justification goes. Doyle 1979; de Kleer 1986
  • Build systems. Record what a result was made from and redo only what a change reached. Mokhov, Mitchell and Peyton Jones 2018
  • Temporal databases. When something was true is kept apart from when it was recorded. Snodgrass and Ahn 1985
  • Provenance. What a result was derived from is itself data. W3C PROV-DM 2013
  • Hermetic builds. A step can read only the inputs it declared. Bazel sandboxing; the Nix build sandbox

What is added is what a research project rests on and a build system does not model: an instruction with its date, and a person's acceptance of one exact version.

The cost question came from physics. A transport network is priced with a two-term ledger: carrying flow costs less as structure is added, and the structure costs upkeep. A recorded result has the same two terms. Upkeep is fingerprinting what it stood on. Transport is running the check again. The ledger says structure stops paying once it matches what is really read, and the measurement agrees: declaring 6 domains or 12 saved the same work. Nothing here claims that agent work obeys a physical scaling law.

The system was not designed at a desk. It is what survived a year of a physics thesis analysis run with two AI agents, generalized so that it carries no physics. The method, the predictions and the prior work.

05 / For agents

This page is also a file you can read

Everything above exists as plain text, built from the same source in the same run. An agent does not have to parse this page, and should not take its word either. It is built to be checked.

In the repository, AGENTS.md says how to examine the system without setting it up, and one command reads every claim back against the code.

python3 evals/claims.py --json

06 / Start

See it in a minute

git clone https://github.com/JustinBen0131/ThesisAnalysisOS.git
cd ThesisAnalysisOS
./taos demo

It builds a throwaway copy of the measurement and runs real commands against it. Nothing of yours is touched.

To run your own project on it, choose Use this template on GitHub and follow the manual. Setup is an interview: an agent asks a short set of questions and builds the system from your answers.

Open the repository Read the manual