Evidence
Every claim, read back
This is docs/CLAIMS.md, with one addition. When
the page was built, each row was checked against the code: a tested row names code and a test that exist, and a measured
row quotes a number its recorded run contains and that run still stands. 42 of 42 hold. Had one not, the build
would have stopped.
Check it yourself in the repository with python3 evals/claims.py.
What a record stood on
| Claim | Code | Check | Standing |
|---|---|---|---|
A recorded result is answered holds, fails, stale or unknown, with a reason; unknown is never turned into another answer | taos_core/basis.py | tests/test_standing.py | holdstested |
| A passed check holds for the bytes it tested and stops holding when one changes, however recent it is | gates.standing | test_a_pass_does_not_cover_a_file_edited_afterwards | holdstested |
| The answer names the file that changed | basis.compare | test_closed_work_stands_until_a_file_changes_and_the_reason_names_the_file | holdstested |
| No route moves a task to review or done unless every check holds, or a waiver with a reason is recorded and shown | TaskStore.transition | test_done_cannot_be_reached_by_a_route_that_skips_the_check, test_a_waived_closure_is_unverified_until_the_check_runs | holdstested |
| A result is shared by every task whose files are the same bytes, and is executed once | gates._key, .taos/gate_results.json | tests/test_standing.py::TestSharedResult | holdstested |
A change to a file no domain claims is unknown; a check naming an undeclared domain is unknown | basis.compare, gates.standing | tests/test_standing.py::TestDomains | holdstested |
| A declared domain holds what its patterns match on disk, so an input git ignores is part of what a check stood on | basis.on_disk, basis.snapshot | test_an_input_git_ignores_is_still_part_of_what_a_check_stood_on | holdstested |
| A check marked isolated can read only the files its result is filed under, and what it writes is discarded | gates._isolated | test_a_check_cannot_pass_on_a_file_it_did_not_declare, test_an_isolated_check_sees_only_what_it_declared_and_leaves_nothing_behind | holdstested |
| Only a person accepts a result, and the acceptance is of one version | TaskStore.accept | test_acceptance_is_of_one_version_and_does_not_carry_over | holdstested |
A work product is current while its declared inputs are the same bytes; with none declared it is unknown | add_evidence, standing._artifacts | test_a_work_product_is_current_while_its_declared_inputs_are_the_same_bytes | holdstested |
A new or lapsed instruction makes work closed under the old ones stale | rules.binding, rules.changes_since | test_a_new_instruction_makes_the_closure_stale_and_is_quoted | holdstested |
Instructions, lessons and the other records
| Claim | Code | Check | Standing |
|---|---|---|---|
| The instruction that governed a scope on a past day is still answerable after it was replaced | rules.governing | test_a_later_replacement_does_not_erase_the_earlier_answer | holdstested |
| A replacement governs from its own date, inside its own scope, until it lapses | rules.displaced_by | the four other tests in TestInstructionsKeepTheirTime | holdstested |
| What governs a task and what was learned reach the agent at start, in the original words | lifecycle.start, capsule_text | tests/test_standing.py::TestDelivery | holdstested |
| A reminder shown is an exposure; only a judged outcome is a repeat | lessons.at, lessons.outcome, lessons.repeats | test_a_reminder_shown_for_correct_work_is_not_a_repeated_mistake | holdstested |
| A setup is credited only when it was observed to run | control.rebuild | test_an_unobserved_profile_earns_no_credit | holdstested |
| A lapsed lease is over | ClaimStore.heartbeat, acquire | tests/test_counterexamples.py::TestALeaseEnds | holdstested |
| A refused start takes nothing | lifecycle.start | test_a_refused_start_takes_nothing | holdstested |
| A label is read strictly and a missing one is counted apart | replay.label | test_the_text_false_is_not_read_as_true, test_an_unlabelled_case_is_not_counted_as_harmless | holdstested |
| A decision-bank run keeps the answers it scored and their hash, so the score can be recomputed | bank.score | the bank test in tests/test_science.py | holdstested |
| A change is not confirmed by a number with no source, and a prediction is written once | changes.score, anchors.predict | test_a_change_is_not_confirmed_by_a_number_from_nowhere, test_a_prediction_cannot_be_rewritten_once_made | holdstested |
The repository's own numbers
| Claim | Code | Check | Standing |
|---|---|---|---|
| A number quoted here is stale once the code it was measured on changes, and the answer names the file | evals/measured.py | test_a_recorded_number_goes_stale_when_the_code_it_measured_changes | holdstested |
| An isolated check sees a fixed environment, not the caller's | _fixed_environment | test_an_isolated_check_does_not_see_the_environment | holdstested |
| Each isolated result records what was closed, found by trying; where the network and the rest of the workspace can be closed they are, and a strict check does not run where they cannot | closes | test_the_record_says_what_was_closed_and_it_was | holdstested |
| The setup answers can say which files each check reads | _config_from | test_setup_can_say_what_a_check_reads | holdstested |
| No row of this file names code or a test that is not there, or a number its recorded run does not contain | evals/claims.py | test_no_claim_names_something_that_is_not_there_or_misquotes_a_recorded_run | holdstested |
| The demo runs from a clean copy and prints each idea with real output | taos_core/demo.py | test_it_runs_from_a_clean_copy_and_shows_each_idea_with_real_output | holdstested |
Measurements
| Claim | Number | Reproduce | Standing |
|---|---|---|---|
| Earlier releases had the counterexample cases: every one at the first, five at the second | 22 of 22 fail at 071a348; 5 of 22 fail at 89f36ce; 22 of 22 pass here | python3 evals/at_revision.py | holdsmeasured |
| The tests notice when a protection is removed. The first run, with 16 faults, caught 15; the missing test was added | 20 of 20 seeded faults caught | python3 evals/mutations.py | holdsmeasured |
| Reuse by basis relied on no stale result and executed what an oracle would | 60 executions, 0 wrong, in 360 results relied on | python3 evals/reuse_frontier.py | holdsmeasured, one workload |
| Age limits trade reruns against stale reuse | 180, 90, 48 executions with 30, 83, 161 wrong | same | holdsmeasured, one workload |
| In the open, under-declared inputs are reused wrongly | 150 wrong in 360 when every check hid one folder | same | holdsmeasured, one workload |
| Isolated, a check that reads what it did not declare never passes, and correct declarations cost no extra executions. Predicted before the code was written | 0 of 366 runs passed when every check hid one folder and ran isolated; 60 executions, 0 wrong with everything declared | same | holdsmeasured, one workload |
| Asking costs time that grows with the tree | 15.4, 40.6, 305.4 ms for 48, 480, 4,800 files | same | holdsmeasured, one laptop |
| Isolating a check costs milliseconds. Measured in pairs that differ only in the isolation, order alternating | 14.0 ms added, the median of 40 paired runs (95% interval 13.6 to 14.7 ms) | same | holdsmeasured, one laptop |
| The suite passes on Python 3.9 and 3.12, Linux and macOS | 151 tests | ./taos selftest, and CI | holdstested |
Not claimed
- That research done with this system comes out more correct. Nothing here measures research outcomes.
- That the cost laws of physical transport networks apply to agent work.
docs/METHOD.mdsays what is shared with that work (a two-term ledger and the question it asks) and what is not (the exponents). - That the controller in
policies/CONTROL.mdallocates effort optimally. It is a bounded one-step rule, and it credits only what it observed. - That an open check's declaration of what it reads is verified. It is trusted, unless the check is marked isolated.
- That an isolated check is a correct check. Isolation makes its result depend only on the files it is filed under and a fixed environment. The network and the rest of the workspace are closed on macOS only, and a file outside the workspace can be read by its absolute path anywhere.
- That anything outside the workspace is seen: the network, the clock, the environment, another machine.
The reuse experiment in full
60 edits, 6 checks over 12 folders, seed 7. After each edit every check is relied on, 360 results in all. A wrong reuse is a result relied on after a file it reads changed. Python 3.12.10 on macOS-27.2.1-arm64-arm-64bit.
| Policy, as recorded | Executions | Wrong reuses |
|---|---|---|
| rerun everything | 360 | 0 |
| rerun when older than 2 consultations | 180 | 30 |
| rerun when older than 4 consultations | 90 | 83 |
| rerun when older than 8 consultations | 48 | 161 |
| TAOS, 1 domain(s) declared | 360 | 0 |
| TAOS, 2 domain(s) declared | 180 | 0 |
| TAOS, 3 domain(s) declared | 120 | 0 |
| TAOS, 4 domain(s) declared | 120 | 0 |
| TAOS, 6 domain(s) declared | 60 | 0 |
| TAOS, 12 domain(s) declared | 60 | 0 |
| oracle (knows what every check reads) | 60 | 0 |
| TAOS, three tasks asking | 60 | 0 |
| TAOS, each check hides one folder it reads | 29 | 150 |
| TAOS, checks isolated, declared correctly | 60 | 0 |
| TAOS, checks isolated, each hides one folder | 366 | 0 |
Predictions, written before the runs
- held P1. with what each check reads declared correctly, no stale result is ever reused, at any granularity
- held P2. the number of executions equals the count derived from the edit trace exactly: nothing extra, nothing missed
- held P3. three tasks asking the same questions execute no more checks than one task does
- held P4. finer declarations never cost more executions, and stop helping once they match what the checks read
- held P5. a check that reads a folder it did not declare does reuse a stale result: the declaration is the limit
- held P6. no age window reaches both the oracle's execution count and zero stale reuse
- held P7. isolated, a check that reads a folder it did not declare never passes, and correct declarations cost nothing extra