Fix the corpus count, recalibrate the token estimate, add a PR template

- check_budget.py counted scripts/__pycache__/*.pyc in its corpus note, so the
  same commit reported 27 scripts on a machine that had run the suite and 16
  on a clean checkout. The routing gate said 16 throughout; two gates
  disagreeing about one tree was the finding.
- TOKENS_PER_CHAR was calibrated against a 67,011-character SKILL.md and the
  docstring's arithmetic (0.30 characters-to-tokens) did not describe that
  measurement. Re-measured here: 54,983 chars -> 12,906 tokens = 4.260; the
  constant and both comments now carry the number this tree measures.
- test_budget_measurement.py pins the corpus exclusion, so a bytecode cache
  cannot re-enter the count and quietly change it again.
- .github/PULL_REQUEST_TEMPLATE.md carries what CONTRIBUTING.md asks a
  contributor for: what was run, the four labels, what has to move with the
  change, and the boundary that was not exercised.
This commit is contained in:
dabao1955
2026-10-24 14:55:10 +08:00
parent f3863f070c
commit f497180e35
3 changed files with 108 additions and 7 deletions
+74
View File
@@ -0,0 +1,74 @@
<!--
Two things this template asks for are not machine-checkable: whether a claim's label is honest, and
whether the change keeps this repository's index in step. The gate output is the cheapest evidence
you can give a reviewer; a green run is not approval, and a blank section is not an omission nobody
will notice.
-->
## What this changes
One or two sentences. Name the file or the behaviour, not the intention.
## What I ran
```bash
python check_repo.py && python check_refs.py && python check_routing.py \
&& python check_commands.py && python check_budget.py && python -m pytest -q
```
Paste the output that decides the question, including any gate that was red before the change and
why. Report only checks that actually executed — `should pass` is not a result, and a run on one
device, one ABI or one sample says so in the line where it is quoted.
```
<decisive output>
```
If `mypy` or `flake8` are not installed locally, say that here: CI runs them, and a contributor
without them cannot reproduce that half.
## Claims
Every claim carries one of the four labels this repository uses. `observed` requires both the command
and its output; `inferred` means it was not run; `unverified` means someone else said so; `unknown`
means nobody looked.
| Claim | Label | Command and result, or what would have to exist for it to be `observed` |
|---|---|---|
| | observed / inferred / unverified / unknown | |
- [ ] No chain is labelled above its weakest step.
- [ ] A negative result says whether it was `absent`, `unknown` or `present but unreadable`, and names
the probe's coverage. Only the first is a finding.
- [ ] A measurement names the version, device or ABI it came from.
## What has to move with it
Check every line that applies. These are the rows that cost a red CI run when they are missed.
- [ ] A reference file — named in `references/routing.md`, reachable from `SKILL.md` or the README,
with a load-this-when head if it passes 100 lines.
- [ ] A script — a row in the routing script table, its capability declared in
`scripts/capabilities.py`, and `--help` working on a bare interpreter.
- [ ] A symptom-index row — mirrored into `references/routing.md`, which is compared rather than
generated.
- [ ] A section heading — every section reference that pointed at the old name is updated.
- [ ] A documented command — the flags are copied from `--help`, not retyped.
- [ ] `SKILL.md` — the budget line is attached above; it is loaded in full on activation.
- [ ] A capability claim — the row in `evidence/capability-matrix.json` **and** the line in
`references/evidence-summary.md`. No gate compares them; that is why the reviewer reads them.
- [ ] A new pitfall — next number, same four headings.
- [ ] `README.md` and `README.zh-CN.md` — the two stay in step.
- [ ] Documentation only — say so explicitly; a blank list is not the same statement.
## Boundaries
Where something could not be exercised here, ship the measured half plus an explicitly labelled
template for the rest, and state what would have to exist for the other half to become `observed`.
- [ ] Anything needing a device, adb, frida, a rooted phone or the network is named as not run here,
rather than described as working.
- [ ] One variable changed per result: if a result moved, it is attributable to the one thing this
change touched.
- [ ] The scan in `scripts/scan_leaks.py` was run over the files being published, and its findings
are resolved — including names and metadata that no pattern reads.
+12 -7
View File
@@ -78,9 +78,10 @@ NAV_LINES = 100 # above this, a reference must open navigably
CORPUS_FILES_NOTE = 110 # references + scripts, reported as a note past this
CORPUS_LINES_NOTE = 26000 # total lines across the same corpus, same treatment
# Characters per token, for the no-tiktoken fallback. **Calibrated on SKILL.md against
# tiktoken/cl100k_base** (47,629 chars -> 11,114 tokens = 4.285), so both measurement paths reach
# the same verdict. Getting this wrong made the gate answer differently on CI than locally.
TOKENS_PER_CHAR = 4.285
# tiktoken/cl100k_base** (54,983 chars -> 12,906 tokens = 4.260, re-measured 2026-10-24), so both
# measurement paths reach the same verdict. Getting this wrong made the gate answer differently on
# CI than locally.
TOKENS_PER_CHAR = 4.260
# A navigable head: one of these must appear in the first NAV_HEAD_LINES lines.
NAV_HEAD_LINES = 40
HEDGE_RE = re.compile(
@@ -143,9 +144,10 @@ def token_estimate(text):
everywhere in this repository, including in this file. When it is not, it falls back to
`characters / TOKENS_PER_CHAR`.
The fallback constant is **calibrated against that measurement** rather than guessed: a
markdown-heavy document tokenizes worse per character than the usual /4 rule of thumb, and this
file measured 0.30 characters-to-tokens on SKILL.md (67,011 chars -> 19,929 tokens).
The fallback constant is **calibrated against that measurement** rather than guessed: this
SKILL.md measures 4.260 characters per token (54,983 chars -> 12,906 tokens, cl100k_base).
Agreeing with the "about four" rule of thumb to within a few percent is not close enough to rely
on, so the number is re-derived when the document changes rather than assumed.
Why the value matters more than the method: the first version used 3.6 and the gate therefore
reached *different verdicts on different machines* -- `WARN over budget` where tiktoken was
@@ -307,8 +309,11 @@ def check_corpus():
green always-loaded count.
"""
refs = sorted(glob.glob(os.path.join(REFS, '**', '*.md'), recursive=True))
# A bytecode cache is not corpus. It is untracked, it appears the first time the suite runs, and
# counting it made this note report 27 scripts where the directory ships 16 -- a measurement that
# changes when a *reader* runs a test is not a measurement of the repository.
scripts = sorted(p for p in glob.glob(os.path.join(SKILL_DIR, 'scripts', '**', '*'), recursive=True)
if os.path.isfile(p))
if os.path.isfile(p) and '__pycache__' not in p.split(os.sep))
def count(paths):
total = 0
+22
View File
@@ -112,3 +112,25 @@ def test_an_index_row_past_the_hard_limit_fails_rather_than_warns(budget):
assert levels(budget.INDEX_ROW_HARD + 10) == {"fail"}, (
"a row past INDEX_ROW_HARD (%d chars) must fail the gate, not warn about it"
% budget.INDEX_ROW_HARD)
def test_the_corpus_count_ignores_a_bytecode_cache(budget, tmp_path, monkeypatch):
"""A measurement that changes when a reader runs the suite measures the reader.
The corpus note once counted scripts/__pycache__/*.pyc, so the same commit reported 27 scripts
on a machine that had run pytest and 16 on a clean checkout. The routing gate said 16 the whole
time; the disagreement between two gates about one tree is what surfaced it.
"""
refs = tmp_path / "references"
refs.mkdir()
(refs / "one.md").write_text("# one\n", encoding="utf-8")
scripts = tmp_path / "scripts"
scripts.mkdir()
(scripts / "one.py").write_text("print(1)\n", encoding="utf-8")
cache = scripts / "__pycache__"
cache.mkdir()
(cache / "one.cpython-314.pyc").write_bytes(b"\x00" * 10)
monkeypatch.setattr(budget, "REFS", str(refs))
monkeypatch.setattr(budget, "SKILL_DIR", str(tmp_path))
notes = budget.check_corpus()
assert notes[0].startswith("corpus: 1 references + 1 scripts = 2 files"), notes[0]