mirror of
https://codeberg.org/reverse-skill-lab/apk-reverse
synced 2026-11-15 19:44:20 +08:00
Fix the corpus count, recalibrate the token estimate, add a PR template
- check_budget.py counted scripts/__pycache__/*.pyc in its corpus note, so the same commit reported 27 scripts on a machine that had run the suite and 16 on a clean checkout. The routing gate said 16 throughout; two gates disagreeing about one tree was the finding. - TOKENS_PER_CHAR was calibrated against a 67,011-character SKILL.md and the docstring's arithmetic (0.30 characters-to-tokens) did not describe that measurement. Re-measured here: 54,983 chars -> 12,906 tokens = 4.260; the constant and both comments now carry the number this tree measures. - test_budget_measurement.py pins the corpus exclusion, so a bytecode cache cannot re-enter the count and quietly change it again. - .github/PULL_REQUEST_TEMPLATE.md carries what CONTRIBUTING.md asks a contributor for: what was run, the four labels, what has to move with the change, and the boundary that was not exercised.
This commit is contained in:
@@ -0,0 +1,74 @@
|
||||
<!--
|
||||
Two things this template asks for are not machine-checkable: whether a claim's label is honest, and
|
||||
whether the change keeps this repository's index in step. The gate output is the cheapest evidence
|
||||
you can give a reviewer; a green run is not approval, and a blank section is not an omission nobody
|
||||
will notice.
|
||||
-->
|
||||
|
||||
## What this changes
|
||||
|
||||
One or two sentences. Name the file or the behaviour, not the intention.
|
||||
|
||||
## What I ran
|
||||
|
||||
```bash
|
||||
python check_repo.py && python check_refs.py && python check_routing.py \
|
||||
&& python check_commands.py && python check_budget.py && python -m pytest -q
|
||||
```
|
||||
|
||||
Paste the output that decides the question, including any gate that was red before the change and
|
||||
why. Report only checks that actually executed — `should pass` is not a result, and a run on one
|
||||
device, one ABI or one sample says so in the line where it is quoted.
|
||||
|
||||
```
|
||||
<decisive output>
|
||||
```
|
||||
|
||||
If `mypy` or `flake8` are not installed locally, say that here: CI runs them, and a contributor
|
||||
without them cannot reproduce that half.
|
||||
|
||||
## Claims
|
||||
|
||||
Every claim carries one of the four labels this repository uses. `observed` requires both the command
|
||||
and its output; `inferred` means it was not run; `unverified` means someone else said so; `unknown`
|
||||
means nobody looked.
|
||||
|
||||
| Claim | Label | Command and result, or what would have to exist for it to be `observed` |
|
||||
|---|---|---|
|
||||
| | observed / inferred / unverified / unknown | |
|
||||
|
||||
- [ ] No chain is labelled above its weakest step.
|
||||
- [ ] A negative result says whether it was `absent`, `unknown` or `present but unreadable`, and names
|
||||
the probe's coverage. Only the first is a finding.
|
||||
- [ ] A measurement names the version, device or ABI it came from.
|
||||
|
||||
## What has to move with it
|
||||
|
||||
Check every line that applies. These are the rows that cost a red CI run when they are missed.
|
||||
|
||||
- [ ] A reference file — named in `references/routing.md`, reachable from `SKILL.md` or the README,
|
||||
with a load-this-when head if it passes 100 lines.
|
||||
- [ ] A script — a row in the routing script table, its capability declared in
|
||||
`scripts/capabilities.py`, and `--help` working on a bare interpreter.
|
||||
- [ ] A symptom-index row — mirrored into `references/routing.md`, which is compared rather than
|
||||
generated.
|
||||
- [ ] A section heading — every section reference that pointed at the old name is updated.
|
||||
- [ ] A documented command — the flags are copied from `--help`, not retyped.
|
||||
- [ ] `SKILL.md` — the budget line is attached above; it is loaded in full on activation.
|
||||
- [ ] A capability claim — the row in `evidence/capability-matrix.json` **and** the line in
|
||||
`references/evidence-summary.md`. No gate compares them; that is why the reviewer reads them.
|
||||
- [ ] A new pitfall — next number, same four headings.
|
||||
- [ ] `README.md` and `README.zh-CN.md` — the two stay in step.
|
||||
- [ ] Documentation only — say so explicitly; a blank list is not the same statement.
|
||||
|
||||
## Boundaries
|
||||
|
||||
Where something could not be exercised here, ship the measured half plus an explicitly labelled
|
||||
template for the rest, and state what would have to exist for the other half to become `observed`.
|
||||
|
||||
- [ ] Anything needing a device, adb, frida, a rooted phone or the network is named as not run here,
|
||||
rather than described as working.
|
||||
- [ ] One variable changed per result: if a result moved, it is attributable to the one thing this
|
||||
change touched.
|
||||
- [ ] The scan in `scripts/scan_leaks.py` was run over the files being published, and its findings
|
||||
are resolved — including names and metadata that no pattern reads.
|
||||
+12
-7
@@ -78,9 +78,10 @@ NAV_LINES = 100 # above this, a reference must open navigably
|
||||
CORPUS_FILES_NOTE = 110 # references + scripts, reported as a note past this
|
||||
CORPUS_LINES_NOTE = 26000 # total lines across the same corpus, same treatment
|
||||
# Characters per token, for the no-tiktoken fallback. **Calibrated on SKILL.md against
|
||||
# tiktoken/cl100k_base** (47,629 chars -> 11,114 tokens = 4.285), so both measurement paths reach
|
||||
# the same verdict. Getting this wrong made the gate answer differently on CI than locally.
|
||||
TOKENS_PER_CHAR = 4.285
|
||||
# tiktoken/cl100k_base** (54,983 chars -> 12,906 tokens = 4.260, re-measured 2026-10-24), so both
|
||||
# measurement paths reach the same verdict. Getting this wrong made the gate answer differently on
|
||||
# CI than locally.
|
||||
TOKENS_PER_CHAR = 4.260
|
||||
# A navigable head: one of these must appear in the first NAV_HEAD_LINES lines.
|
||||
NAV_HEAD_LINES = 40
|
||||
HEDGE_RE = re.compile(
|
||||
@@ -143,9 +144,10 @@ def token_estimate(text):
|
||||
everywhere in this repository, including in this file. When it is not, it falls back to
|
||||
`characters / TOKENS_PER_CHAR`.
|
||||
|
||||
The fallback constant is **calibrated against that measurement** rather than guessed: a
|
||||
markdown-heavy document tokenizes worse per character than the usual /4 rule of thumb, and this
|
||||
file measured 0.30 characters-to-tokens on SKILL.md (67,011 chars -> 19,929 tokens).
|
||||
The fallback constant is **calibrated against that measurement** rather than guessed: this
|
||||
SKILL.md measures 4.260 characters per token (54,983 chars -> 12,906 tokens, cl100k_base).
|
||||
Agreeing with the "about four" rule of thumb to within a few percent is not close enough to rely
|
||||
on, so the number is re-derived when the document changes rather than assumed.
|
||||
|
||||
Why the value matters more than the method: the first version used 3.6 and the gate therefore
|
||||
reached *different verdicts on different machines* -- `WARN over budget` where tiktoken was
|
||||
@@ -307,8 +309,11 @@ def check_corpus():
|
||||
green always-loaded count.
|
||||
"""
|
||||
refs = sorted(glob.glob(os.path.join(REFS, '**', '*.md'), recursive=True))
|
||||
# A bytecode cache is not corpus. It is untracked, it appears the first time the suite runs, and
|
||||
# counting it made this note report 27 scripts where the directory ships 16 -- a measurement that
|
||||
# changes when a *reader* runs a test is not a measurement of the repository.
|
||||
scripts = sorted(p for p in glob.glob(os.path.join(SKILL_DIR, 'scripts', '**', '*'), recursive=True)
|
||||
if os.path.isfile(p))
|
||||
if os.path.isfile(p) and '__pycache__' not in p.split(os.sep))
|
||||
|
||||
def count(paths):
|
||||
total = 0
|
||||
|
||||
@@ -112,3 +112,25 @@ def test_an_index_row_past_the_hard_limit_fails_rather_than_warns(budget):
|
||||
assert levels(budget.INDEX_ROW_HARD + 10) == {"fail"}, (
|
||||
"a row past INDEX_ROW_HARD (%d chars) must fail the gate, not warn about it"
|
||||
% budget.INDEX_ROW_HARD)
|
||||
|
||||
def test_the_corpus_count_ignores_a_bytecode_cache(budget, tmp_path, monkeypatch):
|
||||
"""A measurement that changes when a reader runs the suite measures the reader.
|
||||
|
||||
The corpus note once counted scripts/__pycache__/*.pyc, so the same commit reported 27 scripts
|
||||
on a machine that had run pytest and 16 on a clean checkout. The routing gate said 16 the whole
|
||||
time; the disagreement between two gates about one tree is what surfaced it.
|
||||
"""
|
||||
refs = tmp_path / "references"
|
||||
refs.mkdir()
|
||||
(refs / "one.md").write_text("# one\n", encoding="utf-8")
|
||||
scripts = tmp_path / "scripts"
|
||||
scripts.mkdir()
|
||||
(scripts / "one.py").write_text("print(1)\n", encoding="utf-8")
|
||||
cache = scripts / "__pycache__"
|
||||
cache.mkdir()
|
||||
(cache / "one.cpython-314.pyc").write_bytes(b"\x00" * 10)
|
||||
|
||||
monkeypatch.setattr(budget, "REFS", str(refs))
|
||||
monkeypatch.setattr(budget, "SKILL_DIR", str(tmp_path))
|
||||
notes = budget.check_corpus()
|
||||
assert notes[0].startswith("corpus: 1 references + 1 scripts = 2 files"), notes[0]
|
||||
|
||||
Reference in New Issue
Block a user