exe-reverse/tests/README.md

# tests/ — executable regression suite for the exe-reverse skill

The suite that proves the shipped scripts still do what their docstrings say, offline, with only
the standard library and pytest.

## When to load this

Read it before changing anything under `skills/exe-reverse/scripts/`, and read it before quoting
a coverage claim about this skill. It is the boundary between "the script was written to do X"
and "the script was measured doing X on this host".

## What it runs

| File | What it asserts |
| --- | --- |
| `conftest.py` | Shared fixtures: builds the synthetic PE images at test time, loads a script by path, runs a script as a subprocess with a timeout, and the two synthetic scan corpora. |
| `unit/test_peutil_reader.py` | Header fields against the fixture's declared metadata, RVA to file-offset round trips including the virtual-only tail, the PE32 versus PE32+ optional-header difference, the checksum against a second implementation, the resource tree, named and ordinal imports, the TLS callback, the RSDS/PDB record, relocation blocks, entropy and flag decoding, and the parser's rejections. |
| `unit/test_pepatch_patcher.py` | Pattern parsing and matching, the ambiguity refusal, the expect-mismatch refusal, the length-unproven and length-change cases, the resolver's refusals, the re-read verification, and the CLI-level facts that a dry run writes nothing and a real run leaves its input byte-identical. |
| `unit/test_aob_scan.py` | The verdict boundaries at both sides of each threshold, hit reporting with offsets, RVAs and sections, section filtering, context windows, and every accepted shape of a pattern file. |
| `unit/test_scan_leaks.py` | Every rule firing on a shaped sample, every exemption class staying silent, the three measured deviations from the exemption list, scan position reporting, and the file walk's skips. |
| `unit/test_capabilities_registry.py` | The capability table: every atom's kind, basis and probe name, the transitive closure against an independent walk, the dependency graph's acyclicity, `resolve` on missing atoms and on `file:` atoms that are and are not on disk, the declaration parser including the documentation template it must reject, and the `declared` / `mapped` / `unknown` classification. |
| `unit/test_doctor.py` | The static dependency pass (stdlib, siblings and a third-party name separated without executing the file), `discover_scripts` scanning rather than listing, the per-script table with an unclassified script still counted, and the environment facts with their keys and severity labels. |
| `unit/test_budget_measurement.py` | The budget gate's two measurement paths agree (real tokenizer versus the calibrated fallback), the fallback constant still matches a fresh measurement of `SKILL.md`, and a row past `INDEX_ROW_HARD` fails rather than warns. |
| `unit/test_dart_snapshot_strings.py` | The ELF reader against a generated container, both literal encodings recovered at the offsets the fixture placed them at, the length and limit gates, and the pool counts -- zero on an image carrying neither idiom and non-zero on hand-built bytes carrying each one. |
| `cli/test_cli_surface.py` | Help and usage exit codes for every script, the documented exit codes for each script's error paths, and the negative case that inspect-only runs write nothing into the working directory. |
| `cli/test_cli_coverage.py` | Coverage derived from the scripts directory rather than a hand-written list: every shipped `.py` is either exercised or declared uncovered with a reason, and every one answers `--help` with its usage block. A new script fails here until it is registered. |
| `cli/test_cli_json.py` | Each script's machine-readable output: keys, values, and the conventions that are not what a reader would assume -- including `pe_triage.py`'s identity block on a generated image (and that it writes nothing) and `pe_diff.py`'s self-comparison, which is the control for every other diff. |
| `cli/test_cli_registry_surface.py` | The registry and doctor subprocess contracts: `--list` / `--json` / single-capability exit codes and `RESULT=` tokens, the doctor's three blocks and its 0-versus-3 verdict, and the view flags that print one block only. |
| `unit/test_iat_callsite_restore.py` | The import-layer repair's four decisions on a generated image: the six-byte window on either side of a call, the refusal when the sixth byte is the next instruction, the refusal when two slots hold one address, the pairing rule, and the verdict token that follows from the evidence rather than from a count. |
| `cli/test_cli_iat_callsite.py` | The repair's published contract as a subprocess: a report-only run writes nothing into the working directory, `--out` leaves the input byte-identical and the written sites re-decode to indirect calls, the PE32 displacement is an absolute slot address where the PE32+ one is relative, and a refusal is exit 1 rather than a quiet success. |
| `cli/test_cli_dart_snapshot.py` | The snapshot reader's CLI: the refusals and their exit codes, `--find` case-insensitivity and the exit code a miss produces, `--at` by file offset and by virtual address, and the `--json` document. |
| `integration/test_pipeline_end_to_end.py` | Build, locate, patch, re-read: the patched bytes, the untouched control site, the recomputed checksum, and a control run in which nothing changes. |
| `integration/test_across_builds.py` | Pattern hit counts compared between two builds, including the case where a patch removes the very byte the pattern was located with. |
| `run_tests.py` | The one-command entry point; prints the install line and exits 3 when pytest is missing. |
| `benchmark.md` | The regression matrix: dimension, target, scripts, measured result, strength label. |

## How to run it

```
python -m pytest -q                     # from the repository root: the whole suite
python -m pytest -q -m unit             # one layer: unit, cli or integration
python tests/run_tests.py               # same thing, with a clear message if pytest is missing
python tests/run_tests.py -q -m unit    # arguments are forwarded to pytest
```

If pytest is not installed: `python -m pip install pytest`.

## What it deliberately does not do

It does not attach a debugger, load a driver, open a socket, read a device, or execute a target.
Those routes are conditional on host hardware and policy, and a suite that asserts them is either
fragile or dishonest. The recorded experiments for them live in `benchmark.md` with the strength
label that says whether they were reproduced here.

It does not commit or download a sample. Every PE input is generated into a temporary directory
by `make_fixture_pe.py` on the way in and removed with the session, so the suite carries no
binary, no licence question and no dependency on a target that changes underneath it.

It does not assert on timing. The only time in the suite is a generous per-process timeout, which
exists to turn a hang into a failure rather than to measure throughput. The timings in
`benchmark.md` were taken by hand and are recorded as measurements, not enforced as limits.

It does not test the reference documents' prose. Cross-reference, budget and routing checks belong
to the repository's own gates.

## Fixtures

`conftest.py` builds four images per session, addressing `make_fixture_pe.py`'s own recipes rather
than duplicating their shapes:

| Name | Recipe | Why it is in the suite |
| --- | --- | --- |
| `pe64` | `default` | PE32+ with every structure the builder can emit, five sections and the planted pair. |
| `pe32` | `pe32` | The 32-bit optional-header layout, whose field offsets differ. |
| `bare` | `bare` | No exports, resources, TLS or debug directory, so the absent cases are representable. |
| `fatalign` | `fat-align` | A 4 KiB file alignment, large enough that `.rdata` has no virtual-only tail. |

The builder's `no-virtual-tail` recipe is deliberately not used: with a zero virtual pad its own
guard rejects the layout it names, so the recipe exits 1 and writes nothing. `fatalign` covers the
same property. `cli/test_cli_surface.py` pins that failure so the recipe cannot quietly stay
broken.

`conftest.build_synthetic_elf` builds a second kind of input, because `dart_snapshot_strings.py`
reads an ELF shared object rather than a PE and no sample of any kind may be committed: a valid
ELF64 header, a `.text` of sixteen bytes carrying neither idiom the pool observer counts, a
`.rodata` holding one one-byte (ASCII) literal and one two-byte (UTF-16LE) literal at known
offsets, and a section-name string table. It is a container to hold the reader against, not a
Flutter snapshot: the counts the script's docstring records came from a real 26.8 MB `app.so` and
cannot be reproduced from a generated file, which is why the counting logic is asserted in both
directions -- zero on the fixture, non-zero on hand-built bytes -- and the real measurement stays a
recorded observation.

## Conventions

Marker is the layer: `unit` for pure functions and parsers, `cli` for subprocess contracts,
`integration` for the no-debugger flows. All three are declared in `pytest.ini`, and
`--strict-markers` is on, so an undeclared marker is an error rather than a warning. There is no
slow or deselected tier and therefore no `--runslow` switch: nothing in the suite needs one, and
declaring a marker that no test carries would be decoration.

A missing prerequisite skips with a printed reason: an interpreter older than the 3.9 the scripts
declare, or a local tool a check needs. A wrong result always fails. Nothing is skipped silently,
and no test was weakened to pass.

Untrusted sample text is assembled from fragments in `conftest.py` rather than quoted whole. The
suite has to contain strings shaped like pasted credentials and profile paths in order to prove
the scanner catches them; written whole they would also be caught by the repository's own
pre-publish scan. The runtime values are identical, and the source text matches nothing.

## Deviations the suite pins

Each of these is a measured behaviour that differs from what the corresponding docstring implies.
They are asserted as observations so a fix shows up as a test change instead of as a quietly
gained or lost result, and each is also recorded in `benchmark.md`.

| Where | Measured behaviour |
| --- | --- |
| `make_fixture_pe.py` | The `no-virtual-tail` recipe contradicts its own guard and cannot be built. |
| `peutil.py`, `pepatch.py` | A file truncated mid-header raises `struct.error` and prints a traceback instead of using the documented exit code; a fully non-PE input does use it. |
| `pepatch.py` | A missing `--spec` file reaches an unguarded `open()` and prints a traceback. |
| `aob_scan.py` | A bare JSON array of patterns raises `AttributeError`; only the object form is accepted. |
| `scan_leaks.py` | Exit code 3 is documented for an empty scan surface and is unreachable: an empty root reports `RESULT=clean` and exits 0. |
| `scan_leaks.py` | The bare-GUID exemption matches inside the machine-guid rule's own match text, so that rule cannot produce a finding. |
| `scan_leaks.py` | The `.NET` entry matches any `.net` TLD, so a hit inside such a host is suppressed as a platform term. |
| `scan_leaks.py` | The `<[a-z_]+>` placeholder alternative is wrapped in `\b`, so it cannot match bracketed prose. |
| `hwprobe.py` | Two route rows answer with a sentence instead of one of the three documented states. |
| `pepatch.py` | `--allow-resize` writes in place, so the file does not grow and the bytes after the site are overwritten rather than shifted. |

## Limits

Coverage is bounded by the fixture, not by the reader. A synthetic image exercises every structure
the builder emits and nothing it does not; a route that needs a device, a debugger, a driver or a
network is out of scope here and is listed as not run in `benchmark.md`. The suite also cannot say
whether a pattern will survive a build it has never seen, which is why the comparisons it does make
are between two images it built itself.

The scripts under `skills/exe-reverse/scripts/` were being edited while this suite was written, and
several assertions moved with them. Re-run the suite after any change to those files rather than
trusting a count recorded earlier.