diff --git a/.gitignore b/.gitignore index 87fb8cb..ef71296 100644 --- a/.gitignore +++ b/.gitignore @@ -45,6 +45,11 @@ tools/ # is all it would take to commit them, so the rule is explicit here rather than resting on "nobody # has added them yet". .ref/ +# The same corpus under the undotted name. Both spellings exist because a clone lands as `ref/` +# and a hand-made workspace as `.ref/`, and only one of them was covered: `git add -A` with the +# undotted name staged 3.9 MB of third-party write-ups, upstream .git directories and GPL sources +# into a repository that ships neither. One rule per spelling, so neither depends on the other. +ref/ # sandbox installs made while testing `npx skills add .` against this repo .claude/ diff --git a/docs/tool-verification/EXTENSION-absorption-themida.md b/docs/tool-verification/EXTENSION-absorption-themida.md new file mode 100644 index 0000000..c252379 --- /dev/null +++ b/docs/tool-verification/EXTENSION-absorption-themida.md @@ -0,0 +1,75 @@ +# Extension: absorbing an import-virtualisation pass + +What this pass covered, what it verified externally, and what it deliberately did not assert. The +material was three public write-ups of one protector family plus the source tree of an open-source +unpacker for it, read locally. No URL and no author is recorded here: neither was opened in this +session, and a citation that was not verified is a claim dressed as a reference. + +The organising question is the one that decides what can be written down at all: **what is +mechanically true of an import layer once a protector rewrites it, and what belongs only to the +build the writer happened to have?** + +## What was verified externally + +| Fact | How it was established | Grade | +|---|---|---| +| The import layer is changed in two places, not one: the slot stops holding an address, and the call that read it becomes a relative call | Two write-ups describe the same pair of edits on the same product family, on different builds | `unverified` as target behaviour; `observed` that the sources say it | +| A repair can pair a site to a slot by comparing the site's computed target address with the table's values, then write an indirect call back | Read in the unpacker's fixup routine: it builds an address-to-slot map from the table, reads the relative target, and writes `FF 15`/`FF 25` | `observed` (of the source) | +| A newer generation computes each slot's value from a per-slot constant by arithmetic that is itself virtualised, so the handle-to-API mapping cannot be patched out | The same source states it in a comment at the point where it chooses to trace instead of patch | `observed` (of the source) | +| That unpacker repairs one anti-dump type only, and states that a target virtualising other regions will very likely still crash | Comment at the top of its anti-dump fixer | `observed` (of the source) | +| A dump of a target whose entry is virtualised does not run, and the tool says so rather than shipping it | Its own documentation | `observed` (of the documentation) | +| A working dump of such a target can be bound to the machine and session it was taken on, because some anti-dump checks compare module base addresses | The same documentation | `observed` (of the documentation) | +| A DLL dump loses relocation information | The same documentation | `observed` (of the documentation) | +| Targets from the late 2000s may only unpack on the OS generation they were built for | The same documentation | `observed` (of the documentation) | +| The operating system applies compatibility fixes keyed to an executable's path, so a program that crashed once can behave differently later with identical bytes | The same documentation names the mechanism and the store | `observed` (of the documentation); the store itself was not read here | +| One later build leaves the first code section untouched and only redirects the table | A write-up, on one build | `unverified` | +| A small, fixed set of runtime-owned imports is redirected twice | A write-up, on one build and one compiler | `unverified` | +| The filler byte of the 6-to-5 byte rewrite appears after the call, before it, and not at all, in different builds | Two write-ups plus the unpacker's handling of a site whose call is not at offset 0 | `unverified` | +| One write-up's version and date are internally inconsistent — it dates the build to a year, then calls the same build more than a decade old | Read here, in the source text | `observed` | + +The last row is the reason the material was absorbed as mechanism only. A source that is wrong about +its own subject in one place can be wrong in another, and the wrong part does not announce itself. + +## What this changed in the skill + +- **New reference**, `references/iat-virtualization-and-call-site-rewriting.md`: the two-edit model, + the pairing rule, the six-byte window and its three cases, the version-divergence table, the + double-redirected set, the deliverable decision (repair / extract / observe / report), the limits + an automatic unpacker inherits, and the verification rules. +- **Entry point**: five symptom rows and one route row, because "the import table parses but holds + handles" and "a dump dies at its first imported call" are not findable from the existing rows. +- **`references/protectors-and-packers.md`**: one recognition row and a short section. The file states + that it makes no product-level claims, so the mechanism stayed there and the detail did not. +- **`references/memory-and-dumps.md`**: call sites are part of a dump, not a detail of the import + table, so the rebuild step now says that repairing the slots can restore a table nothing reads. +- **`references/dynamic-debugging.md`**: the instrument rule the second write-up actually + demonstrates — a target that runs at full speed and dies when stopped has a perturbation detector, + and the response is a different breakpoint family, not more hiding options. +- **`references/environment.md`**: the compatibility-shim fact, because it changes what "it ran + once" means and it lives in neither file. +- **`scripts/iat_callsite_restore.py`** with a unit and a CLI layer, registered in the capability + registry as `call_site_restore`. + +## What was deliberately not asserted + +- **No address, register name, opcode listing or script from any single build.** Those are the parts + of the source material that stop being true first, and the skill already carries a rule that a + build-level claim travels in the same sentence as the hash it was observed on. +- **No claim that the script works on a real protected sample.** It was exercised against a + generated image whose three layout cases were written on purpose, so its rows are fixture-observed + and say so. The route it implements is `unverified` against a protector until someone runs it. +- **No claim that repairing the call sites produces a working binary.** Virtualisation, anti-dump and + self-integrity checks are separate problems and the reference says so in its own last section. +- **No claim that the filler byte is always a NOP.** One of the three placements has no filler, which + is exactly why the script refuses that case instead of writing five bytes and a guess. +- **No product-strength opinion**, and none of the numeric claims from the write-ups (counts of + redirected imports, section growth, timings) survived into the reference. + +## Why this route is probed rather than declared + +The repair is arithmetic over a file, so it can be tested without a target: a generated image, three +sites with deliberately different neighbours, and a slot range the import directory does not +describe. That is what the tests do, and it is why the script ships with refusals rather than +fallbacks — every fallback in this area (pick the first matching slot, overwrite the sixth byte, skip +a site silently) produces a file that loads and misbehaves later, which is the failure shape this +repository exists to make visible. diff --git a/docs/tool-verification/FINDINGS.md b/docs/tool-verification/FINDINGS.md index 6f01570..ff3d6ac 100644 --- a/docs/tool-verification/FINDINGS.md +++ b/docs/tool-verification/FINDINGS.md @@ -154,6 +154,30 @@ on this host rather than an error. *Fix and check:* long values wrap onto continuation lines instead of being cut. A test asserts a long value survives into the output intact. +## Repair-script defects + +**F16. A slot's magnitude was read as its meaning, so an ordinary import table looked protected.** + +*Symptom:* the import-layer repair's first version classified every slot whose value was below the +image base as a handle, and its verdict rule turned that class into `needs-trace` — a +protected-image verdict for a file that was never protected. Nothing about the output looked wrong: +the classes were counted, the verdict followed from them, and the report was internally consistent. + +*Root cause:* an unbound import table holds RVAs, and an RVA is also a number below the image base. +The classifier inferred intent from magnitude instead of testing whether the value resolves to +something in the image, or whether a call targets it. The same mistake in the other direction would +have called a real handle an address. + +*How it surfaced:* while reading `peutil.py`'s import walk to build the slot inventory. The reader +returns the table's entries as they are in the file, which is what made "below the base" visibly +ambiguous rather than obviously a handle. + +*Fix and check:* the classes are positions (`zero`, `below_image`, `in_image`, `above_image`) and the +verdict follows from the pairing rather than from a class. The regression is +`tests/unit/test_iat_callsite_restore.py::test_a_clean_image_reports_no_pairs`, which asserts +`no-pairs` on a generated unbound image — the case the old rule reported as `needs-trace`. The +reference states the correction in prose, because the same inference is easy to repeat by hand. + ## Gate, harness and lint defects **F11. The hard index-row limit was computed and then unable to fail anything.** diff --git a/skills/exe-reverse/SKILL.md b/skills/exe-reverse/SKILL.md index d95141a..e4a001d 100644 --- a/skills/exe-reverse/SKILL.md +++ b/skills/exe-reverse/SKILL.md @@ -143,6 +143,11 @@ Load the named file **before your next command**, not after your next three atte | Strings you expect are absent; the import table resolves almost nothing | `references/protectors-and-packers.md` | | One function is a many-thousand-line switch over a value that is loaded from memory | `references/virtualization-and-flattening.md` | | A dump runs but every call lands in a jump table, or the code differs after a restart | `references/protectors-and-packers.md` | +| The import table parses, but its slots hold tags or addresses inside one section instead of the modules they name | `references/iat-virtualization-and-call-site-rewriting.md` | +| Calls in the code are relative (`E8`) where the compiler emits `call [iat]` | `references/iat-virtualization-and-call-site-rewriting.md` | +| A dump loads and dies at the first call into the platform | `references/iat-virtualization-and-call-site-rewriting.md` | +| A dump works here and fails after a reboot or on another host | `references/iat-virtualization-and-call-site-rewriting.md` | +| Every code breakpoint or pause triggers an exception storm, and a full-speed run is fine | `references/dynamic-debugging.md` | | The image you dumped will not load — sections are misaligned or headers are gone | `references/memory-and-dumps.md` | | The file checksum no longer matches and something now refuses the binary | `references/patching.md` | | Your signature-based patch worked on one version and misses on the next | `references/aob-and-signatures.md` | @@ -218,6 +223,7 @@ is the most common way an hour disappears. | The same patch across several versions | `scripts/aob_scan.py`, then its `--across` mode against the other build | `references/aob-and-signatures.md` | | Runtime behaviour to be located | A debugger, then hardware breakpoints if software ones are detected | `references/dynamic-debugging.md` | | Code that cannot be read at all | Protector identification, then the unpack route | `references/protectors-and-packers.md` | +| An import table that parses but holds handles, or relative calls where indirect calls belong | The pairing rule first; the site list is part of the deliverable | `references/iat-virtualization-and-call-site-rewriting.md` | | A .NET assembly | Managed tooling; do not start with a hex editor | `references/managed-dotnet.md` | | Unity IL2CPP (`GameAssembly.dll` + metadata) | Classify the metadata placement before any patch; the names and the code are separate files | `references/unity-il2cpp.md` | | A packaged Python program | Identify the packager and the interpreter version first; both decide whether anything else is readable | `references/python-runtime-bundles.md` | diff --git a/skills/exe-reverse/references/coverage-and-limits.md b/skills/exe-reverse/references/coverage-and-limits.md index f96845c..65af40e 100644 --- a/skills/exe-reverse/references/coverage-and-limits.md +++ b/skills/exe-reverse/references/coverage-and-limits.md @@ -49,7 +49,7 @@ named gap. | Dynamic debugging workflow | `unverified` | the route is described and a debugger exists on at least one host; no breakpoint log, stepping session or debugger transcript is recorded in this repository | an interactive debugger, and a target you are permitted to run | | Runtime instrumentation | `unverified` | an instrumentation runtime is reported present by the host probe (`observed` for that inventory); no run against a target is recorded | the instrumentation runtime, and the target's own build identity | | Injection and hooking | `unverified` | the technique is described, not performed here; the observer-versus-interceptor split in `references/handoff-boundaries.md` decides when it is allowed at all | an injection mechanism and a hook library | -| Dump and repair | `unverified` | nothing here dumps a process image, rebuilds a dumped image, or repairs imports | a dumper, an image rebuilder, and a way to show the repaired image is the image that ran | +| Dump and repair | `unverified` | nothing here dumps a process image or rebuilds one from memory. The import-layer repair behind `scripts/iat_callsite_restore.py` is exercised on a generated image -- pairing, both filler positions, and both refusals (`observed` for the tool's decisions) -- and no protected sample was repaired here | a dumper, an image rebuilder, and a repaired sample that runs on a clean host | | Licensing-mechanism classification | `inferred` | the classification is a decision procedure over observed structure: which layer holds the decision, local versus remote, one check versus a check repeated at every use | nothing to classify with, but a capture path and a second account for the remote rows | | Algorithm recovery | `inferred` | the route is described; no implementation here has been checked against the target's own acceptance of a produced value | a decompiler or symbolic engine, and the target's acceptance as the only real oracle | | Network-activated licensing | `unverified` | no capture, no replay and no account pair is recorded here | a capture path and a test account | diff --git a/skills/exe-reverse/references/coverage/route-inventory.md b/skills/exe-reverse/references/coverage/route-inventory.md index 4e713d2..dfb6fdf 100644 --- a/skills/exe-reverse/references/coverage/route-inventory.md +++ b/skills/exe-reverse/references/coverage/route-inventory.md @@ -31,6 +31,7 @@ becomes `run` only when someone runs it and records the output. | Byte patching | making a change that fits an existing instruction or constant slot | Python only, plus a copy to write to | `run` | | Signature location | finding a site across builds rather than at one address | Python only; a second build for the comparison | `run` | | Build comparison | locating what changed between two builds | two builds | `run` | +| Import-layer repair | restoring the call sites a protector rewrote, once the code is decrypted | a rebuildable image, the address space it was captured in, and the slot values — plus a trace when the slots hold handles rather than addresses | `run` for the rewrite on a generated image; `argued` against a protected target | | Identity scanning | publishing derived material without publishing the target | Python only | `run` | | Managed assembly analysis | reading and changing a .NET assembly | a decompiler or disassembler (`ilspycmd`, `ildasm`) to read it, `de4dot` for a renamed one, an assembler or dnSpyEx to change it — the repository ships none | `mapped` | | Installer and package extraction | getting at the payload of an installer without running it | extraction tooling; the container type decides which | `mapped` | diff --git a/skills/exe-reverse/references/dynamic-debugging.md b/skills/exe-reverse/references/dynamic-debugging.md index d3773e2..49a9aeb 100644 --- a/skills/exe-reverse/references/dynamic-debugging.md +++ b/skills/exe-reverse/references/dynamic-debugging.md @@ -42,6 +42,21 @@ count (which bit controls which slot, how execution breakpoints differ from data processor-specific and must be read from the processor specification for the exact family (`unverified` here — the numbers quoted above are the count only). +**When every code breakpoint produces an exception storm, change the family, not the stealth.** A +protected target can run to completion at full speed and die as soon as it is stopped: a distributed +check reacts to the paused state, or to a byte that was replaced, and the response is a burst of +exceptions that ends the process. Adding hiding options to the same instrument then buys a few more +instructions and no more analysis, because the detector is not looking for a debugger — it is looking +for the perturbation. Two rules follow, and they are measurements rather than preferences: a target +that survives a full-speed run and dies on the first step has told you its detector is +perturbation-based (`inferred`; confirm with the unattached control run described above), and the family that survives is +the one that changes no byte and touches no debug register in the code being watched — a protection +change on a *data* page, or a watch on a region the target allocated for itself. The instrument that +worked in a reported session on this class of target was a memory-access breakpoint on a freshly +allocated region, with no stepping and no register inspection while stopped. Treat that as a lead to +test on your target, not as a procedure: the mechanism is "watch data, do not touch code", and the +address is a property of the sample. + **The decision, therefore:** if the region you need to break in is hashed, verified, or used as a self-check input, a software breakpoint is the wrong instrument and the finding it produces will be "the target died with no explanation". Move to hardware breakpoints, memory breakpoints, or diff --git a/skills/exe-reverse/references/environment.md b/skills/exe-reverse/references/environment.md index 43f9236..1447241 100644 --- a/skills/exe-reverse/references/environment.md +++ b/skills/exe-reverse/references/environment.md @@ -58,7 +58,7 @@ the received baseline, the built output, and the control (an unmodified copy use the launch path and the environment behave normally). Label them in the record by hash, not by name, because names drift and hashes do not. -Two file properties are worth checking once, because both change what a launch means: +Three file properties are worth checking once, because each changes what a launch means: - **Alternate data streams.** A file that arrived through a browser or an archive extraction tool may carry a `Zone.Identifier` stream, and the OS may then gate its execution. The streams on a file are @@ -68,6 +68,15 @@ Two file properties are worth checking once, because both change what a launch m size, same SHA-256; observed here against the desktop text editor in the system directory) is a convenient fixture, but its launch behaviour may depend on its location or on a component the original has and the copy does not. Treat an unexpected exit as an environment question first. +- **Compatibility fixes the operating system applied by itself.** Windows keeps a per-program record + of compatibility fixes it has decided an executable needs, keyed to that executable's path, and + applies them on later launches; a program that crashed once can therefore behave differently on the + next run with byte-identical content (documented behaviour of the compatibility store; `unverified` + here, and the record lives under the current user's software hive in a key naming compatibility + flags). Unpacking is where this costs most: the packer and the target then run under different + conditions, and the difference is in neither file. A target that misbehaves only on this host, only + after an earlier failure, is a candidate; copying it to a different path is the cheapest test and + the result is worth recording, because it changes what "it ran once" means. ## Record the host, because it decides which routes exist diff --git a/skills/exe-reverse/references/evidence-summary.md b/skills/exe-reverse/references/evidence-summary.md index 09b5044..fefd9ac 100644 --- a/skills/exe-reverse/references/evidence-summary.md +++ b/skills/exe-reverse/references/evidence-summary.md @@ -43,6 +43,7 @@ repository generates itself; no third-party sample is required to reproduce any | Patch refusal on an unproven length | a patch with no length reference reports `[length unproven: pass expect or original-length]` instead of asserting equal length | `observed` | same command with `--bytes CC` and no `--expect` | | Byte-pattern location | reports the hit count, each site's offset/RVA/section, and a verdict; a pattern that matches twice is reported as `ambiguous` | `observed` | `python skills/exe-reverse/scripts/aob_scan.py --patterns ""` | | Cross-build pattern comparison | `--across` reports each build's hit count and states when the count changed | `observed` | `python skills/exe-reverse/scripts/aob_scan.py --patterns P --across ` | +| Import-layer repair | on a generated image carrying both filler positions, a site with no filler, and a target two slots hold: two sites are paired and rewritten, the other two are refused by name, the input hash is unchanged, and every written site re-decodes to an indirect call through the slot the report names | `observed` (tool behaviour on a fixture; no protected sample was repaired here) | `python -m pytest -q tests/unit/test_iat_callsite_restore.py tests/cli/test_cli_iat_callsite.py` | | Offline fixture generation | builds a structurally valid PE32 or PE32+ image, deterministic in bytes, with a correct checksum, that the reader parses back | `observed` | `python skills/exe-reverse/scripts/make_fixture_pe.py --out f.exe --json` | | Identity scanning | scans the tree, reports findings with context, prints `RESULT=` and sets an exit code suitable for gating | `observed` | `python skills/exe-reverse/scripts/scan_leaks.py` | | Host capability probing | prints a per-route verdict and reads CPUID through a local stub; bits whose meanings are not verified are printed **raw** rather than decoded | `observed` | `python skills/exe-reverse/scripts/hwprobe.py` | diff --git a/skills/exe-reverse/references/iat-virtualization-and-call-site-rewriting.md b/skills/exe-reverse/references/iat-virtualization-and-call-site-rewriting.md new file mode 100644 index 0000000..0e53415 --- /dev/null +++ b/skills/exe-reverse/references/iat-virtualization-and-call-site-rewriting.md @@ -0,0 +1,198 @@ +# IAT Virtualization and Call-Site Rewriting + +Goal: decide whether the import layer you are looking at is a table of addresses or a table of +handles, and either restore the two things a protector changed together — or establish that the +deliverable does not need them restored. + +## When to load this + +Load it when the import table exists but does not behave like one: + +- the import directory parses, and every slot holds a small value that is not an address, or a + value that lies inside the protector's own section +- a call in the first code section is `E8 rel32` where the compiler's own output for that source + would be `call dword ptr [iat]` +- `FF 15` occurs a handful of times in a code section of a hundred kilobytes, while `E8` occurs + thousands of times +- a dump loads, then faults or exits the first time the program calls an imported API +- someone has asked you to "repair the IAT" and there is no name in the image to repair it to +- you are about to describe a dump as "almost working" + +The last row is the one this file exists for. "Only the IAT is broken" is a diagnosis, and on a +protector that rewrites call sites it is usually half a diagnosis. + +## Two edits, not one + +An import-based call is two objects that a protector can treat separately, and this family of +protectors treats both: + +| Object | What the compiler emitted | What the protector makes it | Restoring it alone leaves | +|---|---|---|---| +| The IAT slot | the address of the imported function, resolved by the loader | a handle: a small tag, or the address of a stub inside the protector's section | a correct table nothing reads — every call still targets the stub directly | +| The call site | `FF 15 disp32` — call through the slot, six bytes | `E8 rel32` — a direct relative call to the stub, five bytes plus one filler byte | every call now reads a slot that still holds a handle, so it lands in the stub that no longer runs | + +A repair that touches one without the other produces a file that loads and then dies in a way that +looks like a bad dump. Both edits are needed, and the second is the one with a layout problem. + +**Why the filler byte matters.** The rewrite is 6 bytes to 5, so the protector has one byte left +over per site. Public write-ups of 32-bit builds describe three placements: a `NOP` after the +relative call, a `NOP` before it, and no filler at all where the byte that follows is the next +instruction (`unverified` for any given build; observed as three distinct forms across two +write-ups). Restoring the site reverses a 5-byte form into a 6-byte form, so it fits only where a +sixth byte is actually available. Where it is not, the repair is a **structural** edit — the +following instruction has to move — and no automatic writer can decide that from bytes alone. + +**One more difference that must be respected.** In a 32-bit image `FF 15 disp32` addresses the slot +absolutely; in a 64-bit image it is RIP-relative, so the displacement is relative to the *end* of +the instruction. The same source-level call compiles to the same two bytes and a different +displacement. Write the wrong one and the site calls whatever lives at that address. + +## Recognising the layer without running anything + +| Evidence | What it indicates | Strength | +|---|---|---| +| Import directory present, and its slots hold values that resolve to nothing in the image | Slot replacement with a tag or an index rather than an address | `inferred` | +| Import directory present, slots hold addresses inside a section that is executable and writable | Slot replacement with a stub address in the protector's own code | `inferred` | +| `E8`/rel32 everywhere, `FF 15`/`FF 25` almost absent, in a native image | Call sites rewritten from indirect to direct | `inferred` | +| A relative call whose computed target lies outside every section of the image | The target is a stub the protector allocated, not code in this image, and the image cannot run as it stands | `inferred` | +| Sections named for the protector, with one large executable and writable region | The stub table lives there | `inferred` | +| A run of short functions in that region, each ending in a jump through a pointer | One stub per imported API, dispatching into the protector's resolver | `inferred` | + +**A value below the image base is not by itself evidence of anything.** An ordinary unbound import +table holds RVAs, and an RVA is also a number below the base. The first version of the repair script +classified every such slot as a handle and reported ordinary unbound images as protected, which is +the failure mode this repository calls a confident wrong answer. The discriminator is whether the +value resolves to something in the image, and whether any call targets it — not its magnitude. A +tool that reports slot classes should report them as positions (below the image, inside it, above +it) and let the pairing decide what they mean. + +Counting the two call forms is the cheapest confirmation and it needs no new tool: +`scripts/aob_scan.py` takes a pattern and returns how many sites matched, so `FF 15` and `E8` over +the same section turn an impression into two numbers. Record both, and record the section they +were counted over. + +## The repair, in order + +1. **Establish the address the code runs at.** Sites are relative, so every target you compute is + the address in the session the image was captured from, not the file's preferred base. A dump + taken at a relocated base is the normal case. Take the base from the capture, not from the header + — and record which one you used. +2. **Inventory the slots.** Three sources, in descending order of trust: the import descriptors if + they survived, the import address table directory's range if it did, and the addresses that a + write watch reported. Say which source the inventory came from; they do not always agree. +3. **Pair a site to a slot by value.** The site's target address equals the slot's value. This is + the whole pairing rule, and it is also the mechanism an open-source unpacker's fixup routine + implements (`observed` in that source: it reads the target from the relative call, looks it up in + an address-to-slot map built from the table, and writes an indirect call back). +4. **Refuse the ambiguous pairs.** Two slots can hold the same address — a second redirect pass, a + bound slot, a duplicated descriptor. A map keyed by value silently keeps the last one, which + produces a file where one call goes through the wrong slot and still works often enough to be + mistaken for a success. Report the duplicate and leave both alone unless a person resolves it. +5. **Decide the sixth byte.** Filler after: rewrite in place. Filler before: the six-byte window + starts one byte earlier. No filler: stop, and report it as a structural repair. +6. **Write, then re-read.** Re-decode every rewritten site from the written file and confirm it now + addresses the intended slot. A patch that assembles and is never reached is the failure this + step exists for. +7. **Report the leftovers.** Sites with no slot and slots with no site are the content of the + report, not noise around it: the first is a call the protector resolved by some other route, the + second is an import nothing in this code section uses. + +## Version divergence, and why this is a map rather than a fact + +One product family, three observable behaviours, from two public write-ups and one unpacker's +source: + +| Behaviour | Where it was observed | What it changes about the plan | +|---|---|---| +| The first code section is modified: call sites become relative calls | 32-bit write-up, older build | Both edits are required, and the dump contains the rewritten sites | +| The first code section is **not** modified; only the slots are redirected | another write-up, a later build | Site rewriting is unnecessary; the work is the slot-to-name mapping and nothing else | +| Every site is still written, but with the bytes already there | an unpacker's source comment | A write-watch fires thousands of times and changes nothing; a plan built on "count the writes" spends hours on no-ops | +| Slot values are computed from a per-slot constant by arithmetic that is itself virtualised | the same source, for its newest supported generation | The handle-to-API mapping **cannot** be derived offline. It is traced or not obtained | + +All four are claims about particular builds, and none is `observed` on a target here: the first two +come from write-ups and the last two from a tool's source, which records what its author found rather +than what this repository ran. The transferable part is the **shape** — one product can present as any +of these, so the first task on a sample is to decide which row it is in, with evidence, before writing +any repair. + +## The set of APIs that is redirected twice + +One write-up reports that a small set of runtime-owned imports is redirected by a second, different +mechanism, so a naive pairing resolves them to the wrong function or to a duplicate of a previous +entry (`unverified`). The members it named are heap and error-state routines plus process exit and +unwind. Two observations about that list are worth more than the list: + +- The set is small and it is **language- and version-specific**. It reflects which imports the + runtime calls before it can call anything else. +- The useful artifact is not the list but the **test**: for each slot the pairing left unresolved, + call it once and see which module and export the call lands in. That answers the question on the + sample in front of you, and it answers it for a build nobody has written about. + +## Deciding the deliverable: repair, extract, observe, or report + +Repairing the import layer assumes there is a native image worth repairing. When the entry point or +the region that matters is virtualised, there is no original entry point in machine code, so there +is nothing to point the header at and no site list to restore. + +| Situation | Deliverable | What "verified" then means | +|---|---|---| +| Code is restored, only the import layer is wrong | a repaired image | it loads, and each restored site's call returns from the intended export | +| Entry or hot region is virtualised, the payload is what is wanted | the payload, as data | it disassembles coherently at a stated base, and its configuration cross-references | +| The question is what the code does, not what the file is | a report | every claim names the site it came from | +| A local change is enough | a live patch | the patch is reached at run time, not merely present | + +A scanner hit on an extracted payload is a **lead in both directions**: a detection is evidence the +bytes are the real thing, and a clean scan is not evidence they are not. Neither substitutes for +reading the entry and the configuration. + +## What an automatic unpacker settles, and what it does not + +An unpacker automates the three steps a person performs by hand: stop at the right moment, capture +what the protector wrote, and apply the fixups offline. Its output is a **dump with the call-site +layer repaired**, and it inherits every limit of the dump it came from. Documented limits, from the +tools' own material (`observed` in their documentation and source): + +| Limit | Consequence | +|---|---| +| Anti-dump measures on the entry or on virtualised code are not repaired | A dump whose entry is virtualised does not run, and the tool says so | +| Some anti-dump checks compare module base addresses captured at run time | The dump may run on the machine and session that produced it and fail after a reboot or elsewhere | +| A DLL dump loses relocation information | The image cannot be loaded anywhere but the base it was captured at | +| Targets built for an OS generation two decades old may only unpack on that generation | The result depends on the unpacking host, which is a property of the experiment | +| Compatibility shims applied by the operating system to a program that crashed before change its behaviour | The unpacker and the target can see different behaviour, and the shim is invisible in the binary | +| Newer builds of the protector detect hardware breakpoints | The instrument question comes before the unpacking question | + +Two of these are verification rules rather than unpacking rules, and they are the ones most often +skipped: **a dump that runs is not yet a portable artifact** — re-run it after a reboot and on a +second host, and record both results; and **the host is part of the experiment** — the same file can +behave differently because the operating system shimmed it, so check before attributing a rare +failure to the protector. + +## Verification + +1. The loader accepting the file is a statement about layout. Exercise the calls before saying more. +2. For each distinct slot that was restored, at least one call through it must return. A run that + never reaches the restored region proves nothing about the repair. +3. A base-address-bound dump is a limitation to state, not a defect to hide: name the base and the + session. +4. Record which slots were paired by value and which by tracing. They carry different confidence. +5. A refusal is a result. "Nine sites restored, one refused because its sixth byte is the next + instruction" is a finished finding; "the IAT is fixed" is not. + +## What this file does not contain + +- No addresses, register names, opcode sequences or script listings from a single build. Those are + properties of one sample and are the first thing that stops being true. +- No claim about any protector's strength. +- No promise that repairing the import layer produces a working binary. Virtualised regions, + anti-dump checks and self-integrity checks are separate problems with separate files. + +## Related material in this skill + +- `references/protectors-and-packers.md` — is this a compressor or a protector, and is unpacking + the right route at all +- `references/memory-and-dumps.md` — turning a captured image into a file the loader accepts +- `references/virtualization-and-flattening.md` — what a virtualised region is, when there is no + entry point to restore +- `references/dynamic-debugging.md` — choosing the instrument, and what a session must record +- `references/verification.md` — what a passing run proves +- `scripts/iat_callsite_restore.py` — the pairing and rewrite above, offline, with refusals diff --git a/skills/exe-reverse/references/memory-and-dumps.md b/skills/exe-reverse/references/memory-and-dumps.md index cc6f88c..4af272b 100644 --- a/skills/exe-reverse/references/memory-and-dumps.md +++ b/skills/exe-reverse/references/memory-and-dumps.md @@ -146,6 +146,12 @@ the loader produced. In order: - *The entry point.* The address the process was executing is not automatically the image's entry point; the entry point is a header field, and it must be restored from a source rather than inferred from where the target happened to be. + - *Call sites that no longer read the table.* A protector that rewrote indirect calls into relative + calls left the dump with code that never touches the import slots, so repairing the slots restores + a table nothing reads. The dump carries the rewritten form and the repair is a second edit over + the first code section, one site at a time, with a layout question at each site. + `references/iat-virtualization-and-call-site-rewriting.md` states the pairing rule, the sixth-byte + decision and the cases that must be refused. 6. **Repairing means, concretely:** a header that parses and whose directory RVAs and sizes agree with the laid-out sections; a section table whose raw offsets and sizes match the bytes actually written; an import situation that is either resolvable at load time or explicitly static with the binding diff --git a/skills/exe-reverse/references/protectors-and-packers.md b/skills/exe-reverse/references/protectors-and-packers.md index fb676f8..adc74c7 100644 --- a/skills/exe-reverse/references/protectors-and-packers.md +++ b/skills/exe-reverse/references/protectors-and-packers.md @@ -56,6 +56,7 @@ All of these come from the file itself, before any execution. | Entropy near maximum across an entire executable section on disk | Compressed or encrypted content | `inferred` | | Entry point RVA outside the first executable section | The stub is not the program; the real entry point is computed later | `inferred` | | Import directory present but resolving to one or two libraries | The real imports are resolved by the stub at run time | `inferred` | +| Import directory present, and its slots hold tags, or addresses inside one section, instead of the modules they name | The table holds handles rather than addresses, and the call sites that used it were probably rewritten to reach the handle directly | `inferred` | | Import directory absent or pointing into a writable section | Import destruction, documented as a feature by more than one commercial protector | `observed` in this project's vendor-documentation pass | | A writable and executable section, or a section that becomes executable after a protection change | Self-modifying stub, or a run-time code region | `inferred` | | A thread-local-storage callback array pointing at code | Work is done before the entry point ever runs | `observed`: the PE specification documents the callback array as a null-terminated array of TLS callback functions, called with a process-attach reason when a new process has started | @@ -66,6 +67,20 @@ All of these come from the file itself, before any execution. JPEG in a resource section produces the same reading as encrypted code. Confirm with a second kind of evidence, and record which evidence you used. +## When the import layer is not an import table + +A protector can leave the import directory structurally intact and replace what it contains: each slot +holds a handle or a stub address instead of a resolved import, and the calls that used to go through +the table became relative calls to that stub. Both halves were changed together, so they are restored +together — and the second half is a layout decision rather than a byte swap, because the rewritten +call is one byte shorter than the indirect call it replaced. + +This is a different problem from a destroyed import directory, and it is the one that produces a dump +which loads, runs, and dies at the first call into the platform. The identification evidence, the +pairing rule and the cases that must be refused are in +`references/iat-virtualization-and-call-site-rewriting.md`. This file keeps only the route decision: +the damage is specific and recoverable, and the deliverable is a site list as much as it is a file. + ## Locating the original entry point The original entry point (OEP) is the address the program's own startup code begins at. The route is diff --git a/skills/exe-reverse/references/routing.md b/skills/exe-reverse/references/routing.md index f46b171..f6ea366 100644 --- a/skills/exe-reverse/references/routing.md +++ b/skills/exe-reverse/references/routing.md @@ -33,6 +33,11 @@ without a check. | Strings you expect are absent; the import table resolves almost nothing | `references/protectors-and-packers.md` | | One function is a many-thousand-line switch over a value that is loaded from memory | `references/virtualization-and-flattening.md` | | A dump runs but every call lands in a jump table, or the code differs after a restart | `references/protectors-and-packers.md` | +| The import table parses, but its slots hold tags or addresses inside one section instead of the modules they name | `references/iat-virtualization-and-call-site-rewriting.md` | +| Calls in the code are relative (`E8`) where the compiler emits `call [iat]` | `references/iat-virtualization-and-call-site-rewriting.md` | +| A dump loads and dies at the first call into the platform | `references/iat-virtualization-and-call-site-rewriting.md` | +| A dump works here and fails after a reboot or on another host | `references/iat-virtualization-and-call-site-rewriting.md` | +| Every code breakpoint or pause triggers an exception storm, and a full-speed run is fine | `references/dynamic-debugging.md` | | The image you dumped will not load — sections are misaligned or headers are gone | `references/memory-and-dumps.md` | | The file checksum no longer matches and something now refuses the binary | `references/patching.md` | | Your signature-based patch worked on one version and misses on the next | `references/aob-and-signatures.md` | @@ -97,6 +102,7 @@ reference that is loaded at the wrong moment costs context and answers nothing. | `references/memory-and-dumps.md` | process memory as evidence, and turning a dump into something loadable | | `references/anti-analysis.md` | the target fights back: which mechanism fired, and how to tell it from a broken host | | `references/protectors-and-packers.md` | the code cannot be read: recognising the layer and working through it | +| `references/iat-virtualization-and-call-site-rewriting.md` | the import table holds handles and the calls stopped being indirect: pairing, rewriting, and what has to be refused | | `references/virtualization-and-flattening.md` | the code is readable but its shape is wrong: dispatchers, opaque predicates, virtual machines | | `references/symbolic-execution.md` | reading has stopped paying: when a solver or an emulator is the cheaper route | | `references/managed-dotnet.md` | a managed assembly, or a mixed-mode one, is the target | @@ -130,6 +136,7 @@ explicit paths for anything it shells out to. `--help` on each one is the author | `scripts/pe_triage.py` | one-pass triage of an unknown binary: identity, build evidence, protective signals with their measurements, surface, and what to do next | | `scripts/pepatch.py` | equal-length byte patches from a spec or from the command line: read-only input, a new output file, an ambiguity guard, a length-provenance rule, checksum recomputation, and every patched byte re-read from the written file | | `scripts/aob_scan.py` | locate a byte pattern and report its hit count, each site's offset/RVA/section, a verdict (`unique` / `ambiguous` / `pervasive` / `no-match`), and hit-count comparison across two builds | +| `scripts/iat_callsite_restore.py` | pair a relative call a protector left behind with the import slot that holds its target, restore the indirect call, and refuse the three cases a byte swap would corrupt: no sixth byte, two slots holding one address, two windows overlapping. Report-only unless `--out` is given | | `scripts/pe_diff.py` | compare two builds by content hash with the volatile header fields normalised, and report the differing ranges with the RVA each maps to | | `scripts/hwprobe.py` | report which routes **this host** supports: CPUID features read through a local stub, CPUID bits printed raw unless their meaning is verified, OS build and boot flags, privilege, and the tool inventory — with an `available` / `needs-` / `unavailable` verdict per route | | `scripts/capabilities.py` | the capability registry: what each capability needs, how to detect it, which script implements it and which reference documents it | diff --git a/skills/exe-reverse/scripts/capabilities.py b/skills/exe-reverse/scripts/capabilities.py index 449ee40..24d810e 100644 --- a/skills/exe-reverse/scripts/capabilities.py +++ b/skills/exe-reverse/scripts/capabilities.py @@ -626,6 +626,9 @@ ATOMS = { 'literals)', 'kind': 'file', 'script': 'dart_snapshot_strings.py', 'est_minutes': 0, 'basis': 'observed'}, + 'file:iat_callsite_restore.py': {'name': 'scripts/iat_callsite_restore.py (import-layer repair)', + 'kind': 'file', 'script': 'iat_callsite_restore.py', + 'est_minutes': 0, 'basis': 'observed'}, # --- host capabilities, probed rather than assumed ------------------------ 'host:elevated': { @@ -771,6 +774,15 @@ CAPABILITIES = { 'note': 'Content-hash comparison with the volatile header fields normalised, so a ' 'same-size replacement is caught and a rebuild is not reported as a change.', }, + 'call_site_restore': { + 'name': 'Restore call sites a protector rewrote, against their import slots', + 'required': ['python310', 'file:iat_callsite_restore.py', 'artifact:dumped_image'], + 'depends_on': ['pe_structure_read'], + 'scripts': ['iat_callsite_restore.py'], + 'note': 'Offline pairing by value, a six-byte rewrite with the filler byte decided per ' + 'site, and a refusal for the site whose sixth byte is the next instruction. It ' + 'repairs an import layer, never a protector: a virtualised region is untouched.', + }, 'fixture_generation': { 'name': 'Build a synthetic PE fixture offline', 'required': ['python310', 'file:make_fixture_pe.py'], @@ -1212,6 +1224,7 @@ SCRIPT_CAPABILITIES = { 'make_fixture_pe.py': ['fixture_generation'], 'scan_leaks.py': ['identity_scan'], 'fold.py': ['record_folding'], + 'iat_callsite_restore.py': ['call_site_restore'], 'capabilities.py': ['host_capability_probe'], 'doctor.py': ['host_capability_probe'], # Not a Python script, but it belongs to a capability all the same: it instruments a live diff --git a/skills/exe-reverse/scripts/iat_callsite_restore.py b/skills/exe-reverse/scripts/iat_callsite_restore.py new file mode 100644 index 0000000..b500355 --- /dev/null +++ b/skills/exe-reverse/scripts/iat_callsite_restore.py @@ -0,0 +1,525 @@ +#!/usr/bin/env python3 +"""Restore call sites a protector rewrote, by pairing them with their import slots. + +Why this exists: on one family of protectors the import layer is changed twice at once. Each slot +stops holding the address of an imported function and starts holding a handle, and each call that +used to read that slot is rewritten from a six-byte indirect call into a five-byte relative call to +the handle. Repairing the slots alone restores a table the code no longer reads; repairing the call +sites alone sends every call back into a protector that is no longer running. Both edits are the same +repair, and the second one has a layout problem that this script refuses to guess at. + +What it does, in order: + + 1. Inventories the slots: from the import descriptors when they survived, from the import address + table's directory range, and from an explicit --iat range. Records which source each slot came + from, because the three do not always agree. + 2. Scans the executable sections for E8/E9 and computes each site's target address in the address + space the image was captured at (--base, not the header's preferred base). + 3. Pairs a site to a slot when the target address *equals* the slot's value. That is the whole + pairing rule. + 4. Decides the sixth byte. The rewrite is five bytes into six, so it fits only where a sixth byte + is available: a NOP after the call (rewrite in place), a NOP before it (the window starts one + byte earlier), or no filler at all -- in which case the sixth byte belongs to the next + instruction and the repair is *structural*. That last case is refused, not forced. + 5. Rewrites, then re-reads the written file and re-decodes every site to confirm it addresses the + intended slot. + +What it refuses, and why a refusal is the useful output: + + * a site whose target matches two slots (two slots holding the same address). A map keyed by value + keeps the last one silently, so one call ends up going through the wrong slot and still works + often enough to be mistaken for a success; + * a site with no filler byte, because writing there overwrites the first byte of the next + instruction -- a length change, which is a structural edit and not a byte swap; + * two candidate windows that overlap, because neither can be written without destroying the other. + +What it does not do: unpack. It assumes you already have an image whose code is decrypted. It does +not devirtualise, does not resolve handles by arithmetic, and does not carry a table of known +protector versions. When the slots hold handles rather than addresses it reports needs-trace and +stops, because the handle-to-API mapping cannot be derived from the file. + +Usage: + python iat_callsite_restore.py dumped.exe # report only, writes nothing + python iat_callsite_restore.py dumped.exe --out fixed.exe # rewrite and verify + python iat_callsite_restore.py dumped.exe --base 0x2B0000 --iat 0x48D000,0x1000 + python iat_callsite_restore.py dumped.exe --json --full-out report.json + +Exit codes: 0 = the analysis completed and no site was refused (result clean, needs-trace, +report-only or repaired), 1 = at least one site was refused or a rewrite failed verification, +2 = usage error, 3 = the input is not a parseable PE. +""" +import argparse +import hashlib +import json +import os +import struct +import sys + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +from peutil import PE, ParseError, DIR_IAT, s32 # noqa: E402 + +DEFAULT_MAX_SITES = 40 +SECTION_EXECUTE = 0x20000000 +SECTION_CODE = 0x00000020 + +CALL = 0xE8 +JMP = 0xE9 +OP_CALL_IND = 0x15 # FF /2, [mem] +OP_JMP_IND = 0x25 # FF /4, [mem] +NOP = 0x90 +SITE_REL_LEN = 5 # E8/E9 + rel32 +SITE_ABS_LEN = 6 # FF 15/25 + disp32 + + +def sha256(data): + return hashlib.sha256(data).hexdigest() + + +def is_executable(section): + c = section.characteristics or 0 + return bool(c & SECTION_EXECUTE) or bool(c & SECTION_CODE) + + +def read_ptr(pe, rva): + """Pointer-sized value at an RVA, or None when the bytes are not in the file.""" + n = 8 if pe.is_pe32plus else 4 + raw = pe.rva_bytes(rva, n) + if len(raw) < n: + return None + return struct.unpack(' (rva, size).""" + if not text: + return None + body = text.replace(':', ',').split(',') + if len(body) != 2: + raise ValueError('expected RVA,SIZE') + return int(body[0], 0), int(body[1], 0) + + +def slot_inventory(pe, iat_override): + """{rva: {'value': int, 'source': str}} plus the list of sources actually used.""" + slots = {} + sources = [] + + for desc in pe.imports(): + used = False + for fn in desc['functions']: + rva = fn.get('iat_rva') + if rva is None or rva in slots: + continue + value = read_ptr(pe, rva) + if value is None: + continue + slots[rva] = {'value': value, 'source': 'import-descriptor'} + used = True + if used: + sources.append('import-descriptor') + + rva, size = pe.directories.get(DIR_IAT, (0, 0)) + if rva and size: + used = False + step = 8 if pe.is_pe32plus else 4 + for slot_rva in range(rva, rva + size - step + 1, step): + if slot_rva in slots: + continue + value = read_ptr(pe, slot_rva) + if value is None: + continue + slots[slot_rva] = {'value': value, 'source': 'iat-directory'} + used = True + if used: + sources.append('iat-directory') + + if iat_override: + start, length = iat_override + used = False + step = 8 if pe.is_pe32plus else 4 + for slot_rva in range(start, start + length - step + 1, step): + if slot_rva in slots: + continue + value = read_ptr(pe, slot_rva) + if value is None: + continue + slots[slot_rva] = {'value': value, 'source': 'explicit-range'} + used = True + if used: + sources.append('explicit-range') + + return slots, sources + + +def classify_slots(pe, base, slots): + """Where each slot's value sits relative to the image. + + The classes describe magnitude, not meaning: a value below the image is an unbound table entry + (an RVA) as easily as it is a handle, and the two are told apart by whether a call targets it, + not by arithmetic. Naming them as if they were interpreted was the first version of this + function and it reported every ordinary unbound import table as protected. + """ + out = {'zero': 0, 'below_image': 0, 'in_image': 0, 'above_image': 0} + end = base + (pe.size_of_image or 0) + for info in slots.values(): + value = info['value'] + if value == 0: + info['class'] = 'zero' + elif value < base: + info['class'] = 'below_image' + elif value < end: + info['class'] = 'in_image' + else: + info['class'] = 'above_image' + out[info['class']] += 1 + return out + + +def scan_sites(pe, base, value_index): + """Walk the executable sections and classify every relative call/jmp. + + Returns (pairs, unpaired, refused_candidates, counts). A pair is a site whose target equals a + slot's value; anything else that leaves the image is reported as unpaired rather than repaired. + """ + pairs = [] + unpaired = [] + counts = {'bytes': 0, 'e8': 0, 'e9': 0, 'examined': 0, 'interior': 0} + sections = [] + + for sec in pe.sections: + if not is_executable(sec): + continue + start = sec.raw_pointer or 0 + size = sec.raw_size or 0 + if not start or size < SITE_REL_LEN: + continue + data = pe.d[start:start + size] + sections.append({'name': sec.name, 'rva': sec.virtual_address, 'raw_size': size}) + counts['bytes'] += len(data) + + for off in range(0, len(data) - SITE_REL_LEN + 1): + op = data[off] + if op != CALL and op != JMP: + continue + counts['e8' if op == CALL else 'e9'] += 1 + rva = sec.virtual_address + off + site_va = base + rva + target_va = site_va + SITE_REL_LEN + s32(data, off + 1) + counts['examined'] += 1 + + slot_rvas = value_index.get(target_va) + if slot_rvas: + pairs.append({ + 'site_rva': rva, + 'kind': 'call' if op == CALL else 'jmp', + 'target_va': target_va, + 'slot_rvas': sorted(slot_rvas), + }) + continue + + target_rva = target_va - base + if pe.section_of_rva(target_rva) is None: + unpaired.append({'site_rva': rva, 'target_va': target_va}) + else: + counts['interior'] += 1 + + return pairs, unpaired, sections, counts + + +def plan_windows(pe, pairs): + """Decide the six-byte window for each pair, or refuse it. + + The filler byte is the whole difficulty: 'NOP after' means the window is the site, 'NOP before' + means it starts one byte earlier, and no filler at all means the rewrite would consume the first + byte of the following instruction. + """ + planned = [] + refused = [] + claimed = [] + + for pair in pairs: + if len(pair['slot_rvas']) > 1: + refused.append({ + 'site_rva': pair['site_rva'], + 'reason': 'ambiguous-slot', + 'detail': 'target %#x is held by %d slots: %s' % ( + pair['target_va'], len(pair['slot_rvas']), + ', '.join('%#x' % r for r in pair['slot_rvas'])), + }) + continue + + site_rva = pair['site_rva'] + after = pe.rva_bytes(site_rva + SITE_REL_LEN, 1) + before = pe.rva_bytes(site_rva - 1, 1) if site_rva >= 1 else b'' + if after == bytes([NOP]): + window = site_rva + elif before == bytes([NOP]): + window = site_rva - 1 + else: + refused.append({ + 'site_rva': site_rva, + 'reason': 'no-filler', + 'detail': 'byte at %#x is %s, byte before is %s; the rewrite needs six bytes ' + 'and the sixth belongs to the next instruction' + % (site_rva + SITE_REL_LEN, _hex(after), _hex(before)), + }) + continue + + if pe.section_of_rva(window) is None or pe.section_of_rva(window + SITE_ABS_LEN - 1) is None: + refused.append({'site_rva': site_rva, 'reason': 'window-outside-section', + 'detail': 'six-byte window at %#x crosses a section boundary' % window}) + continue + + if any(not (window + SITE_ABS_LEN <= a or b <= window) for a, b in claimed): + refused.append({'site_rva': site_rva, 'reason': 'window-overlap', + 'detail': 'window at %#x overlaps another window' % window}) + continue + + claimed.append((window, window + SITE_ABS_LEN)) + planned.append({'site_rva': site_rva, 'window_rva': window, 'kind': pair['kind'], + 'target_va': pair['target_va'], 'slot_rva': pair['slot_rvas'][0]}) + + return planned, refused + + +def _hex(raw): + return ' '.join('%02x' % b for b in raw) if raw else 'absent' + + +def encode_site(pe, base, window_rva, kind, slot_rva): + """FF 15/25 + displacement. PE32 addresses the slot absolutely; PE32+ is RIP-relative.""" + slot_va = base + slot_rva + op = OP_CALL_IND if kind == 'call' else OP_JMP_IND + if pe.is_pe32plus: + disp = slot_va - (base + window_rva + SITE_ABS_LEN) + else: + disp = slot_va + raw = struct.pack(' slot %#010x\n' + % (item['site_rva'], item['window_rva'], item['kind'], item['slot_rva'])) + if len(report['pairs']) > max_sites: + out.write(' ... %d more (raise --max-sites or use --json)\n' + % (len(report['pairs']) - max_sites)) + + out.write('refused %d\n' % len(report['refused'])) + for item in report['refused'][:max_sites]: + out.write(' site %#010x %s: %s\n' + % (item['site_rva'], item['reason'], item['detail'])) + + out.write('unpaired %d\n' % len(report['unpaired'])) + for item in report['unpaired'][:max_sites]: + out.write(' site %#010x relative call leaves the image -> %#x, no slot holds it\n' + % (item['site_rva'], item['target_va'])) + if len(report['unpaired']) > max_sites: + out.write(' ... %d more\n' % (len(report['unpaired']) - max_sites)) + + if report['result'] == 'incomplete': + out.write('next each refusal is a decision for a person: a site with no filler byte is a ' + 'structural edit, and an ambiguous slot needs the trace that says which slot the ' + 'call actually used\n') + elif report['result'] == 'needs-trace': + out.write('next either the table does not hold these targets, or the base is not the ' + 'one the image ran at -- check --base first, because a wrong base produces the ' + 'same report and a different cause. If the base is right, the handle-to-API ' + 'mapping has to come from a trace of the resolver\n') + elif report['result'] == 'report-only': + out.write('next re-run with --out to write the rewrite; the list above is the site list ' + 'the trace would otherwise have to be repeated for\n') + + for failure in report.get('verification_failures', []): + out.write('VERIFY FAILED window %#010x: %s %s\n' + % (failure['window_rva'], failure['reason'], failure.get('detail', ''))) + if report.get('output'): + o = report['output'] + out.write('output %s\n sha256 %s\n checksum declared %#x computed %#x\n' + % (o['path'], o['sha256'], o['checksum_declared'], o['checksum_computed'])) + out.write('RESULT=%s\n' % report['result']) + + +def main(): + ap = argparse.ArgumentParser( + description='Pair relative calls a protector left behind with their import slots, and ' + 'restore the indirect calls they replaced.', + formatter_class=argparse.RawDescriptionHelpFormatter, + epilog='Report first, write second. A refusal is a finding, not a failure of the run:\n' + ' python iat_callsite_restore.py dumped.exe\n' + ' python iat_callsite_restore.py dumped.exe --base 0x2B0000 --out fixed.exe') + ap.add_argument('image', help='the captured or rebuilt PE image to analyse') + ap.add_argument('--out', help='write the repaired image here; without it nothing is written') + ap.add_argument('--base', type=lambda v: int(v, 0), + help='address the image was captured at (default: the header ImageBase)') + ap.add_argument('--iat', type=parse_range, metavar='RVA,SIZE', + help='explicit import-table range, for images whose import directory is gone') + ap.add_argument('--max-sites', type=int, default=DEFAULT_MAX_SITES, + help='site rows printed per list (default %d)' % DEFAULT_MAX_SITES) + ap.add_argument('--json', action='store_true', help='emit the report as JSON') + ap.add_argument('--full-out', metavar='PATH', + help='write the complete report as JSON to PATH and report the path') + args = ap.parse_args() + + if not os.path.isfile(args.image): + sys.stderr.write('error: %s is not a file\n' % args.image) + return 2 + if args.out and os.path.abspath(args.out) == os.path.abspath(args.image): + sys.stderr.write('error: --out must differ from the input; the input is never written\n') + return 2 + if args.max_sites < 0: + sys.stderr.write('error: --max-sites must not be negative\n') + return 2 + + try: + with open(args.image, 'rb') as fh: + blob = fh.read() + pe = PE(blob, args.image) + except (ParseError, OSError) as exc: + sys.stderr.write('error: %s: %s\n' % (args.image, exc)) + return 3 + + report = build_report(args, blob, pe) + wrote = False + + if args.out: + out_blob = bytearray(blob) + base = report['input']['base_used'] + for item in report['pairs']: + raw = encode_site(pe, base, item['window_rva'], item['kind'], item['slot_rva']) + off = pe.rva_to_offset(item['window_rva']) + out_blob[off:off + SITE_ABS_LEN] = raw + with open(args.out, 'wb') as fh: + fh.write(out_blob) + pe_out = PE(bytes(out_blob), args.out) + failures = verify_written(bytes(out_blob), pe_out, base, report['pairs']) + if failures: + report['verification_failures'] = failures + report['output'] = { + 'path': args.out, + 'sha256': sha256(bytes(out_blob)), + 'checksum_declared': pe_out.checksum or 0, + 'checksum_computed': pe_out.computed_checksum(), + 'verification': 'ok' if not failures else 'failed', + } + wrote = True + + report['result'] = decide_result(report, wrote) + + if args.full_out: + with open(args.full_out, 'w', encoding='utf-8') as fh: + json.dump(report, fh, indent=2, sort_keys=True) + fh.write('\n') + + if args.json: + print(json.dumps(report, indent=2, sort_keys=True)) + print('RESULT=%s' % report['result']) + else: + print_report(report, args.max_sites) + if args.full_out: + sys.stdout.write('full record %s\n' % args.full_out) + + return 1 if report['result'] == 'incomplete' else 0 + + +if __name__ == '__main__': + sys.exit(main()) diff --git a/tests/README.md b/tests/README.md index b051fbe..cd4acca 100644 --- a/tests/README.md +++ b/tests/README.md @@ -26,6 +26,8 @@ and "the script was measured doing X on this host". | `cli/test_cli_coverage.py` | Coverage derived from the scripts directory rather than a hand-written list: every shipped `.py` is either exercised or declared uncovered with a reason, and every one answers `--help` with its usage block. A new script fails here until it is registered. | | `cli/test_cli_json.py` | Each script's machine-readable output: keys, values, and the conventions that are not what a reader would assume -- including `pe_triage.py`'s identity block on a generated image (and that it writes nothing) and `pe_diff.py`'s self-comparison, which is the control for every other diff. | | `cli/test_cli_registry_surface.py` | The registry and doctor subprocess contracts: `--list` / `--json` / single-capability exit codes and `RESULT=` tokens, the doctor's three blocks and its 0-versus-3 verdict, and the view flags that print one block only. | +| `unit/test_iat_callsite_restore.py` | The import-layer repair's four decisions on a generated image: the six-byte window on either side of a call, the refusal when the sixth byte is the next instruction, the refusal when two slots hold one address, the pairing rule, and the verdict token that follows from the evidence rather than from a count. | +| `cli/test_cli_iat_callsite.py` | The repair's published contract as a subprocess: a report-only run writes nothing into the working directory, `--out` leaves the input byte-identical and the written sites re-decode to indirect calls, the PE32 displacement is an absolute slot address where the PE32+ one is relative, and a refusal is exit 1 rather than a quiet success. | | `cli/test_cli_dart_snapshot.py` | The snapshot reader's CLI: the refusals and their exit codes, `--find` case-insensitivity and the exit code a miss produces, `--at` by file offset and by virtual address, and the `--json` document. | | `integration/test_pipeline_end_to_end.py` | Build, locate, patch, re-read: the patched bytes, the untouched control site, the recomputed checksum, and a control run in which nothing changes. | | `integration/test_across_builds.py` | Pattern hit counts compared between two builds, including the case where a patch removes the very byte the pattern was located with. | diff --git a/tests/cli/test_cli_iat_callsite.py b/tests/cli/test_cli_iat_callsite.py new file mode 100644 index 0000000..43ead99 --- /dev/null +++ b/tests/cli/test_cli_iat_callsite.py @@ -0,0 +1,124 @@ +"""The import-layer repair's published contract, exercised as a subprocess. + +What a caller can rely on, and what this file pins: + + * without --out, nothing is written and the run is an analysis (exit 0 or 1 depending on whether a + site was refused, never a silent file); + * with --out, the input file is byte-identical afterwards, the output is a different file, and the + six bytes written at each site decode to an indirect call through the slot the report names; + * a refusal is exit 1 and the word "incomplete" -- the failure mode this repository is organised + against is a partial repair reported as a finished one. +""" +import hashlib +import json + +import pytest + +pytestmark = pytest.mark.cli + + +def sha(path): + return hashlib.sha256(path.read_bytes()).hexdigest() + + +def document(text): + return json.JSONDecoder().raw_decode(text)[0] + + +def test_analysis_writes_nothing_and_reports_incomplete(run_script, iat_scenario, tmp_path): + scen = iat_scenario("pe64") + before = sha(scen.path) + result = run_script("iat_callsite_restore.py", scen.path, "--iat", "0x3020,0x18", cwd=tmp_path) + assert result.returncode == 1, result.stderr + assert "RESULT=incomplete" in result.stdout + assert sha(scen.path) == before + assert list(tmp_path.iterdir()) == [] + + +def test_the_rewrite_is_verified_and_leaves_the_input_alone(run_script, iat_scenario, + load_script, tmp_path): + scen = iat_scenario("pe64") + before = sha(scen.path) + out = tmp_path / "fixed.exe" + result = run_script("iat_callsite_restore.py", scen.path, "--iat", "0x3020,0x18", + "--out", out) + assert result.returncode == 1, result.stderr + assert "VERIFY FAILED" not in result.stdout + assert "RESULT=incomplete" in result.stdout + assert sha(scen.path) == before, "the input was modified" + + pe = load_script("peutil.py").PE(out.read_bytes()) + for window in (scen.sites["filler_after"], scen.sites["filler_before"] - 1): + raw = pe.rva_bytes(window, 6) + assert raw[:2] == b"\xFF\x15", raw.hex() + # The refused site is untouched: the byte after its call is still the original instruction. + assert pe.rva_bytes(scen.sites["no_filler"] + 5, 1) == b"\xC3" + + +def test_the_pe32_displacement_is_an_absolute_slot_address(run_script, iat_scenario, + load_script, tmp_path): + scen = iat_scenario("pe32") + out = tmp_path / "fixed32.exe" + result = run_script("iat_callsite_restore.py", scen.path, "--iat", "0x3020,0x18", "--out", out) + assert result.returncode == 1, result.stderr + pe = load_script("peutil.py").PE(out.read_bytes()) + raw = pe.rva_bytes(scen.sites["filler_after"], 6) + assert raw[:2] == b"\xFF\x15" + assert int.from_bytes(raw[2:], "little") == scen.base + scen.slot_rva + + +def test_a_clean_image_exits_zero(run_script, fixtures): + result = run_script("iat_callsite_restore.py", fixtures.pe64.path) + assert result.returncode == 0, result.stderr + assert "RESULT=no-pairs" in result.stdout + + +def test_out_equal_to_the_input_is_a_usage_error(run_script, iat_scenario): + scen = iat_scenario("pe64") + result = run_script("iat_callsite_restore.py", scen.path, "--out", scen.path) + assert result.returncode == 2 + assert "never written" in result.stderr + + +def test_a_non_pe_is_exit_three(run_script, tmp_path): + junk = tmp_path / "notape.bin" + junk.write_bytes(b"MZ" + b"\x00" * 200) + result = run_script("iat_callsite_restore.py", junk) + assert result.returncode == 3 + assert "error:" in result.stderr + + +def test_json_carries_the_document_then_the_verdict_line(run_script, iat_scenario): + scen = iat_scenario("pe64") + result = run_script("iat_callsite_restore.py", scen.path, "--iat", "0x3020,0x18", "--json") + payload = document(result.stdout) + lines = [ln for ln in result.stdout.splitlines() if ln.strip()] + assert lines[-1].startswith("RESULT=") + assert payload["result"] == lines[-1].split("=", 1)[1] + assert payload["input"]["bitness"] == scen.bitness + assert payload["input"]["base_used"] == scen.base + assert sorted(payload["iat"]["classes"]) == ["above_image", "below_image", "in_image", "zero"] + assert [p["site_rva"] for p in payload["pairs"]] == [scen.sites["filler_after"], + scen.sites["filler_before"]] + assert {r["reason"] for r in payload["refused"]} == {"no-filler", "ambiguous-slot"} + assert "output" not in payload, "a report-only run must not claim an output file" + + +def test_full_out_writes_the_same_document_and_says_where(run_script, iat_scenario, tmp_path): + scen = iat_scenario("pe64") + record = tmp_path / "report.json" + result = run_script("iat_callsite_restore.py", scen.path, "--iat", "0x3020,0x18", + "--full-out", record) + assert result.returncode == 1 + assert "full record" in result.stdout + saved = json.loads(record.read_text(encoding="utf-8")) + assert saved["tool"] == "iat_callsite_restore.py" + assert saved["result"] == "incomplete" + + +def test_max_sites_bounds_the_printed_lists(run_script, iat_scenario): + scen = iat_scenario("pe64") + result = run_script("iat_callsite_restore.py", scen.path, "--iat", "0x3020,0x18", + "--max-sites", "0") + assert result.returncode == 1 + assert "more" in result.stdout diff --git a/tests/conftest.py b/tests/conftest.py index 6af9cc7..0a49e32 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -55,6 +55,10 @@ SCRIPT_FILES = ( # script is neither in this tuple nor declared uncovered with a reason. "pe_triage.py", "pe_diff.py", + # The import-layer repair: it pairs a rewritten relative call with the slot that holds its + # target, and refuses the three cases a byte-swapping tool would corrupt. Its scenario is + # generated below, because the decisions it makes are about byte layout, not about any sample. + "iat_callsite_restore.py", ) # A hang detector, not a performance assertion. Nothing in this suite is meant to be slow. @@ -504,6 +508,89 @@ def fixtures(build_fixture): return FixtureSet({name: build_fixture(name) for name in FIXTURE_RECIPES}) +class IatScenario(object): + """A generated image carrying the six-byte decisions the repair script must make. + + Five sites: one with the filler byte after the call, one with it before, one with no filler at + all, one whose target two slots both hold, and the slots themselves in a range the import + directory does not describe (which is the normal case for a captured image). + """ + + __slots__ = ("path", "pe_path", "base", "slot_rva", "dup_slot_rvas", "sites", "bitness") + + def __init__(self, **kw): + for key in self.__slots__: + setattr(self, key, kw.get(key)) + + def __repr__(self): + return "IatScenario(%s)" % self.path + + +@pytest.fixture(scope="session") +def iat_scenario(tmp_path_factory, build_fixture, load_script): + """Build the scenario by editing a fixture the rest of the suite already uses. + + Editing rather than synthesising is deliberate: the script's job is arithmetic over a real + header and a real section table, and a hand-built header would let a wrong RVA mapping agree + with itself. The fixture is copied first, so the shared build stays pristine for every other + test in the session. + """ + peutil = load_script("peutil.py") + root = tmp_path_factory.mktemp("iat-scenario") + cache = {} + + def _build(name="pe64"): + if name in cache: + return cache[name] + built = build_fixture(name) + pe = peutil.PE(built.data) + base = pe.image_base + psize = 8 if pe.is_pe32plus else 4 + data = bytearray(built.data) + + def put(rva, raw): + off = pe.rva_to_offset(rva) + assert off is not None, "scenario RVA %#x has no file offset" % rva + data[off:off + len(raw)] = raw + + def call(rva, target_va): + return b"\xE8" + struct.pack("