Merge PR #6 "Hardware pass (stage 1)" (dabao1955/feat/ete)

Merges the hardware-layer pass onto the dexsql commit: arm64 hardware
watchpoints, ETE differential tracing, kernel-side event injection, SPE/BRBE
sampling and crash forensics, address symbolization and interpreter-trace
de-slicing. 33 files, 6 new scripts, 6 new test modules, 4 new reference
documents.

The only conflict was in evidence/capability-matrix.json: both sides appended
to the same array tail. Resolved by keeping both -- the PR's seven entries
(44 rows total now) -- and re-validated as JSON before staging.

Verified on the merged tree, not on the PR branch alone:

  * check_repo.py, check_refs.py, check_commands.py, check_routing.py and
    check_budget.py all pass with zero problems;
  * pytest: 766 passed, 172 skipped, 25 xfailed -- the 141 tests this PR adds
    all pass alongside the 625 that were already there;
  * the PR's host-side claims were reproduced by hand, not taken on trust:
    `hw_watch.py plan --addr 0x1004 --len 4 --access w` returns
    `cr: 7701` (WCR 0x1E15) exactly as the record says, both refusals exit 2
    with the documented reason strings, and the kernel scaffold's
    generate/verify round trip returns RESULT=ok.

One documentation defect was found and fixed in the merge: EXTENSION-kernel-instrument.md
recorded `verified 3 kernel-side file(s)` while the same page's own generate
output lists four (kpm/src/kwatch.c, kpm/Makefile, lkm/kwatch.c, lkm/Makefile)
and the tool reports 4. The page now says 4.

Everything the PR ships that needs a device or a kernel toolchain stays
labelled unverified in the files themselves, which is the property that made
it mergeable.
This commit is contained in:
inliver233
2026-10-16 02:05:59 +08:00
33 changed files with 5358 additions and 6 deletions
@@ -0,0 +1,75 @@
# Extension verification — ETE differential tracing (gates, reducer, differ)
Covers the hardware-layer pass that added `scripts/ete_capture.py`, `scripts/trace_diff.py` and
`references/ete-diff-tracing.md`. Strength labels as this directory's README defines them.
Reference environment: Linux container host, Python 3.13, adb present with no device. No
coresight-capable hardware exists in this environment at all; the reference device of the
earlier passes (kernel 4.14.186) has the ETE gate closed permanently (`observed` there).
**Headline: the gate table, the recipe and the capture wrapper are host-measured; the
differential reducer is host-measured on synthetic flattened-dispatcher fixtures. No ETE trace
was captured or decoded anywhere in this pass — the producer side is `unverified` and the
`--control` run exists precisely so the next device says so in one command.**
---
## 1. The differ, measured on a synthetic OLLVM shape — observed
Two fixture logs built to the flattened-dispatcher rhythm `D R1 D R2 …` — run A taking
`R5/R7` (wrong password), run B taking `R3/R4` (right password), one log in raw
`stalker_trace.js` grammar, one wrapped by `run_probe.py` (`[ts] [ts] TRACE BLK …`) to prove
the parser survives both:
```
$ python skills/apk-reverse/scripts/trace_diff.py diff --a a.log --b b.log
run A: 8 events, 5 distinct blocks
run B: 8 events, 5 distinct blocks
shared block executions: 6 divergent regions: 2
first divergence: after run #5 (A) / #5 (B), 5/5 events in, last common block liba.so+0x1000
region 1 (replace)
A: liba.so+0x6000 B: liba.so+0x4000
blocks unique to A ...: 0x6000, 0x8000 blocks unique to B ...: 0x4000, 0x5000
```
What the output pins, and why each number is the one the method needs:
- **the identical prefix** (`D R1 D R2`, 5 events) is the shared machinery — exactly where
parsing ends and the input-dependent path begins;
- **first divergence** names the dispatcher entry whose state constant differed;
- **unique-to-each-side** lists are the input-dependent block set, which is the deliverable
the reference file's §4 interprets per obfuscation.
`reduce` on one side prints the histogram with the dispatcher as the wide-margin top
(4 hits vs 1 for every real block) — the same shape `stalker_report.py` documents for Stalker
logs, reached from a different producer.
## 2. Gate and capture plumbing — observed (to the no-device boundary)
`gates` prints the five-gate table; `gates --device` with adb present but no device degrades
to five `OPEN` rows and `RESULT=partial` rather than a traceback — measured. `capture --pid 1`
without a device exits through the pull failure with `RESULT=blocked` and exit code 3
(no-device capability refusal, per the CLI contract) — measured. `recipe` prints the
Termux and cross build paths for perf-with-CORESIGHT — `inferred` from the kernel's
tools/perf CoreSight documentation and the OpenCSD project, not from a build performed here.
## 3. What this pass deliberately did not do
- No perf-with-CORESIGHT was built (host or device): the recipe is a documented path, and a
reader who builds it owes the control record (`capture --control`) before any real capture.
- No decoded cs_etm stream was fed to `trace_diff.py`; the `plain` parser accepts the
address-bearing lines `perf script` emits, but that exact pairing is `unverified`.
- No claim about ETE's runtime overhead or its visibility to targets was measured. The
reference states the mechanism (no code modification, no ptrace) and refuses to go further.
## 4. Facts this pass pinned for the next one
- The differ is **producer-agnostic by design**: the same command answers Stalker logs today
and cs_etm decode output tomorrow. That is why the diff subcommand ships now, not after the
first ETE capture — it is useful the day a Stalker pair exists.
- The five gates each fail **silently** downstream; the reference's table says which silence
each gate produces. The one-instruction control record is the only gate that exercises the
whole pipe, which is why `capture` refuses to continue when it fails.
- Run design is part of the tool contract, not user care: `trace_diff.py` refuses empty logs
with a message that points at the zero-event diagnostics, because an empty trace is a
capture failure wearing an analysis result's clothes.
@@ -0,0 +1,69 @@
# Extension verification — hardware watchpoints (ptrace route, helper, encoders)
Covers the hardware-layer pass that added `scripts/hw_watch.py` and
`references/hardware-watchpoints.md`. Strength labels as this directory's README defines them.
Reference environment: Linux container host, Python 3.13, cross `gcc 16.2.0` targeting
`aarch64-linux-gnu`. No Android device was attached during this pass; `adb` exists on PATH but
reports no devices (`adb: no devices/emulators found`, observed).
**Headline: the encoders, the plan/emit/decode paths and the helper's build are measured on the
host — including a clean static aarch64 compile of the emitted C. No watchpoint was armed on any
device by this pass; the device-side behaviour is `unverified` by this record.**
---
## 1. Encoder acceptance and refusal — observed
```
$ python skills/apk-reverse/scripts/hw_watch.py plan --addr 0x1004 --len 4 --access w --json
{"status": "ok", "specs": [{"kind": "watch", "addr": 4100, "len": 4, "access": "w", "vr": 4096,
"cr": 7701, "cr_fields": "BAS=0xf0 LSC=store PAC=EL0 E=1"}]}
```
WCR `0x1E15` = BAS `0xf0` (bytes 4–7 of the aligned window) | store | EL0 | enabled — the bit
fields decompose back to the inputs exactly. Refusals measured with exit 2 and a reason:
- `--addr 0x7b2000123456 --len 8` (addr&7=6, 6+8>8): *"crosses its aligned 8-byte window"*
- `plan --break 0x1002`: *"breakpoint PC 0x1002 is not 4-byte aligned"*
The canonical values are pinned by unit test (`tests/unit/test_hw_watch.py`): 8-byte rw watch
`WCR=0x1FFD`, address-match EL0 breakpoint `BCR=0x3F05`.
## 2. The emitted helper compiles as a static aarch64 ELF — observed
```
$ python skills/apk-reverse/scripts/hw_watch.py emit --out work/hwwatch
$ gcc -static -O2 -o hwwatch work/hwwatch/hwwatch_helper.c
$ file hwwatch
hwwatch: ELF 64-bit LSB executable, ARM aarch64, version 1 (GNU/Linux), statically linked
```
Zero errors, zero warnings after one fix (a `TRAP_HWBKPT` redefinition against glibc's
`signal.h` needed an `#ifndef` guard; bionic defines the same macro). The compiler here is a
cross glibc toolchain, not the NDK — the binary was not run, and the runtime behaviour of the
ptrace loop (seize/interrupt per tid, `PTRACE_O_TRACECLONE` arming, `TRAP_HWBKPT` siginfo
handling, the disable-step-rearm breakpoint dance, group-stop `PTRACE_LISTEN`) remains
**`unverified`** until a device runs the control procedure in the reference's §3.
## 3. What this pass deliberately did not do
- No device was attached: no `READY watch_slots=N` probe, no slot-capacity read, no control
run, no hit. Every device-behaviour claim in `hardware-watchpoints.md` carries `unverified`.
- `decode` was exercised on synthetic `HIT` lines only (unit test); a real `/proc/<pid>/maps`
pull was not available to feed it.
- `run` (build+push+stream) was exercised to its argument-validation and no-adb exits only;
its `exit 3` on missing adb was measured before a device was ever contacted.
## 4. Facts this pass pinned for the next one
- `slot_capacity` reads `dbg_info & 0xff` from the `NT_ARM_HW_WATCH`/`NT_ARM_HW_BREAK`
regsets — the ABI gdb uses for the same purpose; on a masked kernel this returns 0 and the
helper prints `READY ... watch_slots=0`, which the reference file treats as a **closed
gate**, not a tuning problem.
- The helper's output grammar (`READY`/`ARM`/`HIT`/`EXIT`/`DONE`) is load-bearing: `decode`
and the reference's control procedure both parse it. Changing the helper means regenerating
both ends.
- The honesty ledger (ptrace route leaves `TracerPid` visible) is stated in three places on
purpose — script header, reference §1, evidence summary — because the tempting wrong claim
("invisible") is exactly what a reader under pressure wants to believe.
@@ -0,0 +1,74 @@
# Extension verification — kernel-side instrumentation scaffold (templates, gates, verify)
Covers the hardware-layer pass that added `scripts/kernel_instrument_scaffold.py` and
`references/kernel-event-injection.md`. Strength labels as this directory's README defines
them. This pass follows `EXTENSION-kernel-weapons.md`'s shape deliberately: a generator whose
host behaviour is measured, kernel-side templates that have never been compiled or loaded
anywhere, and the gate table that says so.
Reference environment: Linux container host, Python 3.13, no `aarch64-none-elf-gcc`
(`gates` reports NOT FOUND — observed), no APatch/KernelPatch boot image, no device.
**Headline: `gates`/`generate`/`verify` are measured on the host. Every generated kernel-side
file carries a `NOT COMPILED AND NOT LOADED` banner that `verify` enforces. The
register-rewrite action is labelled experimental in the template itself — it was never run
anywhere, by anyone in this repository.**
---
## 1. Generate + verify round trip — observed
```
$ python skills/apk-reverse/scripts/kernel_instrument_scaffold.py generate \
--out work/kwatch --pid 12345 --watch w:0x7b2000123450:8:w --break b:0x7b20001234c0 --set-reg x0=0x1
wrote work/kwatch
kpm/src/kwatch.c kpm/Makefile lkm/kwatch.c lkm/Makefile
module.prop README.md post-fs-data.sh config/instrument.json
RESULT=ok
$ python skills/apk-reverse/scripts/kernel_instrument_scaffold.py verify --dir work/kwatch
verified 4 kernel-side file(s), 0 problem(s)
RESULT=ok
```
Spec validation reuses the hardware rules and refuses at generation, before any file is
written: a watch crossing its 8-byte window and a misaligned breakpoint PC both exit 2 with
the reason (`observed`, same refusal strings as `hw_watch.py plan` — one rule, one copy in
each tool, both unit-tested).
The generated KPM template embeds: `kfunc_def(...)` declarations **without** `extern` (the
loader-rejection lesson recorded in `EXTENSION-kernel-weapons.md`, restated in the template's
own header), a callback whose first statements are the cheap filter, and the watch config as
generated constants with *"regenerate rather than hand-edit"* in the config JSON.
## 2. The honesty devices this pass ships
- **The primitive map is a capability table, not a wishlist.** Each cell names what the route
can do, and the `no` cells (eBPF cannot create perf breakpoint events; a userspace module
cannot arm anything) are the rows that stop a reader from spending a day on a route that
cannot express the need.
- **The experimental action is fenced three times**: template comment, script header, and the
reference's stop table ("the first run is a measurement"). Rewriting `pt_regs` in a perf
overflow callback is mechanically what a ptrace stop does, and that is exactly as far as
the claim goes.
- **The userspace carrier cannot arm watchpoints and says so in its own README** — the same
structural correction `EXTENSION-kernel-weapons.md` made for syscall masking, applied
before anyone could misread this module the same way.
## 3. What this pass deliberately did not do
- Nothing was compiled: no `aarch64-none-elf-gcc` exists here, no KernelPatch image exists on
any reference device, and the LKM route has no matching kernel source. All three kernel
files are `unverified` templates and are labelled so in the files themselves.
- No claim about whether `register_hw_breakpoint` can be resolved from a KPM on any specific
KernelPatch revision is made; the template says to verify symbol names against the
revision's `kpmodule.h`, which is the same instruction the syscall-mask template gives.
## 4. Facts this pass pinned for the next one
- The kernel's **perf breakpoint layer owning the debug registers** is a gate in its own
right: if `CONFIG_HW_PERF_EVENTS` is out, both templates close regardless of root scheme.
Probe: `ls /sys/bus/event_source/devices/breakpoint` on the device.
- `verify` exists so the banner survives edits: someone will adapt a template on a real
device, strip the banner, and the next reader must be able to tell an adapted-unverified
file from a measured one. A banner replaced by a real measurement is the intended
outcome, and `kernel-event-injection.md`'s checklist says so.
@@ -0,0 +1,100 @@
# Extension verification — sampling, crash forensics, symbolization, de-virtualization
Covers the second hardware-layer pass: `references/hardware-sampling-and-crash.md`,
`scripts/trace_addr2sym.py`, `scripts/vm_handler_slice.py`, and the source-reading
corrections to the flattening recipes. Strength labels as this directory's README defines
them.
Reference environment: Linux container host, Python 3.13, cross `gcc 16.2.0` targeting
`aarch64-linux-gnu` (the toolchain that makes the symbolizer's "real ELF" half testable).
Network access was used once, read-only, to fetch the two public obfuscator implementations
the calibration cites (below). No Android device, no coresight/SPE/BRBE hardware.
**Headline: the address→symbol loop is measured against a real cross-compiled library, the
de-virtualization slicer is measured on synthetic interpreter traces, and the SPE/BRBE
producer side remains documented-and-gated. The flattening-recipe corrections are read from
public source, and that source is named.**
---
## 1. `trace_addr2sym.py` against a real library — observed
```
$ gcc -shared -fPIC -O2 -o libtlib.so tlib.c # cross gcc -> aarch64 ELF64
$ readelf --syms -W libtlib.so
... 00000000000005e0 132 FUNC GLOBAL DEFAULT 9 encrypt_block
... 0000000000000664 4 FUNC GLOBAL DEFAULT 9 decrypt_block
$ python skills/apk-reverse/scripts/trace_addr2sym.py --elf libtlib.so --offs 0x610
0x610 -> encrypt_block+0x30 [vaddr 0x610 file 0x610 .text+0x130]
```
The resolution agrees with `readelf` exactly (0x610 − 0x5e0 = 0x30, inside the sized symbol);
a point outside any function falls to the nearest-below symbol with its section shown
(`0x700 -> _fini+0x98 [.eh_frame+0x40]`), which is GNU-addr2line-shaped behaviour and is
pinned by unit test together with the PT_LOAD `vaddr≠offset` conversion on a hand-built
minimal ELF. That minimal-ELF fixture also covers the stripped-library fallback
(section+offset, no symbols).
## 2. `vm_handler_slice.py` on the calibrated shape — observed
A synthetic interpreter trace in the shape the public IR virtualizer emits
(dispatcher D, handler blocks, one const-materialization segment, an unclosed tail):
```
$ python skills/apk-reverse/scripts/vm_handler_slice.py --log vm.log --opcode-map map.json
dispatcher: libvm.so+0x1000 (41.7% of 24 blocks, 10 hits)
segments: 10 total, 7 unique, avg len 2.00 [last segment UNCLOSED: exit path or truncated window]
opcode trajectory (first 20): OP_LOAD OP_LOAD OP_STORE OP_BR OP_LOAD OP_LOAD OP_BR OP_CALL OP_LOAD OP_RET
```
What the output pins, and why each part is the part a de-virtualization needs: the
**dispatcher share** (with the diagnostic that separates a real dispatch loop from a spin
loop), the **segment table** (one row per handler chain, first block = handler entry =
the `--opcode-map` key), the **trajectory** (the guest program, one signature per executed
guest opcode), the **OP_CALL materialization segment** appearing as its own signature (the
virtualizer's const dispatch — the second observation point the calibration names), and the
**UNCLOSED tail** flagged rather than silently dropped. The auto-detection rule is
top-of-histogram **plus predecessor diversity** (≥3 distinct predecessors), not a naive
count margin: the first implementation used a 4x margin and refused this very fixture —
in a stack-shaped VM the hottest handler executes on every guest instruction, exactly as
often as the dispatcher. That refusal-to-accept is the load-bearing part of the auto rule.
## 3. The flattening-recipe corrections — observed source, inferred runtime
Fetched read-only from the public implementations the user supplied
(github.com/dpkg123/ollvm-aosp-toolchain: branch `ollvm-r614150-clang23-rebase+omp`
[Hikari family] and `ollvm-r614150-clang23+omp` [Pluto family]; commits 63fec41f and
9b7cf75e for the IR virtualizer; 2026-10):
- **Hikari `Flattening.cpp`**: `switchVar` is an entry-block **alloca** written through a
`switchVarAddr` double pointer (indirect store per transition); case values are
`scramble32`-permuted; EH-bearing functions are skipped.
- **Pluto `Flattening.cpp`**: two allocas (`switchVar`, `switchXor`) with a **rolling XOR**
(`enc^=delta; key^=delta; cond=enc^key` per dispatch); case values fully random; the true
case exists only in the `switchCond` register for one instruction; EH/`callbr`/`indirectbr`
functions skipped.
- **Hikari `AntiDebugging.cpp`**: on Darwin/AArch64 it injects **inline-assembly raw `svc`**
with a randomized register-setup order and a **randomized svc immediate**; elsewhere it
links precompiled anti-debug IR. Consequence recorded in the reference: byte signatures
cannot survive the randomization; semantic site-location (`svc_scan.py`) and hardware
observation do not care.
- **The IR virtualizer**: `vm_exec(code, size, vm_args, num)` stubs, boxed-argument array,
11 one-byte opcodes + 19 arithmetic sub-ops, const/call materialization dispatch,
switch-to-if lowering.
These corrected `hardware-watchpoints.md`'s state-variable recipe (the old text assumed a
global and a readable value — wrong for both families as shipped) and added the
family-difference table to `ete-diff-tracing.md` §5. The fetches live under
`tools/_work/refs/` (git-ignored); **runtime behaviour of these passes is `inferred` from
source, never executed here.**
## 4. What this pass deliberately did not do
- No SPE capture, no BRBE record read, no `perf -j any` on any device: those routes ship as
documented-and-gated, and the reference's §5 failure modes say exactly how each gate
fails silently.
- No xvmp-hardened binary was compiled or traced: the slicer's calibration is source-read,
and its first run against a real sample is a measurement, not a confirmation.
- The `--opcode-map` a real target needs is a manual labelling task over the segment table;
this pass generated the fixture's map by hand, which is exactly the workflow the script
documents.
+13
View File
@@ -89,6 +89,19 @@ written down.
| `EXTENSION-detection-pipeline.md` | Naming the check that fires: three arms on a public MASTG target (control alive, probe arm dead in ≈300 ms via `strstr("frida")`, with the `TracerPid=0` versus 4-frida-mappings asymmetry), `svc_scan.py`'s two decoders agreeing on an identical 214-site set, the layered Java-to-svc descent, and the ptrace-free dump boundary on targets whose dex is deflated inside the APK |
| `EXTENSION-dexsql.md` | The SQL query layer adopted as a **third indexer** beside rasc/droidasc/ddc: identical class-definition sets on four archives (zero differences either way), the string xref agreement and the `=` versus `LIKE` match-semantics trap, `decode` against `ddc disasm` line for line, and the persistent cache measured as **not** a speed switch |
## Hardware-layer pass record
The hardware pass (hardware watchpoints, ETE differential tracing, kernel-side event injection)
follows the same shape: what runs on a host is measured, and everything that needs a device or a
kernel toolchain is an explicitly-labelled template whose gate says where it closes.
| File | Covers |
|---|---|
| `EXTENSION-hw-watchpoints.md` | The ptrace hardware-watchpoint route: encoders accepted and refused exactly as the alignment rules demand, and the emitted on-device helper **compiled clean as a static aarch64 ELF** — while no watchpoint was armed on any device, and the ledger that says the ptrace route leaves `TracerPid` visible |
| `EXTENSION-ete-diff.md` | The differential half measured on synthetic flattened-dispatcher fixtures (first divergence, unique-per-side sets), the gate/capture plumbing measured to the no-device boundary, and what was deliberately not done: no perf-with-CORESIGHT build, no cs_etm stream, no overhead claim |
| `EXTENSION-kernel-instrument.md` | The kernel-side scaffold's generate/verify round trip, the spec-refusal rules shared with `hw_watch.py`, the capability map whose `no` cells do the gatekeeping, and the three-time-fenced experimental register-rewrite action |
| `EXTENSION-sampling-crash.md` | The second hardware pass: `trace_addr2sym.py` verified against a real cross-compiled `.so` (readelf-cross-checked), the interpreter-trace slicer measured on the calibrated xvmp shape (including the 4x-margin bug it found in itself), and the flattening-recipe corrections read from the two public obfuscator implementations the user supplied |
## Adding a run
Record the tools, versions and exact commands, and label each claim. Keep target-specific facts out: