mirror of
https://codeberg.org/reverse-skill-lab/apk-reverse
synced 2026-11-15 20:54:31 +08:00
root
ab22b225ff
Add the layer under the software tooling, in the repository's shape: what runs on a host is measured, and every device-bound or kernel-bound piece is a gated, labelled template. - scripts/hw_watch.py: arm64 hardware watchpoints/execution breakpoints. WCR/ BCR encoders with the 8-byte-window and alignment rules (unit-tested to the canonical 0x1FFD/0x3F05), a self-contained on-device ptrace helper that compiles clean as a static aarch64 ELF (observed: cross gcc 16.2), adb orchestration, and hit decoding to module+offset. The stealth ledger is stated three times on purpose: the ptrace route leaves TracerPid visible. - scripts/ete_capture.py: CoreSight ETE capture with the five gates (kernel era, coresight sysfs, cs_etm PMU, perf-with-CORESIGHT, permission), the perf/OpenCSD build recipe, and the per-thread record wrapper whose --control one-instruction run must pass before any real capture. - scripts/trace_diff.py: differential reduction of two execution traces into first-divergence / divergent regions / blocks-unique-to-each-side. Producer-agnostic: Stalker BLK logs (raw or run_probe-wrapped) and decoded cs_etm address streams alike, host-tested on synthetic flattened-dispatcher fixtures. - scripts/kernel_instrument_scaffold.py: KPM/LKM watchpoint + register-rewrite templates plus an honest userspace carrier; every kernel file carries a NOT COMPILED AND NOT LOADED banner that verify enforces, and the register-rewrite action is fenced as experimental in the template itself. - references/hardware-watchpoints.md, references/ete-diff-tracing.md, references/kernel-event-injection.md: the mechanism, the gates, the traps, and the composition recipes (watch the key buffer; diff two runs; flip the decision at the trap), with hand-offs instead of duplication. - Registration: 3 symptom rows (SKILL.md + routing mirror), 3 reference rows, 4 script rows, coverage/route-inventory/evidence rows, capability-matrix entries, known-limitations entry, eval case 4, README trees and coverage paragraphs in both languages, and three EXTENSION evidence records stating exactly what was measured on this host (encoders, compile, fixtures, generate/verify) and what was not (no watchpoint armed, no cs_etm stream, nothing kernel-side compiled or loaded anywhere). Hardware pass 2: SPE/BRBE crash forensics, address symbolization, VM de-virtualization Calibrated against the two public obfuscator implementations the user supplied (dpkg123/ollvm-aosp-toolchain: Hikari-family and Pluto-family flattening, the xvmp IR virtualizer at 63fec41f/9b7cf75e; fetched read-only into tools/_work/refs, git-ignored). Source facts observed, runtime inferred. - scripts/trace_addr2sym.py: runtime addresses -> symbol + static offset, zero-dependency ELF64 (PT_LOAD vaddr<->file-off conversion, .dynsym/.symtab, stripped section+off fallback). Host-verified against a real cross-compiled aarch64 .so with readelf as the independent cross-check, plus a hand-built minimal ELF covering vaddr!=offset; accepts hw_watch hits, trace mod+off keys, or raw address lists. Fixes on the way: --base int coercion, and nearest-below symbol semantics for PCs in inter-symbol gaps. - scripts/vm_handler_slice.py: slice an interpreter trace into handler segments (dispatcher detection = histogram top + predecessor diversity; the first 4x-margin rule refused the calibrated fixture itself and was replaced), signature clustering, open-tail handling, and the guest-opcode trajectory via an optional handler->opcode map. Calibrated against xvmp's documented shape: vm_exec(code,size,vm_args,num) stubs, boxed arguments observable before interpretation, 11 opcodes + 19 arithmetic sub-ops, const-materialization dispatch as a distinct segment. - references/hardware-sampling-and-crash.md: SPE (data flow without a known address; PC + data address per sample), BRBE (branch records when a crash stack cannot be trusted), the crash loop (tombstone -> branch records -> symbolize), and what each adds to the OLLVM/VMP routes. All documented-and-gated: no SPE sample or BRBE record was captured here. - Flattening-recipe corrections from source: Hikari keeps a frame-resident switchVar written through an indirect pointer with scramble32-permuted cases; Pluto stores switchVar/switchXor rolling-XOR so memory only ever holds noise -- hardware-watchpoints.md's state-variable recipe now distinguishes the families, and ete-diff-tracing.md carries the family table plus the xvmp shape. AntiDebugging injects raw svc inline asm with randomized fields, so byte signatures cannot survive it; semantic site-location can. - Registration: 3 symptom rows (untrustworthy crash stack; runtime addresses needing symbols; interpreter loop reading as noise), reference/script rows, coverage/evidence/capability-matrix entries, README trees in both languages, and EXTENSION-sampling-crash.md recording exactly what was measured (real .so symbolization, slicer fixtures, the slicer's self-found dispatcher-rule bug) and what was not (no SPE/BRBE capture, no hardened binary ever traced). Fix all PR #1 review findings by severity; un-entrench the tests that pinned them Major (device-breaking / wrong results): - hw_watch: match the arm64 user_hwdebug_state slot ABI (address register first, control second -- the swap armed nothing correctly) - hw_watch: every post-seize error path now clears slots and detaches; PTRACE_O_EXITKILL turned an analysis failure into a SIGKILL of the target - hw_watch: forward genuine target SIGTRAPs (raise/tgkill/int3); only ptrace event stops are consumed -- the reference's signal-preservation promise - hw_watch: run checks push/build/chmod and the helper's exit status; a dead capture is RESULT=blocked, never ok-no-hits - hw_watch: decode offsets from the module base, not each mapping's start - ete_capture: --control now requires the control record to DECODE (perf script event lines), the documented gate, not just a zero exit - ete_capture: -o before the -- workload separator in --cmd mode; all values shell-quoted into su -c; stale remote perf.data removed first; perf exit status checked (timeout's 124 is the normal window end) - kernel scaffold: --set-reg restricted to x0..x30; the rewrite is gated by its own breakpoint event so a watchpoint hit cannot run it; LKM pins the task (get/put_task_struct) across registration and calls the four-argument register_user_hw_breakpoint; --break rows initialize .addr/.len/.bp_type (there is no .pc member); --id validated against the module-id grammar - trace_addr2sym: exact module matching (path or basename, never substring); load bias aligned against the PT_LOAD (bias = start - (foff - p_offset) - p_vaddr); PCs outside the ELF span are refused instead of landing on the last symbol; --json stdout is a pure document (RESULT= in text mode only) - trace_diff: block keys need 0x or >=8 hex digits (PIDs/counts no longer parse); vm_handler_slice: dispatcher needs predecessor diversity, no frequency-margin fallback; both JSON outputs pure (check_commands' rule) Minor/nitpick: - capability-matrix + evidence-summary: trace reduction status ok->partial (no real-target producer ran it; the file's own definition of ok) - hardware-sampling-and-crash.md and the script examples carry --elf - kernel-event-injection.md no longer claims the callback's first statement is a filter (it prints unfiltered; the filter is a required change) - Elf64 caches symbol addresses for per-row bisect - docstring coverage of the touched surface: 16.67% -> 100% Tests: every json.loads workaround stripping RESULT= is gone; new regression tests cover the slot ABI, release-on-error, SIGTRAP forwarding, non-first mapping offsets, -o placement and su -c quoting, the control decode gate, x31/--id refusals, the guarded rewrite, outside-ELF PCs and bias math, short-token refusal and the dispatcher margin rule. The minimal-ELF fixture now packs a full 8-field Elf64_Phdr (its 7-field pack silently shifted p_memsz onto p_align). 739 passed; flake8/mypy and all five gates clean. Address the follow-up review on 0bd8dfe (6 findings) - ete_capture: two-layer quoting for the control-record, decode and cleanup commands -- shlex.quote spliced between literal single quotes broke out of su -c for any spaced output name; --seconds must be positive (timeout 0 disables the timer), refused before any adb check - kernel scaffold: --set-reg now refuses sp/pc as well (the template has no rewrite path for them; a generate that silently drops the action is the misleading kind of success); both templates arm with a sampling period (LKM via hw_breakpoint_init, KPM spells out sample_period=1) -- a zero period is a non-sampling event whose callback never fires: armed, silent - trace_addr2sym: contains_vaddr tests each PT_LOAD's loaded span instead of the whole-file extent; a PC in the unmapped gap between segments is refused rather than named after the nearest symbol below - trace_diff: a bare all-digit token is indistinguishable from a date or count, so block keys need 0x or a non-decimal hex digit; a date-only log stays refused as a capture failure Tests: two-layer quoting with a spaced control name, --seconds refusal, sp/pc in the rejection list, sampling-period assertions for both templates, a two-segment gap fixture, and a date-only parser case. 743 passed; flake8, mypy and all five gates clean.
101 lines
5.9 KiB
Markdown
101 lines
5.9 KiB
Markdown
# Extension verification — sampling, crash forensics, symbolization, de-virtualization
|
||
|
||
Covers the second hardware-layer pass: `references/hardware-sampling-and-crash.md`,
|
||
`scripts/trace_addr2sym.py`, `scripts/vm_handler_slice.py`, and the source-reading
|
||
corrections to the flattening recipes. Strength labels as this directory's README defines
|
||
them.
|
||
|
||
Reference environment: Linux container host, Python 3.13, cross `gcc 16.2.0` targeting
|
||
`aarch64-linux-gnu` (the toolchain that makes the symbolizer's "real ELF" half testable).
|
||
Network access was used once, read-only, to fetch the two public obfuscator implementations
|
||
the calibration cites (below). No Android device, no coresight/SPE/BRBE hardware.
|
||
|
||
**Headline: the address→symbol loop is measured against a real cross-compiled library, the
|
||
de-virtualization slicer is measured on synthetic interpreter traces, and the SPE/BRBE
|
||
producer side remains documented-and-gated. The flattening-recipe corrections are read from
|
||
public source, and that source is named.**
|
||
|
||
---
|
||
|
||
## 1. `trace_addr2sym.py` against a real library — observed
|
||
|
||
```
|
||
$ gcc -shared -fPIC -O2 -o libtlib.so tlib.c # cross gcc -> aarch64 ELF64
|
||
$ readelf --syms -W libtlib.so
|
||
... 00000000000005e0 132 FUNC GLOBAL DEFAULT 9 encrypt_block
|
||
... 0000000000000664 4 FUNC GLOBAL DEFAULT 9 decrypt_block
|
||
$ python skills/apk-reverse/scripts/trace_addr2sym.py --elf libtlib.so --offs 0x610
|
||
0x610 -> encrypt_block+0x30 [vaddr 0x610 file 0x610 .text+0x130]
|
||
```
|
||
|
||
The resolution agrees with `readelf` exactly (0x610 − 0x5e0 = 0x30, inside the sized symbol);
|
||
a point outside any function falls to the nearest-below symbol with its section shown
|
||
(`0x700 -> _fini+0x98 [.eh_frame+0x40]`), which is GNU-addr2line-shaped behaviour and is
|
||
pinned by unit test together with the PT_LOAD `vaddr≠offset` conversion on a hand-built
|
||
minimal ELF. That minimal-ELF fixture also covers the stripped-library fallback
|
||
(section+offset, no symbols).
|
||
|
||
## 2. `vm_handler_slice.py` on the calibrated shape — observed
|
||
|
||
A synthetic interpreter trace in the shape the public IR virtualizer emits
|
||
(dispatcher D, handler blocks, one const-materialization segment, an unclosed tail):
|
||
|
||
```
|
||
$ python skills/apk-reverse/scripts/vm_handler_slice.py --log vm.log --opcode-map map.json
|
||
dispatcher: libvm.so+0x1000 (41.7% of 24 blocks, 10 hits)
|
||
segments: 10 total, 7 unique, avg len 2.00 [last segment UNCLOSED: exit path or truncated window]
|
||
opcode trajectory (first 20): OP_LOAD OP_LOAD OP_STORE OP_BR OP_LOAD OP_LOAD OP_BR OP_CALL OP_LOAD OP_RET
|
||
```
|
||
|
||
What the output pins, and why each part is the part a de-virtualization needs: the
|
||
**dispatcher share** (with the diagnostic that separates a real dispatch loop from a spin
|
||
loop), the **segment table** (one row per handler chain, first block = handler entry =
|
||
the `--opcode-map` key), the **trajectory** (the guest program, one signature per executed
|
||
guest opcode), the **OP_CALL materialization segment** appearing as its own signature (the
|
||
virtualizer's const dispatch — the second observation point the calibration names), and the
|
||
**UNCLOSED tail** flagged rather than silently dropped. The auto-detection rule is
|
||
top-of-histogram **plus predecessor diversity** (≥3 distinct predecessors), not a naive
|
||
count margin: the first implementation used a 4x margin and refused this very fixture —
|
||
in a stack-shaped VM the hottest handler executes on every guest instruction, exactly as
|
||
often as the dispatcher. That refusal-to-accept is the load-bearing part of the auto rule.
|
||
|
||
## 3. The flattening-recipe corrections — observed source, inferred runtime
|
||
|
||
Fetched read-only from the public implementations the user supplied
|
||
(github.com/dpkg123/ollvm-aosp-toolchain: branch `ollvm-r614150-clang23-rebase+omp`
|
||
[Hikari family] and `ollvm-r614150-clang23+omp` [Pluto family]; commits 63fec41f and
|
||
9b7cf75e for the IR virtualizer; 2026-10):
|
||
|
||
- **Hikari `Flattening.cpp`**: `switchVar` is an entry-block **alloca** written through a
|
||
`switchVarAddr` double pointer (indirect store per transition); case values are
|
||
`scramble32`-permuted; EH-bearing functions are skipped.
|
||
- **Pluto `Flattening.cpp`**: two allocas (`switchVar`, `switchXor`) with a **rolling XOR**
|
||
(`enc^=delta; key^=delta; cond=enc^key` per dispatch); case values fully random; the true
|
||
case exists only in the `switchCond` register for one instruction; EH/`callbr`/`indirectbr`
|
||
functions skipped.
|
||
- **Hikari `AntiDebugging.cpp`**: on Darwin/AArch64 it injects **inline-assembly raw `svc`**
|
||
with a randomized register-setup order and a **randomized svc immediate**; elsewhere it
|
||
links precompiled anti-debug IR. Consequence recorded in the reference: byte signatures
|
||
cannot survive the randomization; semantic site-location (`svc_scan.py`) and hardware
|
||
observation do not care.
|
||
- **The IR virtualizer**: `vm_exec(code, size, vm_args, num)` stubs, boxed-argument array,
|
||
11 one-byte opcodes + 19 arithmetic sub-ops, const/call materialization dispatch,
|
||
switch-to-if lowering.
|
||
|
||
These corrected `hardware-watchpoints.md`'s state-variable recipe (the old text assumed a
|
||
global and a readable value — wrong for both families as shipped) and added the
|
||
family-difference table to `ete-diff-tracing.md` §5. The fetches live under
|
||
`tools/_work/refs/` (git-ignored); **runtime behaviour of these passes is `inferred` from
|
||
source, never executed here.**
|
||
|
||
## 4. What this pass deliberately did not do
|
||
|
||
- No SPE capture, no BRBE record read, no `perf -j any` on any device: those routes ship as
|
||
documented-and-gated, and the reference's §5 failure modes say exactly how each gate
|
||
fails silently.
|
||
- No xvmp-hardened binary was compiled or traced: the slicer's calibration is source-read,
|
||
and its first run against a real sample is a measurement, not a confirmation.
|
||
- The `--opcode-map` a real target needs is a manual labelling task over the segment table;
|
||
this pass generated the fixture's map by hand, which is exactly the workflow the script
|
||
documents.
|