Add the layer under the software tooling, in the repository's shape: what runs on a host is measured, and every device-bound or kernel-bound piece is a gated, labelled template. - scripts/hw_watch.py: arm64 hardware watchpoints/execution breakpoints. WCR/ BCR encoders with the 8-byte-window and alignment rules (unit-tested to the canonical 0x1FFD/0x3F05), a self-contained on-device ptrace helper that compiles clean as a static aarch64 ELF (observed: cross gcc 16.2), adb orchestration, and hit decoding to module+offset. The stealth ledger is stated three times on purpose: the ptrace route leaves TracerPid visible. - scripts/ete_capture.py: CoreSight ETE capture with the five gates (kernel era, coresight sysfs, cs_etm PMU, perf-with-CORESIGHT, permission), the perf/OpenCSD build recipe, and the per-thread record wrapper whose --control one-instruction run must pass before any real capture. - scripts/trace_diff.py: differential reduction of two execution traces into first-divergence / divergent regions / blocks-unique-to-each-side. Producer-agnostic: Stalker BLK logs (raw or run_probe-wrapped) and decoded cs_etm address streams alike, host-tested on synthetic flattened-dispatcher fixtures. - scripts/kernel_instrument_scaffold.py: KPM/LKM watchpoint + register-rewrite templates plus an honest userspace carrier; every kernel file carries a NOT COMPILED AND NOT LOADED banner that verify enforces, and the register-rewrite action is fenced as experimental in the template itself. - references/hardware-watchpoints.md, references/ete-diff-tracing.md, references/kernel-event-injection.md: the mechanism, the gates, the traps, and the composition recipes (watch the key buffer; diff two runs; flip the decision at the trap), with hand-offs instead of duplication. - Registration: 3 symptom rows (SKILL.md + routing mirror), 3 reference rows, 4 script rows, coverage/route-inventory/evidence rows, capability-matrix entries, known-limitations entry, eval case 4, README trees and coverage paragraphs in both languages, and three EXTENSION evidence records stating exactly what was measured on this host (encoders, compile, fixtures, generate/verify) and what was not (no watchpoint armed, no cs_etm stream, nothing kernel-side compiled or loaded anywhere). Hardware pass 2: SPE/BRBE crash forensics, address symbolization, VM de-virtualization Calibrated against the two public obfuscator implementations the user supplied (dpkg123/ollvm-aosp-toolchain: Hikari-family and Pluto-family flattening, the xvmp IR virtualizer at 63fec41f/9b7cf75e; fetched read-only into tools/_work/refs, git-ignored). Source facts observed, runtime inferred. - scripts/trace_addr2sym.py: runtime addresses -> symbol + static offset, zero-dependency ELF64 (PT_LOAD vaddr<->file-off conversion, .dynsym/.symtab, stripped section+off fallback). Host-verified against a real cross-compiled aarch64 .so with readelf as the independent cross-check, plus a hand-built minimal ELF covering vaddr!=offset; accepts hw_watch hits, trace mod+off keys, or raw address lists. Fixes on the way: --base int coercion, and nearest-below symbol semantics for PCs in inter-symbol gaps. - scripts/vm_handler_slice.py: slice an interpreter trace into handler segments (dispatcher detection = histogram top + predecessor diversity; the first 4x-margin rule refused the calibrated fixture itself and was replaced), signature clustering, open-tail handling, and the guest-opcode trajectory via an optional handler->opcode map. Calibrated against xvmp's documented shape: vm_exec(code,size,vm_args,num) stubs, boxed arguments observable before interpretation, 11 opcodes + 19 arithmetic sub-ops, const-materialization dispatch as a distinct segment. - references/hardware-sampling-and-crash.md: SPE (data flow without a known address; PC + data address per sample), BRBE (branch records when a crash stack cannot be trusted), the crash loop (tombstone -> branch records -> symbolize), and what each adds to the OLLVM/VMP routes. All documented-and-gated: no SPE sample or BRBE record was captured here. - Flattening-recipe corrections from source: Hikari keeps a frame-resident switchVar written through an indirect pointer with scramble32-permuted cases; Pluto stores switchVar/switchXor rolling-XOR so memory only ever holds noise -- hardware-watchpoints.md's state-variable recipe now distinguishes the families, and ete-diff-tracing.md carries the family table plus the xvmp shape. AntiDebugging injects raw svc inline asm with randomized fields, so byte signatures cannot survive it; semantic site-location can. - Registration: 3 symptom rows (untrustworthy crash stack; runtime addresses needing symbols; interpreter loop reading as noise), reference/script rows, coverage/evidence/capability-matrix entries, README trees in both languages, and EXTENSION-sampling-crash.md recording exactly what was measured (real .so symbolization, slicer fixtures, the slicer's self-found dispatcher-rule bug) and what was not (no SPE/BRBE capture, no hardened binary ever traced). Fix all PR #1 review findings by severity; un-entrench the tests that pinned them Major (device-breaking / wrong results): - hw_watch: match the arm64 user_hwdebug_state slot ABI (address register first, control second -- the swap armed nothing correctly) - hw_watch: every post-seize error path now clears slots and detaches; PTRACE_O_EXITKILL turned an analysis failure into a SIGKILL of the target - hw_watch: forward genuine target SIGTRAPs (raise/tgkill/int3); only ptrace event stops are consumed -- the reference's signal-preservation promise - hw_watch: run checks push/build/chmod and the helper's exit status; a dead capture is RESULT=blocked, never ok-no-hits - hw_watch: decode offsets from the module base, not each mapping's start - ete_capture: --control now requires the control record to DECODE (perf script event lines), the documented gate, not just a zero exit - ete_capture: -o before the -- workload separator in --cmd mode; all values shell-quoted into su -c; stale remote perf.data removed first; perf exit status checked (timeout's 124 is the normal window end) - kernel scaffold: --set-reg restricted to x0..x30; the rewrite is gated by its own breakpoint event so a watchpoint hit cannot run it; LKM pins the task (get/put_task_struct) across registration and calls the four-argument register_user_hw_breakpoint; --break rows initialize .addr/.len/.bp_type (there is no .pc member); --id validated against the module-id grammar - trace_addr2sym: exact module matching (path or basename, never substring); load bias aligned against the PT_LOAD (bias = start - (foff - p_offset) - p_vaddr); PCs outside the ELF span are refused instead of landing on the last symbol; --json stdout is a pure document (RESULT= in text mode only) - trace_diff: block keys need 0x or >=8 hex digits (PIDs/counts no longer parse); vm_handler_slice: dispatcher needs predecessor diversity, no frequency-margin fallback; both JSON outputs pure (check_commands' rule) Minor/nitpick: - capability-matrix + evidence-summary: trace reduction status ok->partial (no real-target producer ran it; the file's own definition of ok) - hardware-sampling-and-crash.md and the script examples carry --elf - kernel-event-injection.md no longer claims the callback's first statement is a filter (it prints unfiltered; the filter is a required change) - Elf64 caches symbol addresses for per-row bisect - docstring coverage of the touched surface: 16.67% -> 100% Tests: every json.loads workaround stripping RESULT= is gone; new regression tests cover the slot ABI, release-on-error, SIGTRAP forwarding, non-first mapping offsets, -o placement and su -c quoting, the control decode gate, x31/--id refusals, the guarded rewrite, outside-ELF PCs and bias math, short-token refusal and the dispatcher margin rule. The minimal-ELF fixture now packs a full 8-field Elf64_Phdr (its 7-field pack silently shifted p_memsz onto p_align). 739 passed; flake8/mypy and all five gates clean. Address the follow-up review on 0bd8dfe (6 findings) - ete_capture: two-layer quoting for the control-record, decode and cleanup commands -- shlex.quote spliced between literal single quotes broke out of su -c for any spaced output name; --seconds must be positive (timeout 0 disables the timer), refused before any adb check - kernel scaffold: --set-reg now refuses sp/pc as well (the template has no rewrite path for them; a generate that silently drops the action is the misleading kind of success); both templates arm with a sampling period (LKM via hw_breakpoint_init, KPM spells out sample_period=1) -- a zero period is a non-sampling event whose callback never fires: armed, silent - trace_addr2sym: contains_vaddr tests each PT_LOAD's loaded span instead of the whole-file extent; a PC in the unmapped gap between segments is refused rather than named after the nearest symbol below - trace_diff: a bare all-digit token is indistinguishable from a date or count, so block keys need 0x or a non-decimal hex digit; a date-only log stays refused as a capture failure Tests: two-layer quoting with a spaced control name, --seconds refusal, sp/pc in the rejection list, sampling-period assertions for both templates, a two-segment gap fixture, and a date-only parser case. 743 passed; flake8, mypy and all five gates clean.
4.3 KiB
Extension verification — ETE differential tracing (gates, reducer, differ)
Covers the hardware-layer pass that added scripts/ete_capture.py, scripts/trace_diff.py and
references/ete-diff-tracing.md. Strength labels as this directory's README defines them.
Reference environment: Linux container host, Python 3.13, adb present with no device. No
coresight-capable hardware exists in this environment at all; the reference device of the
earlier passes (kernel 4.14.186) has the ETE gate closed permanently (observed there).
Headline: the gate table, the recipe and the capture wrapper are host-measured; the
differential reducer is host-measured on synthetic flattened-dispatcher fixtures. No ETE trace
was captured or decoded anywhere in this pass — the producer side is unverified and the
--control run exists precisely so the next device says so in one command.
1. The differ, measured on a synthetic OLLVM shape — observed
Two fixture logs built to the flattened-dispatcher rhythm D R1 D R2 … — run A taking
R5/R7 (wrong password), run B taking R3/R4 (right password), one log in raw
stalker_trace.js grammar, one wrapped by run_probe.py ([ts] [ts] TRACE BLK …) to prove
the parser survives both:
$ python skills/apk-reverse/scripts/trace_diff.py diff --a a.log --b b.log
run A: 8 events, 5 distinct blocks
run B: 8 events, 5 distinct blocks
shared block executions: 6 divergent regions: 2
first divergence: after run #5 (A) / #5 (B), 5/5 events in, last common block liba.so+0x1000
region 1 (replace)
A: liba.so+0x6000 B: liba.so+0x4000
blocks unique to A ...: 0x6000, 0x8000 blocks unique to B ...: 0x4000, 0x5000
What the output pins, and why each number is the one the method needs:
- the identical prefix (
D R1 D R2, 5 events) is the shared machinery — exactly where parsing ends and the input-dependent path begins; - first divergence names the dispatcher entry whose state constant differed;
- unique-to-each-side lists are the input-dependent block set, which is the deliverable the reference file's §4 interprets per obfuscation.
reduce on one side prints the histogram with the dispatcher as the wide-margin top
(4 hits vs 1 for every real block) — the same shape stalker_report.py documents for Stalker
logs, reached from a different producer.
2. Gate and capture plumbing — observed (to the no-device boundary)
gates prints the five-gate table; gates --device with adb present but no device degrades
to five OPEN rows and RESULT=partial rather than a traceback — measured. capture --pid 1
without a device exits through the pull failure with RESULT=blocked and exit code 3
(no-device capability refusal, per the CLI contract) — measured. recipe prints the
Termux and cross build paths for perf-with-CORESIGHT — inferred from the kernel's
tools/perf CoreSight documentation and the OpenCSD project, not from a build performed here.
3. What this pass deliberately did not do
- No perf-with-CORESIGHT was built (host or device): the recipe is a documented path, and a
reader who builds it owes the control record (
capture --control) before any real capture. - No decoded cs_etm stream was fed to
trace_diff.py; theplainparser accepts the address-bearing linesperf scriptemits, but that exact pairing isunverified. - No claim about ETE's runtime overhead or its visibility to targets was measured. The reference states the mechanism (no code modification, no ptrace) and refuses to go further.
4. Facts this pass pinned for the next one
- The differ is producer-agnostic by design: the same command answers Stalker logs today and cs_etm decode output tomorrow. That is why the diff subcommand ships now, not after the first ETE capture — it is useful the day a Stalker pair exists.
- The five gates each fail silently downstream; the reference's table says which silence
each gate produces. The one-instruction control record is the only gate that exercises the
whole pipe, which is why
capturerefuses to continue when it fails. - Run design is part of the tool contract, not user care:
trace_diff.pyrefuses empty logs with a message that points at the zero-event diagnostics, because an empty trace is a capture failure wearing an analysis result's clothes.