mirror of
https://codeberg.org/reverse-skill-lab/apk-reverse
synced 2026-11-15 20:34:29 +08:00
Add the layer under the software tooling, in the repository's shape: what runs on a host is measured, and every device-bound or kernel-bound piece is a gated, labelled template. - scripts/hw_watch.py: arm64 hardware watchpoints/execution breakpoints. WCR/ BCR encoders with the 8-byte-window and alignment rules (unit-tested to the canonical 0x1FFD/0x3F05), a self-contained on-device ptrace helper that compiles clean as a static aarch64 ELF (observed: cross gcc 16.2), adb orchestration, and hit decoding to module+offset. The stealth ledger is stated three times on purpose: the ptrace route leaves TracerPid visible. - scripts/ete_capture.py: CoreSight ETE capture with the five gates (kernel era, coresight sysfs, cs_etm PMU, perf-with-CORESIGHT, permission), the perf/OpenCSD build recipe, and the per-thread record wrapper whose --control one-instruction run must pass before any real capture. - scripts/trace_diff.py: differential reduction of two execution traces into first-divergence / divergent regions / blocks-unique-to-each-side. Producer-agnostic: Stalker BLK logs (raw or run_probe-wrapped) and decoded cs_etm address streams alike, host-tested on synthetic flattened-dispatcher fixtures. - scripts/kernel_instrument_scaffold.py: KPM/LKM watchpoint + register-rewrite templates plus an honest userspace carrier; every kernel file carries a NOT COMPILED AND NOT LOADED banner that verify enforces, and the register-rewrite action is fenced as experimental in the template itself. - references/hardware-watchpoints.md, references/ete-diff-tracing.md, references/kernel-event-injection.md: the mechanism, the gates, the traps, and the composition recipes (watch the key buffer; diff two runs; flip the decision at the trap), with hand-offs instead of duplication. - Registration: 3 symptom rows (SKILL.md + routing mirror), 3 reference rows, 4 script rows, coverage/route-inventory/evidence rows, capability-matrix entries, known-limitations entry, eval case 4, README trees and coverage paragraphs in both languages, and three EXTENSION evidence records stating exactly what was measured on this host (encoders, compile, fixtures, generate/verify) and what was not (no watchpoint armed, no cs_etm stream, nothing kernel-side compiled or loaded anywhere). Hardware pass 2: SPE/BRBE crash forensics, address symbolization, VM de-virtualization Calibrated against the two public obfuscator implementations the user supplied (dpkg123/ollvm-aosp-toolchain: Hikari-family and Pluto-family flattening, the xvmp IR virtualizer at 63fec41f/9b7cf75e; fetched read-only into tools/_work/refs, git-ignored). Source facts observed, runtime inferred. - scripts/trace_addr2sym.py: runtime addresses -> symbol + static offset, zero-dependency ELF64 (PT_LOAD vaddr<->file-off conversion, .dynsym/.symtab, stripped section+off fallback). Host-verified against a real cross-compiled aarch64 .so with readelf as the independent cross-check, plus a hand-built minimal ELF covering vaddr!=offset; accepts hw_watch hits, trace mod+off keys, or raw address lists. Fixes on the way: --base int coercion, and nearest-below symbol semantics for PCs in inter-symbol gaps. - scripts/vm_handler_slice.py: slice an interpreter trace into handler segments (dispatcher detection = histogram top + predecessor diversity; the first 4x-margin rule refused the calibrated fixture itself and was replaced), signature clustering, open-tail handling, and the guest-opcode trajectory via an optional handler->opcode map. Calibrated against xvmp's documented shape: vm_exec(code,size,vm_args,num) stubs, boxed arguments observable before interpretation, 11 opcodes + 19 arithmetic sub-ops, const-materialization dispatch as a distinct segment. - references/hardware-sampling-and-crash.md: SPE (data flow without a known address; PC + data address per sample), BRBE (branch records when a crash stack cannot be trusted), the crash loop (tombstone -> branch records -> symbolize), and what each adds to the OLLVM/VMP routes. All documented-and-gated: no SPE sample or BRBE record was captured here. - Flattening-recipe corrections from source: Hikari keeps a frame-resident switchVar written through an indirect pointer with scramble32-permuted cases; Pluto stores switchVar/switchXor rolling-XOR so memory only ever holds noise -- hardware-watchpoints.md's state-variable recipe now distinguishes the families, and ete-diff-tracing.md carries the family table plus the xvmp shape. AntiDebugging injects raw svc inline asm with randomized fields, so byte signatures cannot survive it; semantic site-location can. - Registration: 3 symptom rows (untrustworthy crash stack; runtime addresses needing symbols; interpreter loop reading as noise), reference/script rows, coverage/evidence/capability-matrix entries, README trees in both languages, and EXTENSION-sampling-crash.md recording exactly what was measured (real .so symbolization, slicer fixtures, the slicer's self-found dispatcher-rule bug) and what was not (no SPE/BRBE capture, no hardened binary ever traced). Fix all PR #1 review findings by severity; un-entrench the tests that pinned them Major (device-breaking / wrong results): - hw_watch: match the arm64 user_hwdebug_state slot ABI (address register first, control second -- the swap armed nothing correctly) - hw_watch: every post-seize error path now clears slots and detaches; PTRACE_O_EXITKILL turned an analysis failure into a SIGKILL of the target - hw_watch: forward genuine target SIGTRAPs (raise/tgkill/int3); only ptrace event stops are consumed -- the reference's signal-preservation promise - hw_watch: run checks push/build/chmod and the helper's exit status; a dead capture is RESULT=blocked, never ok-no-hits - hw_watch: decode offsets from the module base, not each mapping's start - ete_capture: --control now requires the control record to DECODE (perf script event lines), the documented gate, not just a zero exit - ete_capture: -o before the -- workload separator in --cmd mode; all values shell-quoted into su -c; stale remote perf.data removed first; perf exit status checked (timeout's 124 is the normal window end) - kernel scaffold: --set-reg restricted to x0..x30; the rewrite is gated by its own breakpoint event so a watchpoint hit cannot run it; LKM pins the task (get/put_task_struct) across registration and calls the four-argument register_user_hw_breakpoint; --break rows initialize .addr/.len/.bp_type (there is no .pc member); --id validated against the module-id grammar - trace_addr2sym: exact module matching (path or basename, never substring); load bias aligned against the PT_LOAD (bias = start - (foff - p_offset) - p_vaddr); PCs outside the ELF span are refused instead of landing on the last symbol; --json stdout is a pure document (RESULT= in text mode only) - trace_diff: block keys need 0x or >=8 hex digits (PIDs/counts no longer parse); vm_handler_slice: dispatcher needs predecessor diversity, no frequency-margin fallback; both JSON outputs pure (check_commands' rule) Minor/nitpick: - capability-matrix + evidence-summary: trace reduction status ok->partial (no real-target producer ran it; the file's own definition of ok) - hardware-sampling-and-crash.md and the script examples carry --elf - kernel-event-injection.md no longer claims the callback's first statement is a filter (it prints unfiltered; the filter is a required change) - Elf64 caches symbol addresses for per-row bisect - docstring coverage of the touched surface: 16.67% -> 100% Tests: every json.loads workaround stripping RESULT= is gone; new regression tests cover the slot ABI, release-on-error, SIGTRAP forwarding, non-first mapping offsets, -o placement and su -c quoting, the control decode gate, x31/--id refusals, the guarded rewrite, outside-ELF PCs and bias math, short-token refusal and the dispatcher margin rule. The minimal-ELF fixture now packs a full 8-field Elf64_Phdr (its 7-field pack silently shifted p_memsz onto p_align). 739 passed; flake8/mypy and all five gates clean. Address the follow-up review on 0bd8dfe (6 findings) - ete_capture: two-layer quoting for the control-record, decode and cleanup commands -- shlex.quote spliced between literal single quotes broke out of su -c for any spaced output name; --seconds must be positive (timeout 0 disables the timer), refused before any adb check - kernel scaffold: --set-reg now refuses sp/pc as well (the template has no rewrite path for them; a generate that silently drops the action is the misleading kind of success); both templates arm with a sampling period (LKM via hw_breakpoint_init, KPM spells out sample_period=1) -- a zero period is a non-sampling event whose callback never fires: armed, silent - trace_addr2sym: contains_vaddr tests each PT_LOAD's loaded span instead of the whole-file extent; a PC in the unmapped gap between segments is refused rather than named after the nearest symbol below - trace_diff: a bare all-digit token is indistinguishable from a date or count, so block keys need 0x or a non-decimal hex digit; a date-only log stays refused as a capture failure Tests: two-layer quoting with a spaced control name, --seconds refusal, sp/pc in the rejection list, sampling-period assertions for both templates, a two-segment gap fixture, and a date-only parser case. 743 passed; flake8, mypy and all five gates clean.
184 lines
7.7 KiB
Python
184 lines
7.7 KiB
Python
"""Unit tests for trace_diff.py: parsers, run-length, and the differential.
|
|
|
|
The fixtures are synthetic flattened-dispatcher traces (D R1 D R2 ...), because
|
|
that is the shape the differ exists to answer: same machinery, different
|
|
input-dependent blocks. One fixture is in raw stalker_trace.js grammar, the
|
|
other wrapped by run_probe.py, so the parser's tolerance is under test too.
|
|
"""
|
|
import json
|
|
|
|
import pytest
|
|
|
|
from conftest import load_script, run_script
|
|
|
|
td = load_script("trace_diff.py")
|
|
|
|
DISP = {"D": 0x1000, "R1": 0x2000, "R2": 0x3000, "R3": 0x4000,
|
|
"R4": 0x5000, "R5": 0x6000, "R7": 0x8000}
|
|
|
|
|
|
def stalker_log(seq, wrap=False):
|
|
"""Render a synthetic stalker log from a block sequence."""
|
|
lines = []
|
|
for i, k in enumerate(seq.split()):
|
|
body = "BLK %d liba.so+0x%x" % (i, DISP[k])
|
|
lines.append(("[123.4] [56.7] TRACE %s" % body) if wrap else body)
|
|
lines.append("DONE reason=stop-rpc blocks=8 blk=8 calls=0 truncated=0")
|
|
return "\n".join(lines) + "\n"
|
|
|
|
|
|
RUN_A = "D R1 D R2 D R5 D R7" # wrong password: rejects after two extra blocks
|
|
RUN_B = "D R1 D R2 D R3 D R4" # right password: accepts
|
|
|
|
|
|
@pytest.fixture
|
|
def pair(tmp_path):
|
|
"""Two synthetic trace logs: unwrapped and run_probe-wrapped."""
|
|
a = tmp_path / "a.log"
|
|
b = tmp_path / "b.log"
|
|
a.write_text(stalker_log(RUN_A), encoding="utf-8")
|
|
b.write_text(stalker_log(RUN_B, wrap=True), encoding="utf-8")
|
|
return a, b
|
|
|
|
|
|
class TestParsers:
|
|
def test_stalker_unwrapped_and_wrapped(self, pair):
|
|
"""Stalker unwrapped and wrapped."""
|
|
a, b = pair
|
|
blocks = td.read_blocks(str(a))
|
|
assert blocks == ["liba.so+0x%x" % DISP[k] for k in RUN_A.split()]
|
|
|
|
def test_wrapped_log_parses_identically(self, pair):
|
|
"""Wrapped log parses identically."""
|
|
a, b = pair
|
|
assert td.parse_line_stalker("[h] [d] TRACE BLK 7 liba.so+0x8000") == "liba.so+0x8000"
|
|
|
|
def test_plain_format_keeps_module_keys_and_hex(self, tmp_path):
|
|
"""Plain format keeps module keys and hex."""
|
|
p = tmp_path / "plain.log"
|
|
p.write_text("0x7f0000001000\n7f0000002000 x\nlibb.so+0x9f6c0\nnoise line\n",
|
|
encoding="utf-8")
|
|
blocks = td.read_blocks(str(p), "plain")
|
|
# keys are kept verbatim (lowercased), so 0x-less and 0x-full forms stay distinct
|
|
assert blocks == ["0x7f0000001000", "7f0000002000", "libb.so+0x9f6c0"]
|
|
|
|
def test_plain_format_refuses_ambiguous_decimal_tokens(self, tmp_path):
|
|
"""Bare runs under 8 hex digits are PIDs, counts and timestamp
|
|
fragments; an all-digit 8+ run is indistinguishable from a date --
|
|
taking either as a block key adds noise, and a log with no real
|
|
addresses must stay refused (an empty trace is a capture failure,
|
|
not a result)."""
|
|
p = tmp_path / "pids.log"
|
|
p.write_text("app 1234 567\nsample count 99\ndead\n", encoding="utf-8")
|
|
with pytest.raises(td.TraceError, match="capture"):
|
|
td.read_blocks(str(p), "plain")
|
|
d = tmp_path / "date.log"
|
|
d.write_text("20261015\n", encoding="utf-8")
|
|
with pytest.raises(td.TraceError, match="capture"):
|
|
td.read_blocks(str(d), "plain")
|
|
p2 = tmp_path / "mixed.log"
|
|
p2.write_text("app 1234\n7f0000002000 x\n", encoding="utf-8")
|
|
assert td.read_blocks(str(p2), "plain") == ["7f0000002000"]
|
|
|
|
def test_empty_log_is_a_capture_failure_not_a_result(self, tmp_path):
|
|
"""Empty log is a capture failure not a result."""
|
|
p = tmp_path / "empty.log"
|
|
p.write_text("READY ...\nDONE reason=timeout blocks=0 blk=0 calls=0\n",
|
|
encoding="utf-8")
|
|
with pytest.raises(td.TraceError, match="capture"):
|
|
td.read_blocks(str(p))
|
|
|
|
def test_auto_detection_prefers_stalker_then_plain(self, tmp_path):
|
|
"""Auto detection prefers stalker then plain."""
|
|
p = tmp_path / "mixed.log"
|
|
p.write_text("BLK 1 liba.so+0x1000\n0x7f0000002000\n", encoding="utf-8")
|
|
assert td.read_blocks(str(p)) == ["liba.so+0x1000", "0x7f0000002000"]
|
|
|
|
|
|
class TestReduce:
|
|
def test_histogram_names_the_dispatcher(self, pair):
|
|
"""Histogram names the dispatcher."""
|
|
a, _ = pair
|
|
rep = td.reduce_trace(td.read_blocks(str(a)))
|
|
assert rep["events"] == 8
|
|
assert rep["top"][0] == ("liba.so+0x1000", 4) # D: the wide-margin top
|
|
assert rep["runs"] == 8
|
|
|
|
def test_run_length_collapses_repeats(self):
|
|
"""Run length collapses repeats."""
|
|
assert td.run_length(["D", "D", "D", "R1", "R1"]) == \
|
|
[("D", 3), ("R1", 2)]
|
|
|
|
|
|
class TestDiff:
|
|
def test_first_divergence_after_the_shared_prefix(self, pair):
|
|
"""First divergence after the shared prefix."""
|
|
a, b = pair
|
|
d = td.diff_traces(td.read_blocks(str(a)), td.read_blocks(str(b)))
|
|
fd = d["first_divergence"]
|
|
assert fd is not None
|
|
# shared prefix D R1 D R2 = 5 events on both sides
|
|
assert fd["event_index_a"] == 5 and fd["event_index_b"] == 5
|
|
assert fd["last_common_block"] == "liba.so+0x1000" # the dispatcher
|
|
# shared span = prefix D,R1,D,R2,D (5 runs) + trailing D (1) = 6 runs
|
|
assert d["identical_run_span"] == 6
|
|
|
|
def test_unique_sets_are_the_input_dependent_blocks(self, pair):
|
|
"""Unique sets are the input dependent blocks."""
|
|
a, b = pair
|
|
d = td.diff_traces(td.read_blocks(str(a)), td.read_blocks(str(b)))
|
|
only_a = {k for k, _ in d["unique_to_a"]}
|
|
only_b = {k for k, _ in d["unique_to_b"]}
|
|
assert only_a == {"liba.so+0x6000", "liba.so+0x8000"} # R5, R7
|
|
assert only_b == {"liba.so+0x4000", "liba.so+0x5000"} # R3, R4
|
|
|
|
def test_identical_traces_report_no_divergence(self, pair):
|
|
"""Identical traces report no divergence."""
|
|
a, _ = pair
|
|
blocks = td.read_blocks(str(a))
|
|
d = td.diff_traces(blocks, blocks)
|
|
assert d["first_divergence"] is None
|
|
assert d["divergent_regions"] == 0
|
|
assert "NO DIVERGENCE" in td.render_diff(d)
|
|
|
|
def test_render_names_every_part(self, pair):
|
|
"""Render names every part."""
|
|
a, b = pair
|
|
d = td.diff_traces(td.read_blocks(str(a)), td.read_blocks(str(b)))
|
|
text = td.render_diff(d, top=5)
|
|
for marker in ("first divergence", "region 1", "unique to A", "unique to B"):
|
|
assert marker in text
|
|
|
|
|
|
class TestCli:
|
|
def test_reduce_cli(self, pair):
|
|
"""Reduce cli."""
|
|
a, _ = pair
|
|
r = run_script("trace_diff.py", ["reduce", "--log", str(a), "--top", "3"])
|
|
assert r.returncode == 0 and "RESULT=ok" in r.output
|
|
assert "liba.so+0x1000" in r.output
|
|
|
|
def test_diff_cli_json(self, pair):
|
|
"""Diff cli json."""
|
|
a, b = pair
|
|
r = run_script("trace_diff.py", ["diff", "--a", str(a), "--b", str(b), "--json"])
|
|
assert r.returncode == 0, r.output
|
|
doc = json.loads(r.stdout) # --json stdout is one pure document
|
|
assert doc["status"] == "ok"
|
|
assert doc["first_divergence"]["event_index_a"] == 5
|
|
assert {k for k, _ in doc["unique_to_b"]} == \
|
|
{"liba.so+0x4000", "liba.so+0x5000"}
|
|
|
|
def test_missing_file_is_usage_error(self):
|
|
"""Missing file is usage error."""
|
|
r = run_script("trace_diff.py", ["diff", "--a", "nope.log", "--b", "nada.log"])
|
|
assert r.returncode == 2
|
|
|
|
def test_empty_file_refused_with_the_lesson(self, tmp_path):
|
|
"""Empty file refused with the lesson."""
|
|
p = tmp_path / "e.log"
|
|
p.write_text("DONE reason=timeout blocks=0 blk=0\n", encoding="utf-8")
|
|
r = run_script("trace_diff.py", ["reduce", "--log", str(p)])
|
|
assert r.returncode == 2
|
|
assert "capture failure" in r.output or "zero-event" in r.output
|