5 Commits

Include renames

Author	SHA1	Message	Date
inliver233	b8aa0a529b	Make the budget gate's hard index-row limit actually fail ... `check_index_rows` read `'warn' if hard else 'warn'` -- both branches identical -- so INDEX_ROW_HARD was defined, computed, and unable to fail anything: an index row could grow without bound while every gate stayed green. Found by injecting an over-long row into SKILL.md and requiring the gate to fail. That is the only way a dead threshold shows itself; a gate that has only ever been observed passing is not evidence that it can see anything. A test now pins both sides -- a row past the soft limit warns, a row past the hard limit fails -- and it was checked against the pre-fix behaviour, where it fails.	2026-10-16 19:17:21 +08:00
dabao1955	cd1df50a2a	Complete the xvmp IR virtualizer end to end and measure the de-virtualization chain on a real fixture ... The public xvmp tree ships a compiler half that was never compiled (no TU includes the virtualizer headers; on dev trees they do not build), a weak empty vm_exec stub, silent skips for unsupported IR, and a companion article describing an earlier prototype that is not in the tree. This change completes and measures the chain: - Completed toolchain, merged into all four omp branches of that tree: gated new-PM pass wiring (-mllvm -irobf-irvm, section .irvm), FORMAT v3 bytecode (8-byte aligned slots, explicit load/store widths, stable CC_ cast kinds, per-index GEP byte scales, branch condition = operand 0), a strong-symbol vm_exec interpreter with a 23-check unit suite, and a VMP.h user header. Virtualized vs native output identical at -O0/-O1/-O2 on all four branches. One branch's ungated IRVM call in Obfuscation::runOnModule (rewrote .irvm functions in plain native builds) removed; IRVM is strictly flag-gated. - Fixture measured end to end (fco+bcfobf+subobf+IRVM build): native-identical output; 17,961-step guest trace via VMP_TRACE; 24.6M-instruction host trace via a ptrace single-stepper; vm_handler_slice yields 95,184 segments / 18 unique signatures; the fla outer state machine separates as a shuttle class, round-boundary segments split the trajectory, positional alignment names every handler segment (59/59 in a round; counts match the guest histograms exactly), zero unmapped with --opcode-map. - Tooling hardening measured on the real fixture: opcode maps key by signature (first-block keys collide under a shared dispatch successor); apply_map keeps first-block keys as fallback, regression tests added. - Environment boundary recorded: frida stalker does not work in these containers (measured past the syscall layer: primitives pass, agent loads, zero denied syscalls; the failure is inside the injected agent's hook machinery), an outer ptrace tracer conflicts with frida spawn, and the ptrace stepper is the working host capture path. Design lesson: a pure virtualized call can still be LICM-hoisted by the compiling optimizer. Records: EXTENSION-sampling-crash.md SS5-SS6 (final-state rewrite), known-limitations.md, ete-diff-tracing.md SS5, vm_handler_slice.py header, capability matrix.	2026-10-16 14:45:03 +08:00
root	ab22b225ff	Hardware pass: hardware watchpoints, ETE differential tracing, kernel-side event injection ... Add the layer under the software tooling, in the repository's shape: what runs on a host is measured, and every device-bound or kernel-bound piece is a gated, labelled template. - scripts/hw_watch.py: arm64 hardware watchpoints/execution breakpoints. WCR/ BCR encoders with the 8-byte-window and alignment rules (unit-tested to the canonical 0x1FFD/0x3F05), a self-contained on-device ptrace helper that compiles clean as a static aarch64 ELF (observed: cross gcc 16.2), adb orchestration, and hit decoding to module+offset. The stealth ledger is stated three times on purpose: the ptrace route leaves TracerPid visible. - scripts/ete_capture.py: CoreSight ETE capture with the five gates (kernel era, coresight sysfs, cs_etm PMU, perf-with-CORESIGHT, permission), the perf/OpenCSD build recipe, and the per-thread record wrapper whose --control one-instruction run must pass before any real capture. - scripts/trace_diff.py: differential reduction of two execution traces into first-divergence / divergent regions / blocks-unique-to-each-side. Producer-agnostic: Stalker BLK logs (raw or run_probe-wrapped) and decoded cs_etm address streams alike, host-tested on synthetic flattened-dispatcher fixtures. - scripts/kernel_instrument_scaffold.py: KPM/LKM watchpoint + register-rewrite templates plus an honest userspace carrier; every kernel file carries a NOT COMPILED AND NOT LOADED banner that verify enforces, and the register-rewrite action is fenced as experimental in the template itself. - references/hardware-watchpoints.md, references/ete-diff-tracing.md, references/kernel-event-injection.md: the mechanism, the gates, the traps, and the composition recipes (watch the key buffer; diff two runs; flip the decision at the trap), with hand-offs instead of duplication. - Registration: 3 symptom rows (SKILL.md + routing mirror), 3 reference rows, 4 script rows, coverage/route-inventory/evidence rows, capability-matrix entries, known-limitations entry, eval case 4, README trees and coverage paragraphs in both languages, and three EXTENSION evidence records stating exactly what was measured on this host (encoders, compile, fixtures, generate/verify) and what was not (no watchpoint armed, no cs_etm stream, nothing kernel-side compiled or loaded anywhere). Hardware pass 2: SPE/BRBE crash forensics, address symbolization, VM de-virtualization Calibrated against the two public obfuscator implementations the user supplied (dpkg123/ollvm-aosp-toolchain: Hikari-family and Pluto-family flattening, the xvmp IR virtualizer at 63fec41f/9b7cf75e; fetched read-only into tools/_work/refs, git-ignored). Source facts observed, runtime inferred. - scripts/trace_addr2sym.py: runtime addresses -> symbol + static offset, zero-dependency ELF64 (PT_LOAD vaddr<->file-off conversion, .dynsym/.symtab, stripped section+off fallback). Host-verified against a real cross-compiled aarch64 .so with readelf as the independent cross-check, plus a hand-built minimal ELF covering vaddr!=offset; accepts hw_watch hits, trace mod+off keys, or raw address lists. Fixes on the way: --base int coercion, and nearest-below symbol semantics for PCs in inter-symbol gaps. - scripts/vm_handler_slice.py: slice an interpreter trace into handler segments (dispatcher detection = histogram top + predecessor diversity; the first 4x-margin rule refused the calibrated fixture itself and was replaced), signature clustering, open-tail handling, and the guest-opcode trajectory via an optional handler->opcode map. Calibrated against xvmp's documented shape: vm_exec(code,size,vm_args,num) stubs, boxed arguments observable before interpretation, 11 opcodes + 19 arithmetic sub-ops, const-materialization dispatch as a distinct segment. - references/hardware-sampling-and-crash.md: SPE (data flow without a known address; PC + data address per sample), BRBE (branch records when a crash stack cannot be trusted), the crash loop (tombstone -> branch records -> symbolize), and what each adds to the OLLVM/VMP routes. All documented-and-gated: no SPE sample or BRBE record was captured here. - Flattening-recipe corrections from source: Hikari keeps a frame-resident switchVar written through an indirect pointer with scramble32-permuted cases; Pluto stores switchVar/switchXor rolling-XOR so memory only ever holds noise -- hardware-watchpoints.md's state-variable recipe now distinguishes the families, and ete-diff-tracing.md carries the family table plus the xvmp shape. AntiDebugging injects raw svc inline asm with randomized fields, so byte signatures cannot survive it; semantic site-location can. - Registration: 3 symptom rows (untrustworthy crash stack; runtime addresses needing symbols; interpreter loop reading as noise), reference/script rows, coverage/evidence/capability-matrix entries, README trees in both languages, and EXTENSION-sampling-crash.md recording exactly what was measured (real .so symbolization, slicer fixtures, the slicer's self-found dispatcher-rule bug) and what was not (no SPE/BRBE capture, no hardened binary ever traced). Fix all PR #1 review findings by severity; un-entrench the tests that pinned them Major (device-breaking / wrong results): - hw_watch: match the arm64 user_hwdebug_state slot ABI (address register first, control second -- the swap armed nothing correctly) - hw_watch: every post-seize error path now clears slots and detaches; PTRACE_O_EXITKILL turned an analysis failure into a SIGKILL of the target - hw_watch: forward genuine target SIGTRAPs (raise/tgkill/int3); only ptrace event stops are consumed -- the reference's signal-preservation promise - hw_watch: run checks push/build/chmod and the helper's exit status; a dead capture is RESULT=blocked, never ok-no-hits - hw_watch: decode offsets from the module base, not each mapping's start - ete_capture: --control now requires the control record to DECODE (perf script event lines), the documented gate, not just a zero exit - ete_capture: -o before the -- workload separator in --cmd mode; all values shell-quoted into su -c; stale remote perf.data removed first; perf exit status checked (timeout's 124 is the normal window end) - kernel scaffold: --set-reg restricted to x0..x30; the rewrite is gated by its own breakpoint event so a watchpoint hit cannot run it; LKM pins the task (get/put_task_struct) across registration and calls the four-argument register_user_hw_breakpoint; --break rows initialize .addr/.len/.bp_type (there is no .pc member); --id validated against the module-id grammar - trace_addr2sym: exact module matching (path or basename, never substring); load bias aligned against the PT_LOAD (bias = start - (foff - p_offset) - p_vaddr); PCs outside the ELF span are refused instead of landing on the last symbol; --json stdout is a pure document (RESULT= in text mode only) - trace_diff: block keys need 0x or >=8 hex digits (PIDs/counts no longer parse); vm_handler_slice: dispatcher needs predecessor diversity, no frequency-margin fallback; both JSON outputs pure (check_commands' rule) Minor/nitpick: - capability-matrix + evidence-summary: trace reduction status ok->partial (no real-target producer ran it; the file's own definition of ok) - hardware-sampling-and-crash.md and the script examples carry --elf - kernel-event-injection.md no longer claims the callback's first statement is a filter (it prints unfiltered; the filter is a required change) - Elf64 caches symbol addresses for per-row bisect - docstring coverage of the touched surface: 16.67% -> 100% Tests: every json.loads workaround stripping RESULT= is gone; new regression tests cover the slot ABI, release-on-error, SIGTRAP forwarding, non-first mapping offsets, -o placement and su -c quoting, the control decode gate, x31/--id refusals, the guarded rewrite, outside-ELF PCs and bias math, short-token refusal and the dispatcher margin rule. The minimal-ELF fixture now packs a full 8-field Elf64_Phdr (its 7-field pack silently shifted p_memsz onto p_align). 739 passed; flake8/mypy and all five gates clean. Address the follow-up review on 0bd8dfe (6 findings) - ete_capture: two-layer quoting for the control-record, decode and cleanup commands -- shlex.quote spliced between literal single quotes broke out of su -c for any spaced output name; --seconds must be positive (timeout 0 disables the timer), refused before any adb check - kernel scaffold: --set-reg now refuses sp/pc as well (the template has no rewrite path for them; a generate that silently drops the action is the misleading kind of success); both templates arm with a sampling period (LKM via hw_breakpoint_init, KPM spells out sample_period=1) -- a zero period is a non-sampling event whose callback never fires: armed, silent - trace_addr2sym: contains_vaddr tests each PT_LOAD's loaded span instead of the whole-file extent; a PC in the unmapped gap between segments is refused rather than named after the nearest symbol below - trace_diff: a bare all-digit token is indistinguishable from a date or count, so block keys need 0x or a non-decimal hex digit; a date-only log stays refused as a capture failure Tests: two-layer quoting with a spaced control name, --seconds refusal, sp/pc in the rejection list, sampling-period assertions for both templates, a two-segment gap fixture, and a date-only parser case. 743 passed; flake8, mypy and all five gates clean.	2026-10-15 15:33:28 +08:00
inliver233	dd8057cfcb	Make the command gate machine-independent, and calibrate the budget fallback ... CI went red on `1dc1d9b` in both jobs while the identical command passed locally, and the reason is worth keeping: the checks depended on state that only exists on a developer's machine. check_commands.py -- the gate's verdict depended on a gitignored directory The previous commit's fix recognised a workbench script by matching its name against a walk of `tools/`. That tree is gitignored, so a clean checkout has none of it: `dcc.py` matched locally and skipped, and on CI nothing matched, it fell through to `missing-script`, and the gate reported drift for three quoted lines. A check whose verdict changes with a directory CI does not have is not a check. Two changes remove the dependency: * `qualified_basenames()` collects every `.py` the *document itself* writes with a path, so a bare name the document has already located elsewhere is a skip, computed from the text. * `./dcc.py` is a declared upstream entry point, checked before the workbench-name test. * a bare name after a `cd` that resolves nowhere is `after-cd-unresolved` -- unverifiable, which is not the same claim as broken, and is now reported as such instead of as drift. Verified by running the whole CI sequence against a tree with `tools/` hidden: five gates exit 0, the suite passes. That experiment is the one that should have been run before the first push. check_budget.py -- the measurement was installation-dependent The gate measures the always-loaded body in tokens, preferring `tiktoken` and falling back to `characters / N`. `N` had been guessed at 3.6 rather than calibrated, so a machine with tiktoken saw ~11.1k tokens and passed while a machine without it saw ~14.2k and warned -- and CI's 3.11 job is the one without it. `N` is now the measured 4.285 characters per token (47,629 chars -> 11,114 tokens on SKILL.md), both paths agree within one token, and `tests/unit/test_budget_measurement.py` forces both paths and asserts they do not straddle the budget. CI now installs `tiktoken` so the comparison exercises the real tokenizer instead of skipping. Also: two `.py` files are formatted for consistency, and the reconstruction record documents the CI-only failure, since "it passed on my machine" is the lesson rather than an aside. Gates: check_repo 0 problem, check_refs 0 dangling / 0 warning, check_routing coherent, check_budget 0 failure / 0 warning (both measurement paths), check_commands 154 checked / 0 drift, pytest 617 passed on 3.14 and 609 on 3.11.	2026-09-22 17:09:22 +08:00
inliver233	2beade67f2	Reconstruction pass: make the gates tell the truth, make the package self-contained, keep the one-hop symptom index ... An external audit of this repository landed on four structural findings, and this pass is the response to them. Everything below was re-measured on this tree before being acted on; two of the audit's claims needed a correction of my own, and both corrections are noted in place. **The gates were optimistic, and one of them did not read content at all.** doctor.py the capability table was hard-coded (`repack + sign = bool(java) and python`), so a machine with a JDK and no zipalign/apksigner was told it could re-sign; the script inventory was a hand-written list of 28 against 51 files on disk, omitting the nine most important scripts. It now scans the directory (`registered/checked = 52 / 52`, unjudgeable entries counted as `unknown`, never skipped) and resolves capability as a real requires-closure, reporting BLOCKED with the missing piece and an installable next step. Measured: the old doctor reports `[OK] repack + sign` on a JDK-only PATH where keytool, jarsigner, zipalign, apksigner and all eight smali jars are absent. The report is allowed to look bad; that is the point. check_commands.py (new) anchor checks prove a section exists, never that a command runs. This parses each script's argparse table with `ast` and checks every documented invocation against it. 146 commands / 41 interfaces, 0 drift. It caught `sig_probe.py --live --pkg`, which does not exist -- the reference had been wrong since it was written. check_routing.py (new) the symptom index now has one home (`SKILL.md`, because symptom -> file must stay one hop) and the inventory has another (`references/routing.md`). This gate proves the mirror agrees, that every reference file is named, and that every script is named. **so_constpatch.py was corrupting APKs it rebuilt.** The writer copied filename and timestamp onto a fresh ZipInfo and recompressed the whole archive: 435-entry real package, `extra` lost on 205 entries, `external_attr` rewritten on all 435, 132 untouched entries recompressed, and `resources.arsc` pushed to an unaligned offset. Android's own `zipalign -c -v 4` fails the old output and passes the new one; a 26-case fixture suite pins it, and `extractNativeLibs="false"` is now refused by default with a pointer at `repack.py`. Fixing it exposed the same defect in `repack.py`, where `^lib/[^/]+\.so$` cannot match `lib/arm64-v8a/*.so` -- so its alignment handling had never once fired on a real APK. **The package was not self-contained.** 59 references pointed at the repository root, which `npx skills add` does not install. They now converge on `references/evidence-summary.md` (capability, conclusion, strength, evidence that ships) plus `evidence/` with the machine-readable companions: 8 references remain and each is explicitly labelled "repository root, not shipped". A drill in which the new files were copied to an isolated directory and asked three questions answered all three. **SKILL.md: 572 lines / 15,733 tokens -> 398 lines / ~10,900 tokens**, without losing the two things that had to stay. The symptom index remains inline; the reference and script indexes moved to the on-demand routing file. The freed budget went into making the rules *stronger*, not shorter: the constraints section is now a table of thirteen non-negotiable constraints each paired with the check that catches a violation, and the tooling obligations are instructions whose violation is a defect (hedging language in SKILL.md: 1 line, down from a paragraph of hedged advice). Long references gained a navigation head so a reader can decide from the top whether to load one. **New: an executable test layer.** `tests/` (unit, CLI contract, no-device integration) plus `pytest.ini`, `tests/run_tests.py`, a minimal CI job, and `evals/evals.json` written but never run -- stated as such in the file. 608 passed / 158 skipped / 25 xfailed. The 25 xfails are a work order, not a pass: thirteen scripts report a missing input by exiting 1 instead of 2, seven let a traceback escape, four treat `--help` as an error, and `preflight --json` lacks a `status` field. A regression row for `dex_patch_bytes.py` reproduces a previous artifact byte-for-byte (`67ae1e58e7a44777e5ac8f9f823ab21e…`) and additionally pins the four bytes that may move. **Two real defects the new tests found in the tooling itself.** `check_repo.py` decoded subprocess output with the locale encoding, so on a GBK console a script printing an em dash in its usage line was reported as crashed; it now decodes UTF-8 explicitly. And a CLI contract test drove a *real attached device* through `coldstart.py` -- logcat, screenshots, five orphan `adb logcat` processes -- which is fixed twice over: child processes run in a private temporary cwd, and the fifteen device-bound scripts are never executed beyond `--help`/argument parsing. **Quoting on the device side is now one implementation.** `devsh.su` escaped only double quotes, so `;`, `$( )` and backticks in a caller-supplied value still reached the device shell while `coldstart.Device.shell` quoted correctly. Both use `scripts/device_shell.py` now: POSIX single-quote escaping, argv arrays by default, and identifier validation that refuses a malformed package, path or component with exit 2 instead of a traceback. The integration test that used to pin the *defect* (it asserted `su -c "…"` with `\"` escapes) now asserts the property, across five metacharacter payloads. Gates: check_repo 0 problems (including the content-reading identity scan over git ls-files), check_refs 0 dangling / 0 warning, check_routing coherent (53 symptom / 45 reference / 54 script rows), check_budget 0 failure / 0 warning with the always-loaded body now measured in tokens rather than in "narrative lines with the index excluded", check_commands 146 checked / 0 drift, pytest 608 passed. Not done, and stated rather than implied: CI has never run on a runner; the evals have never been executed; the 25 xfails are open; no device-side install/launch was performed in this pass.	2026-09-22 01:37:51 +08:00