mirror of
https://codeberg.org/reverse-skill-lab/apk-reverse
synced 2026-11-15 20:34:29 +08:00
eca9c19ee492bd24cfb4a72d67f4caf73f8cfac7
9
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 inliver233
|
1b94651b1e |
Fix the bare path that turned CI red
The README named `references/routing.md` without the `skills/apk-reverse/` prefix, and `check_repo.py` requires that form in the README. CI was right; the commit that introduced it was pushed without re-running the gates, which is the one step CONTRIBUTING.md asks a contributor to run and I skipped. No behaviour changes: the path is now spelled the way the gate requires in both language editions. The local reproduction is the same as CI's, so this was a process failure rather than a platform-dependent gate — worth stating plainly, because "it passed locally" is exactly the excuse the gate cannot hear. |
||
|
inliver233
|
7d6b4c7fe9 |
Do not overstate what the progressive split buys
SKILL.md loads in full; a reference loads when the symptom index or the routing table sends you to one. That is usually described as cheaper context, and only half of it has been measured here: the always-loaded body has a measured size and a gate that fails when it grows. The other half has not been measured -- how many references one real question pulls in, and whether fetching them one at a time costs more than the alternative would have. A question that needs four references is not obviously cheaper than one longer file, and each reference also carries the index row you have to read to find it. The split's economy is therefore `unknown` and the body budget is `observed`. Saying so is the point: the label discipline is worth nothing if the repository exempts its own design choices from it. |
||
|
inliver233
|
5c3ba9cd2e |
Stop reporting a tool's resource failure as a finding about the target
A JVM reader that dies on a large or deeply nested input -- OutOfMemoryError, an OOM kill, a container limit, a timeout -- prints an empty result that looks exactly like a reader that read the file and found nothing. That is the `present but unreadable` case SKILL.md §Coverage names, and it is where a wrong "that class is not in there" gets recorded. The new section in toolchain.md says what a failed analysis actually supports (a claim about the route, not about the artifact) and the three things to check before a negative from a heavy tool becomes a finding: the exit code and stderr rather than stdout alone, a re-run with room to work, and the indexed-query route that answers most location questions without materialising anything. A deadline you set yourself gets the same reading -- it reported nothing about the target. Also: the README claimed the scripts "work identically on Windows, macOS and Linux", while every gate run, every test result and every figure in the evidence record comes from one Windows host. The claim now carries the strength it actually has -- measured on Windows, `unverified` elsewhere -- and names the usual suspects (path separators, `os.pathsep` in `APKREV_TOOLS`, console encodings, ADB shell quoting) so a first run on another platform has somewhere to look. |
||
|
inliver233
|
6ab905b2c8 |
Four-state labels, and the negatives that used to look alike
Every claim carried one of three labels, and the third was doing two jobs. `unverified` meant
both "someone asserted it and nobody here paid for the counters" and "no evidence was gathered
at all" -- different statements with different next actions. This adds `unknown` for the second.
The distinction that pays for itself lives inside a *negative* result, because an empty probe has
three causes and only one of them is a finding:
absent -- coverage reached the question, nothing is there
present but unreadable -- the thing is there and this route cannot read it
unknown -- nothing covered the question
The middle one is the dangerous one: it produces byte-identical output to the first, and it is the
normal case on the input this skill is aimed at. A packed APK returning four classes is not an APK
with four classes, and "the string was not in the dex" is a statement about a probe. The rule now
says to name the probe's coverage and run a control that must hit before writing "not present", and
to record `unknown` about the probe rather than `absent` about the target.
Also in this pass:
* P47 -- two tools disagree about the same artifact, and the disagreement is your parser. One
extraction rule reused across outputs that do not share a format: an unescaped `$` dropped every
descriptor containing one, and dotted names were compared against DEX descriptors so one side
parsed to nothing. The comfortable failure is the one that confirms what you already expected,
which is why it survived the sanity check it should not have.
* dexsql: a cache is second-hand evidence and it ages silently. A stale cache answers for the old
artifact with no error, and `--trusted-input` skips revalidating the recorded source inputs --
so a cached run supports "true of the artifact the cache was built from", not "true of the
artifact in front of me".
* ete-diff-tracing: a divergence is a location, not a cause. The differential method produces a
positional claim, and three readings it does not authorise are written down beside the step that
produces it.
* CONTRIBUTING.md -- the gates, what a green run does and does not establish, the sync rules a
change has to keep (a new reference must be named in the inventory; renaming a heading breaks
every section reference that pointed at it), and the two failure modes to watch when editing the
gates: a limit that cannot fail, and a gate whose answer varies with the runner. README links it
from both language editions.
|
||
|
dabao1955
|
cd1df50a2a |
Complete the xvmp IR virtualizer end to end and measure the de-virtualization chain on a real fixture
The public xvmp tree ships a compiler half that was never compiled (no TU includes the virtualizer headers; on dev trees they do not build), a weak empty vm_exec stub, silent skips for unsupported IR, and a companion article describing an earlier prototype that is not in the tree. This change completes and measures the chain: - Completed toolchain, merged into all four omp branches of that tree: gated new-PM pass wiring (-mllvm -irobf-irvm, section .irvm), FORMAT v3 bytecode (8-byte aligned slots, explicit load/store widths, stable CC_ cast kinds, per-index GEP byte scales, branch condition = operand 0), a strong-symbol vm_exec interpreter with a 23-check unit suite, and a VMP.h user header. Virtualized vs native output identical at -O0/-O1/-O2 on all four branches. One branch's ungated IRVM call in Obfuscation::runOnModule (rewrote .irvm functions in plain native builds) removed; IRVM is strictly flag-gated. - Fixture measured end to end (fco+bcfobf+subobf+IRVM build): native-identical output; 17,961-step guest trace via VMP_TRACE; 24.6M-instruction host trace via a ptrace single-stepper; vm_handler_slice yields 95,184 segments / 18 unique signatures; the fla outer state machine separates as a shuttle class, round-boundary segments split the trajectory, positional alignment names every handler segment (59/59 in a round; counts match the guest histograms exactly), zero unmapped with --opcode-map. - Tooling hardening measured on the real fixture: opcode maps key by signature (first-block keys collide under a shared dispatch successor); apply_map keeps first-block keys as fallback, regression tests added. - Environment boundary recorded: frida stalker does not work in these containers (measured past the syscall layer: primitives pass, agent loads, zero denied syscalls; the failure is inside the injected agent's hook machinery), an outer ptrace tracer conflicts with frida spawn, and the ptrace stepper is the working host capture path. Design lesson: a pure virtualized call can still be LICM-hoisted by the compiling optimizer. Records: EXTENSION-sampling-crash.md SS5-SS6 (final-state rewrite), known-limitations.md, ete-diff-tracing.md SS5, vm_handler_slice.py header, capability matrix. |
||
|
inliver233
|
e151d7d605 |
Merge PR #6 "Hardware pass (stage 1)" (dabao1955/feat/ete)
Merges the hardware-layer pass onto the dexsql commit: arm64 hardware
watchpoints, ETE differential tracing, kernel-side event injection, SPE/BRBE
sampling and crash forensics, address symbolization and interpreter-trace
de-slicing. 33 files, 6 new scripts, 6 new test modules, 4 new reference
documents.
The only conflict was in evidence/capability-matrix.json: both sides appended
to the same array tail. Resolved by keeping both -- the PR's seven entries
(44 rows total now) -- and re-validated as JSON before staging.
Verified on the merged tree, not on the PR branch alone:
* check_repo.py, check_refs.py, check_commands.py, check_routing.py and
check_budget.py all pass with zero problems;
* pytest: 766 passed, 172 skipped, 25 xfailed -- the 141 tests this PR adds
all pass alongside the 625 that were already there;
* the PR's host-side claims were reproduced by hand, not taken on trust:
`hw_watch.py plan --addr 0x1004 --len 4 --access w` returns
`cr: 7701` (WCR 0x1E15) exactly as the record says, both refusals exit 2
with the documented reason strings, and the kernel scaffold's
generate/verify round trip returns RESULT=ok.
One documentation defect was found and fixed in the merge: EXTENSION-kernel-instrument.md
recorded `verified 3 kernel-side file(s)` while the same page's own generate
output lists four (kpm/src/kwatch.c, kpm/Makefile, lkm/kwatch.c, lkm/Makefile)
and the tool reports 4. The page now says 4.
Everything the PR ships that needs a device or a kernel toolchain stays
labelled unverified in the files themselves, which is the property that made
it mergeable.
|
||
|
inliver233
|
cbbeb58343 |
Adopt dexsql as a third indexer, parity-checked against rasc, droidasc and ddc
LSPosed/dexsql exposes the whole dex as SQL tables (callers, field accesses,
numeric constants, debug lines, annotations, hiddenapi flags) plus the APK
manifest and resources.arsc, plus an MCP server. It is a third indexer at the
same stage as ASC, so it is registered the same way:
* scripts/capabilities.py gains tool:dexsql and the dex_query_sql capability,
so doctor.py reports it instead of guessing. INSTALL_HINTS replaces the
rasc-only hardcoded path list, so a release download or a from-source build
that landed outside PATH is not reported BLOCKED.
* references/dexsql-sql-query-layer.md carries the table model, the recipes,
the MCP tool list and the boundaries; references/toolchain.md gains the
Tier-1 row and a section; routing.md, SKILL.md, both READMEs and the
evidence record are updated in step.
What was measured, on four archives (1.4 MB / 1,396 classes, 4.3 MB / 647,
38.7 MB / 4 hardened, 34.8 MB / 30,768):
* class-definition sets identical to rasc, droidasc and ddc on all four,
zero differences in either direction;
* string cross-reference agrees with ASC site for site -- and the match
semantics differ, which is recorded as a trap (`=` is not `findrefs`);
* decode emits the same instruction sequence as ddc disasm, line for line;
* the persistent cache is NOT a speed switch: 0.121 s -> 0.098 s on a 34.8 MB
archive, a net loss on a small one. Recorded as measured, not as sold.
Full class listing, best of 3: dexsql 0.136 s against droidasc 0.200 s and
rasc 0.055 s on the 34.8 MB archive, so it does not replace rasc for a class
list -- it answers the joins and aggregates the other two cannot express.
|
||
|
root
|
ab22b225ff |
Hardware pass: hardware watchpoints, ETE differential tracing, kernel-side event injection
Add the layer under the software tooling, in the repository's shape: what runs on a host is measured, and every device-bound or kernel-bound piece is a gated, labelled template. - scripts/hw_watch.py: arm64 hardware watchpoints/execution breakpoints. WCR/ BCR encoders with the 8-byte-window and alignment rules (unit-tested to the canonical 0x1FFD/0x3F05), a self-contained on-device ptrace helper that compiles clean as a static aarch64 ELF (observed: cross gcc 16.2), adb orchestration, and hit decoding to module+offset. The stealth ledger is stated three times on purpose: the ptrace route leaves TracerPid visible. - scripts/ete_capture.py: CoreSight ETE capture with the five gates (kernel era, coresight sysfs, cs_etm PMU, perf-with-CORESIGHT, permission), the perf/OpenCSD build recipe, and the per-thread record wrapper whose --control one-instruction run must pass before any real capture. - scripts/trace_diff.py: differential reduction of two execution traces into first-divergence / divergent regions / blocks-unique-to-each-side. Producer-agnostic: Stalker BLK logs (raw or run_probe-wrapped) and decoded cs_etm address streams alike, host-tested on synthetic flattened-dispatcher fixtures. - scripts/kernel_instrument_scaffold.py: KPM/LKM watchpoint + register-rewrite templates plus an honest userspace carrier; every kernel file carries a NOT COMPILED AND NOT LOADED banner that verify enforces, and the register-rewrite action is fenced as experimental in the template itself. - references/hardware-watchpoints.md, references/ete-diff-tracing.md, references/kernel-event-injection.md: the mechanism, the gates, the traps, and the composition recipes (watch the key buffer; diff two runs; flip the decision at the trap), with hand-offs instead of duplication. - Registration: 3 symptom rows (SKILL.md + routing mirror), 3 reference rows, 4 script rows, coverage/route-inventory/evidence rows, capability-matrix entries, known-limitations entry, eval case 4, README trees and coverage paragraphs in both languages, and three EXTENSION evidence records stating exactly what was measured on this host (encoders, compile, fixtures, generate/verify) and what was not (no watchpoint armed, no cs_etm stream, nothing kernel-side compiled or loaded anywhere). Hardware pass 2: SPE/BRBE crash forensics, address symbolization, VM de-virtualization Calibrated against the two public obfuscator implementations the user supplied (dpkg123/ollvm-aosp-toolchain: Hikari-family and Pluto-family flattening, the xvmp IR virtualizer at 63fec41f/9b7cf75e; fetched read-only into tools/_work/refs, git-ignored). Source facts observed, runtime inferred. - scripts/trace_addr2sym.py: runtime addresses -> symbol + static offset, zero-dependency ELF64 (PT_LOAD vaddr<->file-off conversion, .dynsym/.symtab, stripped section+off fallback). Host-verified against a real cross-compiled aarch64 .so with readelf as the independent cross-check, plus a hand-built minimal ELF covering vaddr!=offset; accepts hw_watch hits, trace mod+off keys, or raw address lists. Fixes on the way: --base int coercion, and nearest-below symbol semantics for PCs in inter-symbol gaps. - scripts/vm_handler_slice.py: slice an interpreter trace into handler segments (dispatcher detection = histogram top + predecessor diversity; the first 4x-margin rule refused the calibrated fixture itself and was replaced), signature clustering, open-tail handling, and the guest-opcode trajectory via an optional handler->opcode map. Calibrated against xvmp's documented shape: vm_exec(code,size,vm_args,num) stubs, boxed arguments observable before interpretation, 11 opcodes + 19 arithmetic sub-ops, const-materialization dispatch as a distinct segment. - references/hardware-sampling-and-crash.md: SPE (data flow without a known address; PC + data address per sample), BRBE (branch records when a crash stack cannot be trusted), the crash loop (tombstone -> branch records -> symbolize), and what each adds to the OLLVM/VMP routes. All documented-and-gated: no SPE sample or BRBE record was captured here. - Flattening-recipe corrections from source: Hikari keeps a frame-resident switchVar written through an indirect pointer with scramble32-permuted cases; Pluto stores switchVar/switchXor rolling-XOR so memory only ever holds noise -- hardware-watchpoints.md's state-variable recipe now distinguishes the families, and ete-diff-tracing.md carries the family table plus the xvmp shape. AntiDebugging injects raw svc inline asm with randomized fields, so byte signatures cannot survive it; semantic site-location can. - Registration: 3 symptom rows (untrustworthy crash stack; runtime addresses needing symbols; interpreter loop reading as noise), reference/script rows, coverage/evidence/capability-matrix entries, README trees in both languages, and EXTENSION-sampling-crash.md recording exactly what was measured (real .so symbolization, slicer fixtures, the slicer's self-found dispatcher-rule bug) and what was not (no SPE/BRBE capture, no hardened binary ever traced). Fix all PR #1 review findings by severity; un-entrench the tests that pinned them Major (device-breaking / wrong results): - hw_watch: match the arm64 user_hwdebug_state slot ABI (address register first, control second -- the swap armed nothing correctly) - hw_watch: every post-seize error path now clears slots and detaches; PTRACE_O_EXITKILL turned an analysis failure into a SIGKILL of the target - hw_watch: forward genuine target SIGTRAPs (raise/tgkill/int3); only ptrace event stops are consumed -- the reference's signal-preservation promise - hw_watch: run checks push/build/chmod and the helper's exit status; a dead capture is RESULT=blocked, never ok-no-hits - hw_watch: decode offsets from the module base, not each mapping's start - ete_capture: --control now requires the control record to DECODE (perf script event lines), the documented gate, not just a zero exit - ete_capture: -o before the -- workload separator in --cmd mode; all values shell-quoted into su -c; stale remote perf.data removed first; perf exit status checked (timeout's 124 is the normal window end) - kernel scaffold: --set-reg restricted to x0..x30; the rewrite is gated by its own breakpoint event so a watchpoint hit cannot run it; LKM pins the task (get/put_task_struct) across registration and calls the four-argument register_user_hw_breakpoint; --break rows initialize .addr/.len/.bp_type (there is no .pc member); --id validated against the module-id grammar - trace_addr2sym: exact module matching (path or basename, never substring); load bias aligned against the PT_LOAD (bias = start - (foff - p_offset) - p_vaddr); PCs outside the ELF span are refused instead of landing on the last symbol; --json stdout is a pure document (RESULT= in text mode only) - trace_diff: block keys need 0x or >=8 hex digits (PIDs/counts no longer parse); vm_handler_slice: dispatcher needs predecessor diversity, no frequency-margin fallback; both JSON outputs pure (check_commands' rule) Minor/nitpick: - capability-matrix + evidence-summary: trace reduction status ok->partial (no real-target producer ran it; the file's own definition of ok) - hardware-sampling-and-crash.md and the script examples carry --elf - kernel-event-injection.md no longer claims the callback's first statement is a filter (it prints unfiltered; the filter is a required change) - Elf64 caches symbol addresses for per-row bisect - docstring coverage of the touched surface: 16.67% -> 100% Tests: every json.loads workaround stripping RESULT= is gone; new regression tests cover the slot ABI, release-on-error, SIGTRAP forwarding, non-first mapping offsets, -o placement and su -c quoting, the control decode gate, x31/--id refusals, the guarded rewrite, outside-ELF PCs and bias math, short-token refusal and the dispatcher margin rule. The minimal-ELF fixture now packs a full 8-field Elf64_Phdr (its 7-field pack silently shifted p_memsz onto p_align). 739 passed; flake8/mypy and all five gates clean. Address the follow-up review on 0bd8dfe (6 findings) - ete_capture: two-layer quoting for the control-record, decode and cleanup commands -- shlex.quote spliced between literal single quotes broke out of su -c for any spaced output name; --seconds must be positive (timeout 0 disables the timer), refused before any adb check - kernel scaffold: --set-reg now refuses sp/pc as well (the template has no rewrite path for them; a generate that silently drops the action is the misleading kind of success); both templates arm with a sampling period (LKM via hw_breakpoint_init, KPM spells out sample_period=1) -- a zero period is a non-sampling event whose callback never fires: armed, silent - trace_addr2sym: contains_vaddr tests each PT_LOAD's loaded span instead of the whole-file extent; a PC in the unmapped gap between segments is refused rather than named after the nearest symbol below - trace_diff: a bare all-digit token is indistinguishable from a date or count, so block keys need 0x or a non-decimal hex digit; a date-only log stays refused as a capture failure Tests: two-layer quoting with a spaced control name, --seconds refusal, sp/pc in the rejection list, sampling-period assertions for both templates, a two-segment gap fixture, and a date-only parser case. 743 passed; flake8, mypy and all five gates clean. |
||
|
newliver666
|
7b6c6932a9 | README: add a theme-aware header, live badges, and a Simplified Chinese edition |