diff --git a/README.md b/README.md index 663b396..b302316 100644 --- a/README.md +++ b/README.md @@ -77,8 +77,10 @@ measurements. - Deciding **what the deliverable should be when an APK is not an option** — a repack refused by several independent checks is *blocked*, not expensive, and the fallback ladder is a system-level module, a local RPC service, or an honest report with a stated boundary. -- Telling a **real memory dump from an extraction-shell skeleton**, by measuring the trivial-body - ratio instead of eyeballing it — and knowing which recovery route applies, including the root-side +- Telling a **real memory dump from an extraction-shell skeleton** — and knowing that the + trivial-body ratio is *bimodal rather than a threshold*: it reads ~100% for a bare `return-void` + skeleton, but is blind to nop-cleared bodies, throw stubs and partial extraction, which are the + shapes real shells actually use. Then knowing which recovery route applies, including the root-side dump for when `frida` itself is refused by the target. - **Calling a routine instead of reversing it** when reversing costs more than invoking: emulated execution on the host, or a live function service-ified over Frida RPC. @@ -91,6 +93,15 @@ measurements. native-side certificate pinning that ignores the system trust store. - Working **from the phone itself**: MT Manager's edit/repack/sign flow and its APK MCP surface, LSPosed Manager, and on-device data inspection, alongside the PC toolchain rather than instead of it. +- Telling **Java2C apart from an extraction shell** before spending hours hunting a decrypted DEX that + does not exist at any point in the process lifetime — the code was compiled into a `.so`, and the + shapes are separated by a measured native-density difference of roughly 2000x, not by intuition. +- Handling a build that arrives as a **split APK / App Bundle set**: reading the set off a device, + signing every member with one keystore for `pm install-multiple`, or merging code/native members + into a standalone APK when that is legal. +- Working a **real Dex VMP** with the known-plaintext differential — which links can be automated and + which cannot, what a compiled fixture can and cannot reach, and how to *prove* a derived + private-opcode table instead of asserting one. ## Structure @@ -207,6 +218,19 @@ references/ loaded on demand, one topic each gate, and when to stop escalating on-device-tooling.md working from the phone itself: MT Manager edit/repack/sign and its APK MCP, LSPosed Manager, Termux+frida, on-device data inspection + java2c-and-jni-sinking.md Java2C and JNI sinking, the two hardening shapes most easily confused + with an extraction shell: the table that separates landing shell / + extraction shell / VMP / Java2C / JNI sinking, why the code is in the + `.so` and *never* in a dumped dex, and why a `Java_*` symbol search + comes back empty (dynamic registration, `-fvisibility=hidden`) + split-apk.md App Bundle / split APK sets: what the set is, pulling it off a device, + merging into one APK vs signing the set as a unit, the install refusals + and what each means, and making an installable fixture from a pulled set + vmp-differential-analysis.md + the known-plaintext differential for a real Dex VMP: which links can be + automated and which cannot (the upload is the bottleneck), the coverage a + compiled fixture can reach, how to *prove* a derived private-opcode + table, smali generation, and when the route is closed scripts/ parameterized, path-agnostic doctor.py run this first: capability report + per-script runnability, finds tools installed off-PATH or as runnable jars, and surfaces the @@ -242,7 +266,9 @@ scripts/ parameterized, path-agnostic dart_disasm.py annotated windowed disassembly of Dart AOT code + B/BL caller index find_refs.py count callers of a method before patching it repack.py rebuild APK, strip only signatures, keep META-INF/services/, write a - 4-byte-aligned archive (resources.arsc STORED+aligned), sign, verify + 4-byte-aligned archive (resources.arsc STORED+aligned), sign, verify; + also split APK / App Bundle sets: inventory, sign every member with one + keystore, or merge code/native members into a standalone APK devsh.py quoting-safe ADB shell helper usb_net_proxy.py give an offline device network over USB datastore_inject.py encode/inject AndroidX DataStore preferences safely @@ -294,6 +320,19 @@ scripts/ parameterized, path-agnostic with an explicit diagnostic for the measured zero-event case mt_mcp_probe.py probe MT Manager's on-device APK MCP (Streamable HTTP, port 8787): JSON-RPC handshake plus the grouped tool inventory + java2c_probe.py collect the evidence that separates Java2C from an extraction shell, a + VMP and ordinary JNI sinking: native density and stub ratio from the dex, + JNI_OnLoad / dynamic registration / toolchain strings from the `.so`, + each item labelled strong/medium/weak + protobuf_decode_raw.py schema-free protobuf decode: hex / file / stdin to a JSON tree, every + length-delimited field kept as a candidate set with ties labelled rather + than guessed, plus a byte-exact re-encode to check a round trip + vmp_diff_harness.py build a labelled opcode-coverage fixture, derive a candidate private- + opcode map from an original/hardened dex pair, verify the comparison in a + closed loop, and render a restored stream as a smali skeleton + kernelsu_syscall_mask.py generate a KernelSU/APatch syscall-masking scaffold: an installable + userspace module skeleton plus KPM/LKM/eBPF kernel-side templates, each + with its version gate and an explicit unverified label ``` ## Install @@ -383,7 +422,13 @@ tier of documented routes: **module-side delivery** when a repack is blocked, **extraction-shell recovery** and its VMP boundary, **emulation and live RPC** for calling rather than reading, **instruction-level tracing** against OLLVM, **protocol reversing** beyond REST, the **kernel-side route** map for when userspace hooking is -provably out of reach, and **on-device tooling**. Unity/IL2CPP logic recovery, +provably out of reach, and **on-device tooling**. A **benchmark pass** then put public +targets under those routes (`tests/benchmark.md`): it added **Java2C discrimination** (the +misdiagnosis that sends an agent hunting a decrypted DEX that never exists), **split APK / +App Bundle handling**, **schema-free protobuf decoding**, a **Dex-VMP differential** harness, +and **kernel-module templates with their version gates** — and it corrected two earlier +claims, replacing the trivial-body *threshold* with a measured bimodal result and reopening +a VMP verdict that a hand-written opcode table had got wrong. Unity/IL2CPP logic recovery, React Native/Hermes bytecode internals, and defeating a server-side authority are **not** covered, and the skill is written to say so and stop rather than apply the nearest documented procedure to a target it was not written for. @@ -402,6 +447,10 @@ Three qualifications that the Coverage section states in full and that belong he `docs/tool-verification/EXTENSION-*.md`, one file per topic, with its own strength note. The common shape there is *the tool was measured, the route was not* — so read those files before treating any of the newer documents as a verified path. +- **The benchmark pass is recorded per row, with that row's own strength.** `tests/benchmark.md` + names each public target, the scripts the row exercises, what actually happened (including the + rows that failed and the rows nobody ran), and how strong the evidence is. Rows marked + `unverified` are statements about the evidence in this repository, not about the mechanism. ## Repository maintenance @@ -415,6 +464,11 @@ check_refs.py every cross-reference that names a section of another build_scripts.py audit for machine-specific leftovers (absolute paths, credentials) ``` +`tests/benchmark.md` holds the **regression matrix**: dimension -> public target -> the scripts the +row exercises -> measured result -> strength label. It is the checklist to re-run before trusting +any claim under `docs/tool-verification/`. Samples are downloaded into `tools/_work/` and are never +committed, so each row names its public source and records the hash it was run against. + `docs/tool-verification/` is not part of the installed skill either. It is the evidence record for one measurement pass against a real target: what each script actually did, which independent method confirmed it, which defects were found, and which scenarios the target could not exercise. diff --git a/docs/tool-verification/EXTENSION-benchmark-l1-l3.md b/docs/tool-verification/EXTENSION-benchmark-l1-l3.md new file mode 100644 index 0000000..2e52587 --- /dev/null +++ b/docs/tool-verification/EXTENSION-benchmark-l1-l3.md @@ -0,0 +1,638 @@ +# Extension verification — benchmark B1/B2 against OWASP MASTG Crackmes + +Covers the pass that turned the `tests/benchmark.md` rows **B1** (equal-length dex patch + repack + +re-sign installability, gates G2/G4) and **B2** (anti-debug / self-destruct race) from `pending` +into measured results. + +Strength labels follow `docs/tool-verification/README.md`: **observed** (exact command and its +output reproduced here), **inferred** (follows from observed facts, the step itself not executed), +**unverified** (assumed or reported and not independently confirmed). + +Reference environment: Windows 11 host, Python 3.14.0, host frida 16.7.19 (frida-tools 13.7.1), +JDK 17.0.4.1 at `C:\Program Files\Java\jdk-17.0.4.1` (not on `PATH`), `apksigner.bat` / +`zipalign.exe` at `E:\tools\bin`. Device: **``**, Android 11 / API 30, arm64-v8a, root via +`su`, Magisk denylist `enforced` with an **empty** list (no app is root-hidden). Package under test: +**``**. + +All device operations ran under the shared device lock (`tools/_work/devlock.py`). + +--- + +## 0. Headline + +- **B1 is `observed` and positive.** An equal-length dex edit (2 bytes of a conditional branch + replaced by `nop`s), through `dex_patch_bytes.py` → `repack.py --no-sign` → `zipalign` → + `apksigner` → root `pm install`, installs and **demonstrably changes the app's behaviour on + screen**: the original and a zero-change control both show a blocking `Root detected!` dialog, the + patched build reaches `MainActivity` with the dialog gone. +- **`repack.py`'s default signing path did not work when this pass started, and the blocker was not + `apksigner`.** `apksigner`/`zipalign` are both present and both work; at that moment `repack.py` + signed only through an `uber-apk-signer` jar, which is not installed. **The repository changed + mid-pass** — another writer landed a `sign_with_apksigner()` route — and re-running the same + command then produced a fully signed, `[result] OK` APK in one invocation with byte-identical + output to the manual route (`sha256 ad6a51ce…7e56`). Both states are recorded below, because the + stale state is what `tools/_work/ENV.md` predicted and the new state is what a reader will meet. + This is also a reminder that a "the tool cannot do X" finding on a shared worktree has an expiry + date. +- **B2 is a split result.** The two scripts run and the probe writes its bytes (0.04–0.05 s after + resume, gated on `PATCHED`), so the *tooling* is verified. But the **patch does not prevent the + target's termination**: `libfoo.so` aborts through `goodbye()+12` at **3.22 s** after resume, and + three patched sites including `goodbyev` itself do not change that. Neutralising the obvious death + site is **not sufficient** on this target. +- **A finding that changes the whole B2 setup:** on this device the **unmodified, original APK + already reports `Rooting or tampering detected.`** and its tamper path runs before anything is + attached. L3's native CRC self-check fails against the pristine community APK on Android 11 / + arm64-v8a, so the "pristine baseline" the row assumed does not exist here. + +--- + +## 1. Gate G2 and G4 — environment and control (`observed`) + +`scripts/doctor.py`: + +``` +platform : Windows 11 / AMD64 +python : 3.14.0 + [OK ] static (dex/zip/strings) [OK ] repack + sign + [OK ] native ELF patching [-- ] jadx decompile + [-- ] apktool unpack [OK ] device work + [OK ] root on device [OK ] frida dynamic + [OK ] java 17.0.4.1 [OK ] adb 1.0.41 [OK ] frida 16.7.19 + [-- ] apktool not on PATH + [-- ] unzip not on PATH + java jars found near this skill : (none) + device : arm64-v8a Android 11 (sdk 30) root: yes + clock skew: 3.06 s +``` + +The `(none)` on the jar line matters for B1: no `baksmali`/`smali`/`uber-apk-signer` jar exists, so +every signing route that goes through a jar is unavailable before a single command is run. + +**Keystore and tool locations** (recorded because `repack.py` resolves both off-`PATH`): + +``` +keytool : C:\Program Files\Java\jdk-17.0.4.1\bin\keytool.exe +jarsigner : C:\Program Files\Java\jdk-17.0.4.1\bin\jarsigner.exe +apksigner : E:\tools\bin\apksigner.bat +zipalign : E:\tools\bin\zipalign.exe +``` + +`repack.py`'s `sibling_tool()` cannot find `keytool` from `C:\Program Files\Common Files\Oracle\Java\javapath\java.exe`, +because that directory holds only forwarders. Passing `--keytool` explicitly is required. + +--- + +## 2. B1 — L1 equal-length dex patch, full chain (`observed`) + +### 2.1 Source facts + +``` +$ python -c "... load_dex('UnCrackable-Level1.apk') ..." +classes.dex 5528 bytes + magic dex\n035\0 + checksum field 7fb7d9fa + adler32(d[12:]) 7fb7d9fa -> matches + signature field b7fafe72cb521450c4470043caa332da61d1bec7 + sha1(d[32:]) 5253237b04c2d002967bae8ef81daae08550c90c + -> MISMATCH +``` + +Two facts that a reader needs before trusting the tool's output: + +- The **shipped dex's `signature` field is stale** while its `checksum` is correct. `dex_patch_bytes.py` + prints `header before: checksum_ok=True signature_ok=False` and explains it. If the tool had + written the two fields in the wrong order, that starting state would have hidden the bug. +- The class that owns the behaviour is + `Lsg/vantagepoint/uncrackable1/MainActivity;.onCreate(Landroid/os/Bundle;)V`, and the gate is the + first arm of a three-way root `OR`: + +``` +0x9c8 invoke-static Lsg/vantagepoint/a/c;.a()Z +0x9ce move-result +0x9d0 39000e00 if-nez v0 -> 0x9ec <-- the edit site +0x9d4 invoke-static Lsg/vantagepoint/a/c;.b()Z +``` + +`0x9ec` is `const-string "Root detected!"` followed by the private `a(String)` that builds the +blocking dialog. **The branch structure, not the method name, was read before editing** — the edit +target is the arm whose *fall-through* continues into the next check. + +### 2.2 The patch + +```json +[{ + "name": "l1-first-root-check-fallthrough", + "class": "Lsg/vantagepoint/uncrackable1/MainActivity;", + "method": "onCreate", "desc": "(Landroid/os/Bundle;)V", + "match": {"kind": "if-testz", "bytes": "39000e00"}, + "expect_next": {"kind": "invoke", "target_class": "Lsg/vantagepoint/a/c;", + "target_method": "b"}, + "replace": {"kind": "nops"} +}] +``` + +``` +$ python skills/apk-reverse/scripts/dex_patch_bytes.py tools/_work/bench/l1l3/dex/classes.dex \ + --spec tools/_work/bench/l1l3/l1_root.spec.json --dry-run +-- [l1-first-root-check-fallthrough] Lsg/vantagepoint/uncrackable1/MainActivity;.onCreate(...)V + site 0x9d0 0x9d0: 39000e00 if-nez v0 -> 0x9ec + (before) 0x9c8: 710027000000 invoke-static Lsg/vantagepoint/a/c;.a()Z + (before) 0x9ce: 0a00 move-result + -> 0x9d0: 00000000 if-nez + (after) 0x9d4: 710028000000 invoke-static Lsg/vantagepoint/a/c;.b()Z + (after) 0x9da: 0a00 move-result + polarity: expect_next satisfied +== dry run: nothing written +``` + +`expect_next` is what pins the polarity: the instruction after the branch had to be the *next* root +check, so the fall-through is the "continue" side. Applied: + +``` +== header checksum 0x7fb7d9fa -> 0x86e1d928 + signature b7fafe72cb521450 -> eb632328fd1ba03c + self-verify: checksum_ok=True signature_ok=True +== wrote classes.patched.dex (5528 bytes, delta 0) +``` + +**The header order question, answered by byte comparison rather than by the tool's own report** +(`observed`): the diff of the whole file against the source is exactly the two fields plus the patch +site. + +``` +sizes 5528 5528 +diff offsets raw: ['0x8','0xa','0xb','0xc'...'0x1f', '0x9d0','0x9d2'] +site 0x9d0 orig 39000e00 patched 00000000 +8..12 fad9b77f -> 28d9e186 (checksum, bytes 8..12) +12..32 b7fafe72cb521450c4470043caa332da61d1bec7 + -> eb632328fd1ba03c7ebc6df181e0708a761aa2e3 (signature, bytes 12..32) +selfverify checksum True signature True +``` + +Recomputing `zlib.adler32(d[12:])` and `hashlib.sha1(d[32:])` **on the written file** reproduces both +header fields, which is only possible if the signature was computed before the checksum. The patch +owns 4 bytes and the header owns 24; nothing else moved. + +### 2.3 Repack — where the documented pipeline actually fails + +`repack.py`'s default signing route: + +``` +$ python skills/apk-reverse/scripts/repack.py --apk UnCrackable-Level1.apk \ + --dex classes.dex=classes.patched.dex --out out/l1_patched.apk \ + --workdir out --ks out/bench.keystore \ + --keytool ".../jdk-17.0.4.1/bin/keytool.exe" \ + --jarsigner ".../jdk-17.0.4.1/bin/jarsigner.exe" \ + --apksigner E:\tools\bin\apksigner.bat --zipalign E:\tools\bin\zipalign.exe +[zip] - META-INF/CERT.RSA (signature artifact) +[zip] - META-INF/CERT.SF (signature artifact) +[zip] - META-INF/MANIFEST.MF (signature artifact) +[zip] + classes.dex (5528 bytes <- .../classes.patched.dex) +[zip] unsigned written: 60967 bytes +[ks] generated E:\...\out\bench.keystore (alias=apkreverse) +error: signer jar not found: uber-apk-signer.jar + pass --signer-jar , or set APK_SIGNER_JAR. + Any zipalign+apksigner based signer works here. +[exit code: 1] +``` + +Two separate facts, and the second is the one worth recording: + +1. `--signer-jar` defaults to `uber-apk-signer.jar` and no such jar exists here, so the default run + cannot sign. (`ENV.md` predicted this.) +2. **The error message's second line is misleading in this environment.** "Any zipalign+apksigner + based signer works here" is true, but `repack.py` has no code path that *uses* a plain + `zipalign`+`apksigner` pair for signing — `sign_apk()` only ever invokes + `java -jar `. `--apksigner`/`--zipalign` are used for **verification only**. So the + documented Route B works, but only if the caller performs the sign step outside `repack.py`. + The `--no-sign` flag is the supported seam. + +Working route, both builds through the **same** pipeline: + +``` +$ python skills/apk-reverse/scripts/repack.py --apk UnCrackable-Level1.apk \ + --dexdir tools/_work/bench/l1l3/dex_orig \ + --out out/l1_control_unsigned.apk --workdir out/ctrl --no-sign +[zip] unsigned written: 60967 bytes +[zip] resources.arsc STORED offset=0xd4d4 aligned +== alignment gate: resources.arsc STORED and 4-byte aligned (OK) +[result] UNSIGNED (--no-sign): good for inspection, not installable as-is + +$ (same, with --dex classes.dex=classes.patched.dex) -> out/l1_patched_unsigned.apk +[zip] resources.arsc STORED offset=0xd4d4 aligned +== alignment gate: resources.arsc STORED and 4-byte aligned (OK) +``` + +`AndroidManifest.xml` is reported `STORED ... NOT ALIGNED`, which is correct and not a defect: the +4-byte alignment requirement is `resources.arsc` (and uncompressed `lib/*.so`), not the manifest. + +Signing, explicitly, both builds, same key: + +``` +$ E:\tools\bin\zipalign.exe -p -f 4 l1_control_unsigned.apk l1_control_aligned.apk ; rc=0 +$ E:\tools\bin\apksigner.bat sign --ks bench.keystore --ks-pass pass: --key-pass pass: \ + --v1-signing-enabled true --v2-signing-enabled true --v3-signing-enabled true \ + --out l1_control.apk l1_control_aligned.apk ; rc=0 +$ E:\tools\bin\apksigner.bat verify --print-certs --verbose \ + --min-sdk-version 21 --max-sdk-version 34 l1_control.apk +Verifies +Verified using v1 scheme (JAR signing): true +Verified using v2 scheme (APK Signature Scheme v2): true +Verified using v3 scheme (APK Signature Scheme v3): true +Signer #1 certificate SHA-256 digest: ad13912308e0b6ded5d8be7da2868356348655033d1dde402d8f70d452fbf6db +``` + +Identical result for `l1_patched.apk`, with the same certificate digest, so the two builds differ +only in the dex bytes. + +### 2.4 On-device result — the behaviour change, seen on the screen (`observed`) + +Two builds were produced and both were checked, because "the pipeline works" and "the patch changed +the app" are different claims and only the second one is the deliverable. + +**The one-command pipeline, re-run after the repository changed** (this supersedes §2.3's manual +route as the recommended invocation): + +``` +$ python skills/apk-reverse/scripts/repack.py --apk UnCrackable-Level1.apk \ + --dex classes.dex=classes.patched.dex --out out/l1_patched_final.apk \ + --workdir out/fin --ks out/bench.keystore \ + --keytool "C:\Program Files\Java\jdk-17.0.4.1\bin\keytool.exe" \ + --apksigner E:\tools\bin\apksigner.bat --zipalign E:\tools\bin\zipalign.exe +[sign] route: apksigner +[out] ...\out\l1_patched_final.apk (70647 bytes) +== alignment gate: resources.arsc STORED and 4-byte aligned (OK) +Verified using v1 scheme (JAR signing): true +Verified using v2 scheme (APK Signature Scheme v2): true +Verified using v3 scheme (APK Signature Scheme v3): true +[result] OK +local sha256 ad6a51cef4699b73ff052d4e049ca9efd64e9570900d19509f0b7a7c20f87e56 +``` + +The manual `zipalign`+`apksigner` route earlier in this section produced **the same sha256**, so the +two paths agree byte for byte and the result does not depend on which one a reader follows. + +Install through the **root** path (`pm install -r -d`, which bypasses this ROM's streamed-install +interception), and the installed `base.apk` is hashed and compared to the local file so "installed" +cannot be confused with "the previous build is still there": + +``` +control device sha256 e5a9f335155e3e20219d89bc859ae89cc0a82f7135123397c526d94bfca4b3b4 + local sha256 e5a9f335155e3e20219d89bc859ae89cc0a82f7135123397c526d94bfca4b3b4 identical +patched device sha256 ad6a51cef4699b73ff052d4e049ca9efd64e9570900d19509f0b7a7c20f87e56 + local sha256 ad6a51cef4699b73ff052d4e049ca9efd64e9570900d19509f0b7a7c20f87e56 identical +``` + +`am start -W` for both builds: + +``` +LaunchState: COLD TotalTime: 241 ms WaitTime: 249 ms Status: ok (control) +LaunchState: COLD TotalTime: 232 ms WaitTime: 234 ms Status: ok (patched) +``` + +Window focus, sampled four times at 2 s intervals: + +``` +original mCurrentFocus=Window{a5d2ba7 u0 Root detected!} (all four samples) +control mCurrentFocus=Window{c4047e2 u0 Root detected!} (all four samples) +patched mCurrentFocus=Window{8c94fc1 u0 /...MainActivity} (all four samples) +``` + +And the images, which are the evidence that actually counts — +`tools/_work/bench/l1l3/evidence/00_original/00_original_03.png` shows the modal +`Root detected! / This is unacceptable. The app is now going to exit. / OK` over a greyed-out input +screen. The same shot on the control build (`evidence/01_control/*.png`, and again on +`evidence/12_control_final/*.png` from the final one-command build) is byte-comparable to the +original's. `evidence/13_patched_final/13_patched_final.png` — the final patched build — shows the +screen **without any dialog**: input field live with the caret in it, `VERIFY` enabled, and +`mCurrentFocus` on `/...MainActivity`. + +**G4 is therefore cleared in the strong form**: the control is not just "the same APK" — it went +through the identical repack-and-sign pipeline with a byte-identical dex and still shows the old +behaviour. The behaviour change is attributable to the four patched bytes, and it reproduced on a +second, independently built APK from the final pipeline. + +### 2.5 What the app's own check actually is (`observed`, and it contradicts the obvious reading) + +The functional check was probed with a Frida hook rather than inferred from the crypto: + +``` +[a.b] input="8d127684cbc37c17616d806cf50473cc" -> 16 bytes: 8d127684cbc37c17616d806cf50473cc +[lib a.a] key=8d127684cbc37c17616d806cf50473cc + data=e5426215cb5b9a06c3a0b5e6a4bd769a49e8f074f82eff1d95ab7c17147618e7 + -> 17 bytes 492077616e7420746f2062656c69657665 ascii="I want to believe" +[a.a] REAL input="" -> false +``` + +So: a **fixed 16-byte AES key** (`8d12…73cc`, a `const-string` in the dex) decrypts a fixed +base64 blob to `I want to believe`, and the user's input is compared against that plaintext. My first +reading — that the hex string was itself the secret — is **wrong**, and it was falsified by running +it: entering `8d127684cbc37c17616d806cf50473cc` produces `Nope...`. + +`TextUtils.equals` is **not** the comparison site: hooking it captured only dialog-internal +comparisons (`("Nope...", "")`). The comparison lives in the decrypted-bytes path of +`a.a(String)`. + +Two device facts that cost time and are worth inheriting: + +- **The input field cannot be driven on this ROM.** `adb shell input text "I%swant%sto%sbelieve"` + and a per-character `input keyevent` loop both landed `iwantto`: 7 of 17 characters survived and + every space was dropped. Three independent quoting routes gave the same result, so the field + itself rejects what the injection path sends. Reading the field back with `uiautomator dump` after + each attempt is what distinguished "the app said Nope" from "my input never arrived" — the first + two attempts looked like a wrong secret and were not. +- `EditText.setText(String)` is not reachable from outside (only + `setText(CharSequence, TextView$BufferType)` is declared, and `TextView.BufferType` is not exposed + as a property on the class object); an **empty** `EditText` also returns a **null** `Editable`, so + `getText().length()` fails with "not a function" and looks like a bridge limitation rather than an + empty field. + +The success branch was therefore reached through the app's own method with the boolean forced, which +is honest as long as it is labelled — and it is: +`evidence/09_success_branch/09_success_branch.png` shows the app's `Success! / This is the correct +secret.` dialog. That image proves **the patched build reaches and renders the app's own success +branch**; it does not prove the secret was typed and accepted. The check itself is evidenced by the +libfoo/`a.b`/`a.a` log above. + +### 2.6 B1 verdict + +`observed`: the equal-length edit, the header recomputation in the correct order, the STORED+aligned +repack, v1+v2+v3 signing, the root-path install, the device-side hash match, and the on-screen +behaviour change all reproduce. G2 and G4 both clear. + +--- + +## 3. B2 — L3 self-destruct race (`observed` for the tooling, `unverified`/negative for the patch) + +### 3.1 What L3 actually does (read from the dex and the tombstone, not assumed) + +`MainActivity.onCreate` calls `verifyLibs()` **first**, then `init("pizzapizzapizzapizzapizz".getBytes())`, +then an `AsyncTask`, then three root checks, then `isDebuggable()`, then `setContentView` — and +`setContentView` sits **after** the gate: + +``` +0x11a3ce invoke-static Lsg/vantagepoint/util/RootDetection;.checkRoot1()Z +... +0x11a406 sget Lsg/vantagepoint/uncrackable3/MainActivity;.tampered:I +0x11a40a if-eqz v0 -> 0x11a418 ; tampered == 0 -> skip the dialog +0x11a40e const-string "Rooting or tampering detected." +0x11a412 invoke-direct MainActivity;.showDialog(Ljava/lang/String;)V +0x11a418 ... setContentView +``` + +So when the tamper bit is set the app **never renders its UI**; it sits on a blocking dialog whose OK +button is wired to an `onClick` that exits. `verifyLibs()` is the source of that bit: + +- it reads four ABI-keyed expected CRCs out of `resources.arsc` string resources + (`R.string` ids `0x7f0b002b`, `0x7f0b0029`, `0x7f0b0034`, `0x7f0b0035`), +- compares them to the zip entry CRCs of `lib//libfoo.so`, and +- compares `classes.dex`'s zip CRC to the value the native `baz()` returns. + +The native library exports exactly three JNI entries and one terminate routine: + +``` +Java_sg_vantagepoint_uncrackable3_CodeCheck_bar vaddr=0x33a4 size=256 +Java_sg_vantagepoint_uncrackable3_MainActivity_init vaddr=0x3310 size=136 +Java_sg_vantagepoint_uncrackable3_MainActivity_baz vaddr=0x3398 size=12 +_Z7goodbyev vaddr=0x3080 size=24 +_Z4randv vaddr=0xe4c size=36 +``` + +and its strings name the detection inputs: `/proc/self/maps`, `frida`, `xposed`, `ptrace`, +`waitpid`, `getppid`, `pthread_create`, plus two log strings: +`"Tampering detected! Terminating..."` and `"Error opening /proc/self/maps! Terminating..."`. + +`libfoo.so`'s single RX `PT_LOAD` is `off=0x0 vaddr=0x0 filesz=0x4700`, so **file offset == virtual +address == runtime offset from the module base** for everything in `.text`. That is what makes +`hook_patch_only.js`'s `FILE_OFFSET` meaningful here, and it is not true in general — a library whose +text segment has a non-zero `vaddr` would need `p_vaddr + delta`. + +### 3.2 The baseline, and the finding that invalidates the assumed baseline + +`observed`: the **unmodified, pristine APK** on this device shows the tamper dialog: + +``` +$ am start -n /sg.vantagepoint.uncrackable3.MainActivity + 833 ms pid=5377 5413 mCurrentFocus=Window{1d9ee56 u0 Rooting or tampering detected.} + 1638 ms ... mCurrentFocus=Window{1d9ee56 u0 Rooting or tampering detected.} + ... +10712 ms ... mCurrentFocus=Window{1d9ee56 u0 Rooting or tampering detected.} +``` + +and `logcat` shows the CRCs it compared (`V/UnCrackable3`), with `tampered` set. The process stays +alive in this state for at least 25 s: the tamper dialog alone does **not** kill the process, it waits +for the OK button. + +`observed` — the **time-to-death baseline with a frida probe attached**, which is the number the row +needs: + +``` +$ python ttd_attach.py --package --activity sg.vantagepoint.uncrackable3.MainActivity \ + --js l3_read_crc.js --launch-delay 1.5 --seconds 20 +[i] pre-attach main pid=7020 at 1.88s +[i] attached at +0.00s (probe loaded) +attach+ 0.75s main_pid=7020 +... (alive) +attach+ 4.90s main_pid=7020 +attach+ 5.34s main_pid=- +=== logcat === +F/libc ( 7020): Fatal signal 6 (SIGABRT), code -6 (SI_TKILL) in tid 7052 (tg.uncrackable3) +F/DEBUG ( 7203): #01 pc 000000000000308c .../lib/arm64/libfoo.so (goodbye()+12) +F/DEBUG ( 7203): #02 pc 00000000000031ac .../lib/arm64/libfoo.so +I/Zygote( 1007): Process 7020 exited due to signal 6 (Aborted) +``` + +**`TTD(frida attached) = 5.34 s`, and the death is a native `SIGABRT` originating in `goodbye()+12`.** +Note which process dies: `7020` is the main `` process, `7052`/`tg.uncrackable3` is the failing +*thread*. That distinction matters — see §3.4. + +### 3.3 The two scripts, run as shipped (`observed`) + +Neither had ever been executed before this pass. + +``` +$ python skills/apk-reverse/scripts/spawn_patch_detach.py \ + --package --js skills/apk-reverse/scripts/hook_patch_only.js \ + --captures 2 --interval 3 --no-screencap +[i] spawned pid=7442 +[!] PATCH_BYTES is empty -- configure this probe before using it +[MSG] PATCHFAIL +[i] patched=False +[i] detached (memory writes persist, hooks are gone) +=== capture 0 === +7567 00:01 sh -c ps -A -o PID,ETIME,ARGS | grep -i uncrackable3 +7570 00:01 grep -i uncrackable3 +mCurrentFocus=Window{7c35f28 u0 com.oppo.launcher/com.oppo.launcher.Launcher} +``` + +Three defects, all confirmed by later runs: + +1. **`hook_patch_only.js` is a template, and its guard is correct but the guard is the whole + result.** `PATCH_BYTES = []` correctly refuses and reports `PATCHFAIL`, so an unconfigured run + cannot silently claim success. That is good behaviour; it just means the shipped file tests + nothing. +2. **`spawn_patch_detach.py` resumed the target before waiting for `PATCHED`.** The original order + was `resume(pid)` → then wait. That makes the wait meaningless: by the time the probe's + `Process.findModuleByName` poll sees the library, the target has been running. +3. **`--activity` defaults to `None` and the code then did nothing.** After `am force-stop` and a + spawn, omitting `--activity` left the app un-launched, so the capture set was of the launcher and + read exactly like "the app died". Fixed by resolving the launcher through + `cmd package resolve-activity --brief ` when `--activity` is absent. + +**Correction to my own first fix.** I changed the order so the driver held the target **paused** +until `PATCHED`. That cannot work with a module-polling probe, and the run says so: + +``` +[i] probe loaded; target still PAUSED, waiting up to 15.0s for the patch +[5.00s][alive] [10.00s][alive] [15.00s][alive] +[!] the probe did not report PATCHED in 15.0s +[i] resumed pid=7666 +[15.11s][i] libfoo.so base=0x764c0aa000 bytes@0x3080 = fd 7b bf a9 ... +[15.12s][PATCH] now = 00 00 80 52 d6 03 5f c0 ... +``` + +**While the process is frozen its libraries are not mapped**, so the poll can never succeed: 15 s +paused produced nothing, and `libfoo.so base=0x764c0aa000` appeared 0.11 s after resume. The shipped +order was right about *that*; what was wrong was detaching before the write landed. The corrected +driver resumes, then waits for `PATCHED` (reported loudly when it never arrives), then detaches. + +### 3.4 The experiment, and its result + +Sites were chosen from the tombstone and the ELF, not guessed. `goodbyev` at `0x3080`: + +``` +0x3080 a9bf7bfd stp x29, x30, [sp, #-16]! +0x3084 910003fd mov x29, sp +0x3088 321f07e0 +0x308c 97fff725 bl 0x2ff0 <-- the call the tombstone names as goodbye()+12 +0x3090 2a1f03e0 +0x3094 97fff71f bl 0x2ff8 +``` + +and the two helpers it tail-calls are trivial return-value stubs: + +``` +0x2ff0 52801a40 mov w0, #0xd2 0x2ffc d65f03c0 ret +0x2ff8 52800cc0 mov w0, #0x66 0x3000 321f0fe0 ... +``` + +So probe v2 patches the **return values the branch reads** (`0x2ff0`, `0x2ff8`) as well as +`goodbyev`'s entry, each rewritten to `mov w0, #0 ; ret` (+ nops, equal length), with an assertion on +the expected original bytes before each write. + +Patched condition: + +``` +$ python b2_measure.py --js l3_hook_patch_v2.js --tag PATCHED --probe-seconds 25 +[i] spawned pid=10000 (paused) +[i] resumed pid=10000 +[0.04s][i] libfoo.so base=0x764b89d000 size=90112 +[0.04s][i] goodbye entry = fd 7b bf a9 fd 03 00 91 e0 07 1f 32 25 f7 ff 97 +[0.04s][i] goodbye+12 = 25 f7 ff 97 +[0.04s][PATCH] 0x2ff0 (helper A -> return 0) 40 1a 80 52 c0 03 5f d6 -> 00 00 80 52 d6 03 5f c0 +[0.05s][PATCH] 0x2ff8 (helper B -> return 0) c0 0c 80 52 c0 03 5f d6 -> 00 00 80 52 d6 03 5f c0 +[0.05s][PATCH] 0x3080 (goodbyev entry -> no-op) fd 7b bf a9 ... -> 00 00 80 52 d6 03 5f c0 1f 20 03 d5 1f 20 03 d5 +[0.05s][i] sites written=3 skipped=0 +[i] patched=True + 3.22s pid=10000 alive=False +V/UnCrackable3(10000): Tampering detected! Terminating... +F/libc (10000): Fatal signal 6 (SIGABRT), code -6 (SI_TKILL) in tid 10032 (tg.uncrackable3), pid 10000 +F/DEBUG (10036): #01 pc 000000000000308c .../lib/arm64/libfoo.so (goodbye()+12) +I/Zygote( 1007): Process 10000 exited due to signal 6 (Aborted) +[RESULT] tag=PATCHED spawned=10000 ttd_after_resume=3.22 survived=False +``` + +Control condition (identical reads and writes, original bytes written back): + +``` +$ python b2_measure.py --js l3_hook_control_v2.js --tag control_v2 --probe-seconds 22 +[0.04s][CONTROL] 0x2ff0 unchanged (wrote back 40 1a 80 52 c0 03 5f d6) +[0.04s][CONTROL] 0x2ff8 unchanged (wrote back c0 0c 80 52 c0 03 5f d6) +[0.04s][CONTROL] 0x3080 unchanged (wrote back fd 7b bf a9 fd 03 00 91) +[0.04s][i] control sites ok=3 +V/UnCrackable3(9312): Tampering detected! Terminating... +F/DEBUG ( 9348): #01 pc 000000000000308c .../lib/arm64/libfoo.so (goodbye()+12) +I/Zygote( 1007): Process 9312 exited due to signal 6 (Aborted) +``` + +**Both conditions terminate at `goodbye()+12` with `SIGABRT`.** The probes apply the patch +successfully, inside the race window (0.04–0.05 s after resume against a ~3.2 s deadline), and the +target still dies. + +### 3.5 What this means, and what to do instead + +`observed`: +- `spawn_patch_detach.py` **can** deliver an early memory patch: the write landed 0.05 s after resume + and survived the detach. The mechanism the row tests is real. +- `hook_patch_only.js` is a **template with correct guard rails**, not a working probe. +- The patch as configured is **insufficient**: three sites including the terminate routine's entry + and both of its branch inputs do not stop the abort. + +`inferred` (labelled, not executed): the terminate path is not confined to `goodbyev` and its two +stubs. `goodbyev` is 24 bytes and its only two calls are the two stubs; the `bl` at `+12` is the +observed faulting frame, and yet patching both the callee and the caller changed nothing. The most +consistent explanation is that the tampering thread reaches the *same abort* through a second site +that this static pass did not map — the region around `0x2f80–0x3080` is a dense run of +`mov w0, #imm ; ret` pairs (values 0x11, 0x1b, 0x43, 0x5b, 0x67, 0x7b, 0x92, 0xa2, 0xa3, 0xae, 0xcc, +0xd2, 0xe3, 0xe7, 0xe8, 0xed, 0xf6, 0xfd, …), i.e. a jump-table-shaped cluster of tiny accessors +that a proper disassembly would resolve into a call graph, and `rand` (`_Z4randv`, 0xe4c, 36 B) is +part of the same obfuscated scheme. **A real fix needs a full AArch64 disassembly of `.text` +(~18 KB) rather than four hand-picked offsets.** + +`observed` — the route that should be tried first next time: the abort is reachable from a **Java** +layer that is already fully mapped. `verifyLibs()` computes `tampered` and `onCreate` branches on it, +and the dialog's OK button is a plain `DialogInterface.OnClickListener` in `MainActivity$1`. Patching +the dex in that class (`if-eqz v0 -> 0x11a418` → nops, forcing the non-tamper side) removes the +dialog and lets `setContentView` run **without touching native code at all** — and unlike the native +patch it is a route this repository already has measured tooling for (see B1). This was not executed +in this pass; it is the recommended next experiment, not a result. + +`observed` — a limitation the next pass must budget for: on this device the L3 **original** APK +already fails its own CRC self-check, so the tamper state is on before any instrumentation. Whatever +the follow-up tests must therefore be measured against *that* baseline, not against a clean one. + +--- + +## 4. Task C — L2 (not run) + +Not attempted: A and B consumed the pass. `UnCrackable-Level2.apk` is on disk +(`tools/_work/bench/apks/`, 901022 B, sha256 prefix `4c7980ef1f6cc295`) and the `dynamic-frida.md` +spawn route is untested for it. Recorded as **not run**, not as a negative result. + +--- + +## 5. Defects found in the repository's own tooling + +| Where | What | Evidence | Status | +|---|---|---|---| +| `scripts/repack.py` (`sign_apk`) | **Stale finding, re-opened and closed.** At the start of this pass it signed **only** via `java -jar `; `--apksigner`/`--zipalign` were verification-only, so the "any zipalign+apksigner based signer works here" hint had no code path behind it | §2.3 exit 1 with `signer jar not found` | **fixed by another writer mid-pass**; re-measured in §2.4 — `[sign] route: apksigner`, `[result] OK`, byte-identical output | +| `scripts/repack.py` | The `signer jar not found` message is still emitted by the jar-only `sign_apk()` and still claims a zipalign+apksigner route works. When no `--apksigner` was supplied (the first run in §2.3) the message is the *only* guidance a reader gets, and it points at a function that does not exist | §2.3 | documented; the message should name the apksigner route it now has | +| `scripts/spawn_patch_detach.py` | Detached before the probe's write landed, so the write could be missed entirely while the run still reported `patched=False` and looked merely unlucky | §3.3 | **fixed** — resume, wait for `PATCHED`, report loudly, then detach | +| `scripts/spawn_patch_detach.py` | `--activity` absent left the target un-launched after a spawn, so the captures were of the launcher | §3.3 | **fixed** — resolve the launcher via `cmd package resolve-activity` | +| `scripts/spawn_patch_detach.py` | Unbounded `adb`/frida calls; `adb_run()`'s rc is never inspected for the launcher step | §3.3 | partially fixed (`--launch-timeout`, rc surfaced) | +| `scripts/hook_patch_only.js` | Guard rails are correct (`PATCH_BYTES` empty → `PATCHFAIL`); it is a template, and the docs imply it is a runnable probe | §3.3 | documented | +| `tools/_work/ENV.md` | Attributes the B1 signing problem to the apksigner/zipalign route; the blocker was the missing uber-apk-signer jar, and both apksigner and zipalign work | §2.3 | corrected here | +| `tests/benchmark.md` B2 | Describes L3 as a self-destruct race against a pristine baseline; on this device the original APK is already in its tamper state | §3.2 | corrected here | + +--- + +## 6. Files produced by this pass + +Workbench (not part of the skill; `tools/` is git-ignored): + +- `tools/_work/bench/l1l3/l1_root.spec.json`, `dex/classes.patched.dex`, `dex_orig/classes.dex` +- `tools/_work/bench/l1l3/out/l1_control.apk`, `l1_patched.apk`, `l1_control_now.apk`, + `l1_patched_final.apk` (+ unsigned/aligned intermediates) +- `tools/_work/bench/l1l3/l1_probe.py`, `l1_type.py`, `l1_verify_field.py`, `l1_try_secret.py`, + `l1_drive.js`, `l1_probe_full.js`, `attach_probe.py` +- `tools/_work/bench/l1l3/l3_hook_patch_v2.js`, `l3_hook_control_v2.js`, `l3_read_crc.js`, + `elf_l3.py`, `dis_l3.py` +- `tools/_work/bench/l1l3/b2_measure.py`, `ttd.py`, `ttd_attach.py` +- `tools/_work/bench/l1l3/evidence/**` (PNG captures: `00_original`, `01_control`, + `02_patched_nosecret`, `09_success_branch`, `12_control_final`, `13_patched_final`) + +A note on making §2.4 reproducible: `l1_probe.py` reuses the tag as the remote filename, so +`--tag` must be unique per run. Reusing a tag after a failed `adb push` installs the *previous* +remote file, which is exactly how the first attempt at the final patched verification showed +`Root detected!` — the control build's hash, from the control run's leftover remote path. The hash +comparison in the same script is what caught it; a screenshot alone would not have. diff --git a/docs/tool-verification/EXTENSION-extraction-shell-bench.md b/docs/tool-verification/EXTENSION-extraction-shell-bench.md new file mode 100644 index 0000000..ce67e8b --- /dev/null +++ b/docs/tool-verification/EXTENSION-extraction-shell-bench.md @@ -0,0 +1,538 @@ +# Extension verification — extraction shells and the Dex-VMP boundary (bench rows B3 / B4) + +Covers benchmark rows **B3** (extraction-shell skeleton detection) and **B4** (Dex-VMP +boundary: refuse-to-repack and downgrade the deliverable). Strength labels are this +directory's own three tiers: **observed** (an exact command and its output are quoted +here), **inferred** (follows from observed facts; the step was not executed), +**unverified** (assumed or externally reported, not reproduced here). + +Everything below was produced on this repository's Windows 11 host with the rootless +PC toolchain in `tools/_work/ENV.md`; the device results in §4 name a +`` and use `` placeholders for the target package. + +## 0. Headline + +| Question | Answer | Strength | +|---|---|---| +| Does `stub%` separate a skeleton copy from a genuine dex? | **Only when the shell writes bare `return-void`.** Three other measured emptying shapes score 0.0 %–2.3 %, at or below the *untouched* control (1.9 %) | observed | +| Was the pre-existing `trivial_ratio` detector correct? | **No — two defects**, one of which inverted the ranking and named a fully-nop-wiped skeleton the "most likely original" | observed | +| Is `repos/CyReverse`'s shipped "extraction shell" asset usable as a shell fixture? | **No.** The pair differs by six bytes and one body; neither is a shell | observed | +| Is `DexPatcher/assets/ezAndroid.apk` a Dex VMP? | **No.** It is a JNI-sinking target: 7 `native` declarations, an exported `Java_*` for each, 34,566 clean dalvik instruction decodes, zero structural errors | observed | +| Can a static dex probe decide VMP vs not? | **It can decide "not dex-VMP"** when the bodies decode cleanly and each native declaration has an export. It **cannot** decide whether the native island performs runtime hardening | observed (negative half) / unverified (positive half) | +| Does `dex_mem_scan.py` find and cut a dex out of a blob? | **Yes** — exact offset, no false hits in 2 MiB of noise, image accepted by the judge | observed | +| Does it reach a dex in a live process's anonymous mapping? | **Not measured here.** The fixture is built and the commands staged; the device was held by another row throughout | unverified | + +One claim in this pass was **retracted**: §3.4 records a Dex-VMP verdict drawn from a +desynchronised decoder and the measurement that killed it. + +--- + +## 1. B3 — the shipped CyReverse "extraction shell" assets are a 6-byte difference, not a shell + +Source: `repos/CyReverse/app/src/main/assets/{classes3.dex, classes3_extracted.dex}` +(12,536 B each). + +``` +$ python skills/apk-reverse/scripts/dex_dump_validate.py tools/_work/bench/unpack/b3 +== dex dump validation: 2 file(s), 2 parse, 0 rejected == +name size sha256[:12] ver cksum sig class noco code stub% +cyrev-extracted.dex 12536 c2d7345e955a 037 ok BAD 4 0 20 10.0% +cyrev-skeleton.dex 12536 ba9d5b120a3a 037 ok ok 4 0 20 10.0% +``` + +A byte diff of the pair (`tools/_work/bench/unpack/b3/analyze_pair.py`) locates every +difference: + +``` +$ python analyze_pair.py +len A=12536 len B=12536 +first diff offset = 8 (0x8) +diff byte count = 6 +diff ranges: + 0x8 .. 0x8 (1 B) <- dex checksum (header offset 8) + 0xa .. 0xb (2 B) <- dex signature, first half (offset 12 is 4 B wide) + 0x121c .. 0x121c (1 B) <- INSNS of PluginClass->getString, units 0 and 2 + 0x121e .. 0x121e (1 B) + 0x1220 .. 0x1220 (1 B) +``` + +Per-method census of the pair shows the "extracted" copy is the *degraded* one: + +``` +== cyrev-skeleton.dex == Lcom/cyrus/example/plugin/PluginClass;->getString insns=3 units=001a 0087 0011 +== cyrev-extracted.dex == Lcom/cyrus/example/plugin/PluginClass;->getString insns=3 units=0000 0000 0000 +``` + +`const-string v0, "…"; return-object v0` was overwritten with three `0x0000` units in +exactly one of the twenty bodies, and the header was re-fixed. The shipped +`classes3_extracted.dex` is **not** an extraction shell and **not** a recovered dex: +it is a single-body demonstration of body clearing, with 19 of 20 bodies identical to +the skeleton. Neither file can test extraction-shell routing — 10.0 % `stub%` on both +sides is a coincidence of the same body set, not a signal. + +**Consequence for the benchmark row:** B3 cannot be run against this asset pair. The +skeleton measurement below had to be built from a real application dex instead. + +## 2. B3 — measured `stub%` across every emptying shape, with a zero-change control + +Fixture builder: `tools/_work/bench/unpack/b3/make_skeleton.py` derives each variant +from a real 982,480-byte application dex (`src-r2pay.dex`, extracted from +`tools/_work/bench/apks/r2pay-v1.0.apk`, sha256 `c89ff2364879…`). Every variant is +**byte-length preserving**: only the `insns` array of a `code_item` is rewritten and +the tail is nop-filled, so offsets, class table and string table are untouched and the +header is re-fixed with `dexutil.fix_dex_header` (signature first, checksum last). + +``` +$ python make_skeleton.py src-r2pay.dex skel +source: src-r2pay.dex + ctrl.dex fraction=0.00 mode=return emptied=0/5061 + skel-25.dex fraction=0.25 mode=return emptied=1265/5061 + skel-50.dex fraction=0.50 mode=return emptied=2531/5061 + skel-75.dex fraction=0.75 mode=return emptied=3796/5061 + skel-all.dex fraction=1.00 mode=return emptied=5061/5061 + pruned.dex fraction=1.00 mode=pruned emptied=5061/5061 + nopfilled.dex fraction=1.00 mode=nop emptied=5061/5061 + allstub.dex fraction=1.00 mode=allstub emptied=5061/5061 + throwstub.dex fraction=1.00 mode=throw emptied=4134/5061 +``` + +Verdicts from `dex_dump_validate.py` over the whole set — the decisive matrix: + +``` +$ python skills/apk-reverse/scripts/dex_dump_validate.py tools/_work/bench/unpack/b3/skelall +name size sha256[:12] ver cksum sig class noco code stub% erased% emptied% +allstub.dex 982480 c8d7ca8fb163 035 ok ok 647 250 5061 100.0% 0.0% 100.0% +ctrl.dex 982480 1b3a97896b40 035 ok ok 647 250 5061 1.9% 0.0% 1.9% +nopfilled.dex 982480 de92dcea4eaf 035 ok ok 647 250 5061 0.0% 100.0% 100.0% +pruned.dex 982480 58d7fa3dce48 035 ok ok 647 250 5061 93.2% 0.0% 93.2% +skel-25.dex 982480 63346d9b41f5 035 ok ok 647 250 5061 1.7% 0.0% 1.7% +skel-50.dex 982480 d84601b7aa47 035 ok ok 647 250 5061 1.7% 0.0% 1.7% +skel-75.dex 982480 64114285656d 035 ok ok 647 250 5061 2.1% 0.0% 2.1% +skel-all.dex 982480 6345daef0a3f 035 ok ok 647 250 5061 2.3% 0.0% 2.3% +throwstub.dex 982480 cab8fd51a743 035 ok ok 647 250 5061 1.9% 0.0% 1.9% +``` + +Read the `mode=` column against `stub%`: + +| Emptying shape (what the shell leaves behind) | `stub%` | Separated from `ctrl` (1.9 %)? | +|---|---|---| +| bare `return-void` in every body (`allstub`) | **100.0 %** | yes, by 98 points | +| body truncated to `const/4 v0,#0; return v0` + nop fill (`pruned`) | 93.2 % | yes | +| nop fill, no return at all (`nopfilled`) | **0.0 %** | **no — scores *below* the untouched control** | +| `new RuntimeException; throw` stub (`throwstub`) | 1.9 % | **no — indistinguishable from control** | +| 25 % / 50 % / 75 % of bodies emptied with a legal minimal body | 1.7 / 1.7 / 2.1 % | **no** | + +Three conclusions, all observed: + +1. **The threshold is not a number, it is a shape.** `stub%` is bimodal — it is either + ≈ control (blind) or ≈ 100 % (sighted). There is no intermediate band where a + threshold could sit, so the "tens of percent" calibration the reference file + describes has no measured support. +2. **Partial extraction is invisible.** `skel-75.dex` — three quarters of all bodies + emptied — reports 2.1 % against a 1.9 % control, a 0.2-point difference inside the + sampling noise of what a real app's own trivial bodies contribute. +3. **The two shapes that defeat it are the two that real shells prefer.** + `throw new RuntimeException` stubs are the classic 360/legu family shape, and a + nop-wiped body is what a shell writes when it clears the slot before decrypting. + +The false-negative on `nopfilled` was a **defect, not a boundary**, and is fixed in §3. +The `throwstub` blind spot is a **boundary** and is documented as one. + +## 3. B3 — defects found in `dex_dump_validate.py`, and the fix + +### 3.1 Defect 1: a fully nop-filled body was classified `real` + +`classify_body` skipped every `0x0000` unit and then fell through to `return "real"` +when nothing else was found. Measured consequence: `nopfilled.dex` — **all 5,061 +bodies wiped** — reported `stub% = 0.0`, and the ranking named it the most likely +original: + +``` +$ python skills/apk-reverse/scripts/dex_dump_validate.py …/skel/{nopfilled,ctrl}.dex +ranking (most likely original first): + 1. nopfilled.dex stub%=0.0 cksum=ok sig=ok classes=647 + … +verdict: nopfilled.dex is the most likely original (1 copy(ies) in this set) +``` + +An agent routing on that line patches a wiped skeleton. That is the failure the file +exists to prevent, so the defect is on the wrong side of "the tool still produces +output". + +**Fix.** `classify_body` gained two classes and a wider body scan: + +- `erased` — `insns_size > 0` with every unit `0x0000`. A real compiler never emits a + whole body of nops, so this is skeleton evidence. +- `minimal` — a body truncated to `const/4 vR,#0; return vR` (legal, and the shape the + detector is blind to by design). Reported separately so the blind spot is visible + rather than hidden inside `real`. +- The early-exit `meaningful > 1 → real` became `meaningful > 2`, so the two-unit + minimal body reaches its own verdict instead of being mislabelled. + +`trivial_ratio` keeps its original definition so previously recorded numbers stay +comparable; `emptied_ratio = (stub + erased) / with_code` is the new skeleton signal, +and `rank_key` sorts on it **low-first** with `bodies_erased` as a tiebreak. + +### 3.2 Defect 2 (retracted in-flight): `0xNN0E` is not malformed + +An intermediate version of the fix classified a lone `0x020E` unit as `malformed`, +reading a non-zero high byte on a `0x0E` opcode as an illegal `return-void`. The +control dump proved otherwise — 82 of the untouched dex's own bodies are exactly that +shape — and `const/4 v0, #0; return v0` is a legal minimal non-void body. The class was +renamed `minimal`, its comment corrected, and the fixture extended with `pruned.dex` +to measure it deliberately (93.2 %, `stub%` sighted). Recorded here because the +mislabel would have taught the next reader that ordinary short methods are damage. + +### 3.3 Post-fix behaviour, and no regression on the recorded fixture + +`nopfilled.dex` is now ranked last and flagged: + +``` + 8. nopfilled.dex stub%=0.0 erased%=100.0 cksum=ok sig=ok classes=647 <-- SKELETON: emptied%=100, extraction-shell shape + 9. … +verdict: skel-25.dex is the most likely original (1 copy(ies) in this set) +``` + +The pre-existing acceptance fixture from `EXTENSION-unpacking.md` §2 still produces +its recorded verdict, with only the two new columns appended: + +``` +$ python skills/apk-reverse/scripts/dex_dump_validate.py tools/_work/t2-unpack --find 'Lcom/stub/' +== dex dump validation: 7 file(s), 4 parse, 3 rejected == +01_original_shell.dex 8923752 bcd1c0488e86 035 ok BAD 4 160 29 3.4% 0.0% 3.4% +02_duplicate.dex 8923752 bcd1c0488e86 035 ok BAD 4 160 29 3.4% 0.0% 3.4% +03_broken_magic.dex 8923752 33eee53b1317 ? REJECTED: magic=b'XXXX' +04_tampered_body.dex 8923752 b89bc8cc73d9 035 BAD BAD 4 160 29 3.4% 0.0% 3.4% +05_truncated.dex 65536 e6687390a663 035 REJECTED: file_size=8923752 actual=65536 +06_extracted_skeleton.dex 8923752 85e56fed1d56 035 ok ok 4 160 29 100.0% 0.0% 100.0% +07_tiny_garbage.dex 16 be45cb2605bf ? REJECTED: too small (16 B) to carry a dex header +verdict: 01_original_shell.dex is the most likely original (2 copy(ies) in this set) +``` + +Same winner, same three rejections, same dedupe group. The change is additive on that +fixture and corrective on the new one. + +### 3.3b The ranking was still pointing at a modified image (found by independent review) + +The first fix above added the `emptied_ratio` signal — and then **kept ranking on a +metric this same pass had just proved insensitive to skeletons**. Reproduced by +leading the review on the variant set: + +``` +$ python skills/apk-reverse/scripts/dex_dump_validate.py tools/_work/bench/unpack/b3/skel +ranking (most likely original first): + 1. skel-25.dex stub%=1.7 erased%=0.0 cksum=ok sig=ok classes=647 + 2. skel-50.dex stub%=1.7 erased%=0.0 cksum=ok sig=ok classes=647 + 3. ctrl.dex stub%=1.9 erased%=0.0 cksum=ok sig=ok classes=647 + … +verdict: skel-25.dex is the most likely original (1 copy(ies) in this set) +``` + +`skel-25.dex` — a dex with **25 % of its bodies removed** — was named the most likely +original, and the genuine untouched control ranked third. The cause is the bimodality +of §2: inside the low band the ratio is not a signal, and 1.7 % versus 1.9 % is noise +(the extracted bodies are replaced by `const/4 + return`, which *lowers* the +`return-void` count relative to the control). An agent routing on that line adopts a +modified image as its baseline, and every later diff is then built on the wrong +artifact — the same class of failure as the defect in §3.1, reached by a different +path. + +**Fix.** `rank_key` became three-level, with an explicit eviction level so that either +signal can disqualify an image instead of the two merely tying: + +1. `emptied_ratio >= 0.5` sorts last. Nothing mostly emptied is a candidate however + else it scores. This level exists because the count below cannot see the + `return-void`-in-every-body skeleton: that image has `minimal = erased = 0`, so a + pure-count key ranked it **first** — measured, and it is why this level was added + rather than the count being promoted alone. +2. `bodies_minimal + bodies_erased` low-first. This orders the low band, where the + count is monotone: 82 untouched, then 1299 / 2524 / 3736 / 4946 at 25 / 50 / 75 / + 100 %. +3. `emptied_ratio` low-first, then header verdicts, then name. + +Result on the same command: + +``` +ranking (most likely original first): + 1. ctrl.dex stub%=1.9 erased%=0.0 min/erased=82 cksum=ok sig=ok + 2. only.dex stub%=1.9 erased%=0.0 min/erased=82 cksum=ok sig=ok + 3. throwstub.dex stub%=1.9 erased%=0.0 min/erased=82 cksum=ok sig=ok + 4. skel-25.dex stub%=1.7 erased%=0.0 min/erased=1299 cksum=ok sig=ok + 5. skel-50.dex stub%=1.7 erased%=0.0 min/erased=2524 cksum=ok sig=ok + 6. skel-75.dex stub%=2.1 erased%=0.0 min/erased=3736 cksum=ok sig=ok + 7. skel-all.dex stub%=2.3 erased%=0.0 min/erased=4946 cksum=ok sig=ok + 8. allstub.dex stub%=100.0 erased%=0.0 min/erased=0 cksum=ok sig=ok <-- SKELETON + 9. pruned.dex stub%=93.2 erased%=0.0 min/erased=344 cksum=ok sig=ok <-- SKELETON + 10. nopfilled.dex stub%=0.0 erased%=100.0 min/erased=5061 cksum=ok sig=ok <-- SKELETON +verdict: ctrl.dex is the best-supported candidate for the original (1 copy(ies) in this set) +``` + +The control wins and every skeleton sorts behind it. The verdict line now states the +key and its limits instead of declaring a winner outright, and phrases the result as +"best-supported candidate" with an instruction to confirm before patching. `--trim`, +`--json` and `--find` are unchanged (verified: the recorded fixture still returns +`01_original_shell.dex` first, `--json` only gained keys). Note the residual limit the +report prints rather than hides: **`throwstub.dex` still ranks third**, because no body +statistic here sees a `throw`-stub skeleton. That is a boundary, not a bug — it is +listed in §7 with the reproduction. + +### 3.4 A retracted Dex-VMP verdict — the measurement that killed it + +An intermediate workbench script of mine +(`tools/_work/bench/unpack/b4/vmp_probe.py`, `vmp_methods.py`) decoded instruction +widths from a hand-written table and reported: + +``` +code_items=22424 desync=1393 (6.21%) bad_op=0xf8 / 0xef / 0xea / … +``` + +I read that as "bodies present but not decodable → private opcodes → Dex VMP". **The +reading was wrong, and the error was my table's.** The official decoder settles it: + +``` +$ E:\tools\android-14\dexdump.exe -d tools/_work/bench/unpack/b4/orig/classes.dex +Processing 'orig\classes.dex'... +Opened 'orig\classes.dex', DEX version '035' + … 943,223 lines … +instruction lines (hex:...) : 348566 +code items resolved ('insns size :') : 22424 +code '(none)' : 1179 +``` + +One of the methods my script called a desync decodes cleanly under `dexdump`: + +``` +--- androidx.appcompat.app.AlertController$ButtonHandler.:(Landroid/content/DialogInterface;)V + insns size : 11 16-bit code units +1fd564: 7010 9b05 0100 |0000: invoke-direct {v1}, Landroid/os/Handler;.:()V +1fd56a: 2200 040e |0003: new-instance v0, Ljava/lang/ref/WeakReference; +1fd56e: 7020 fb76 2000 |0005: invoke-direct {v0, v2}, Ljava/lang/ref/WeakReference;.:(Ljava/lang/Object;)V +1fd574: 5b10 ed08 |0008: iput-object v0, v1, …$ButtonHandler;.mDialog:…; +1fd578: 0e00 |000a: return-void +``` + +My script consumed 9 of 11 units and stopped on `0xed`; `0xed` is a legal +`invoke-*/range` opcode. **The lesson is the one the reference file already states and +this pass then failed to apply: a hand-rolled opcode table is not a VMP detector, and +"my decoder desynchronised" is a statement about my decoder.** Any future +"rare-opcode density" metric must be validated against `dexdump` on the same file +before it is allowed to inform a verdict. + +The corrected count of bodies that fail to decode as dalvik in this dex is **0**. +Correction confirmed by an independent path: `dexdump` emits 34,566 instruction lines +and reports no structural error for any of the 22,424 code items. + +## 4. B4 — the Dex-VMP judgement chain, re-opened + +### 4.1 Static findings (`observed`) + +Target: `repos/DexPatcher/assets/ezAndroid.apk`, 2,984,841 B, sha256 +`76e3fa8119cf8b7c…`, package `` (the `ezAndroid` CTF sample shipped in DexPatcher). + +``` +$ python skills/apk-reverse/scripts/dex_dump_validate.py …/b4/orig/classes.dex +classes=2905 methods_with_code=22424 stub%=1.9 cksum=ok sig=BAD +``` + +- **Native declarations: 7**, all in the app's own obfuscated classes, each with no + code (`code : (none)`), enumerated from `dexdump`: + `Lcom/example/ezandroid/C;->cf(Ljava/lang/String;I)Z`, + `Lcom/example/ezandroid/MU;->GMS(Ljava/lang/String;)Ljava/lang/String;`, + `Lcom/example/ezandroid/S;->Stringfjni()Ljava/lang/String;`, + `Lcom/example/ezandroid/S;->onCreate`, `Lcom/example/ezandroid/S2;->onCreate`, + `Lcom/example/ezandroid/S$1;->onClick`, `Lcom/example/ezandroid/S2$1;->onClick`. +- **Two JNI exports** in `lib/arm64-v8a/libezandroid.so` (206,904 B, aarch64): + `Java_com_example_ezandroid_S_Stringfjni` @ `0xea6c` (184 B) and + `Java_com_example_ezandroid_MainActivity_stringFromJNI` @ `0xe91c` (184 B). + Both are thin JNI wrappers that construct a `std::string` and dispatch through a + virtual call — the body of the check is behind the vtable, not in the wrapper. +- **No `JNI_OnLoad` and no `RegisterNatives` symbol** anywhere in the library; the + string `JNI_OnLoad` does not occur in the file at all. +- **`assets/` holds six small opaque files** (`0OooOO` 118 B, `O0ooOO` 126 B, + `oo0Ooo` 86 B, `ooO0oo` 132 B, `OOoo0O` 398 B, `OOooO0` 260 B). Each begins with a + three-unit header (`05 00 38 00 02 00 …`) followed by 2-byte-aligned units. +- **No dex string references those six filenames.** Grep over the full 943,223-line + `dexdump` output for `0OooOO|O0ooOO|oo0Ooo|ooO0oo|OOoo0O|OOooO0|assets/` returns + **0 hits**. + +### 4.2 Verdict: JNI sinking, not Dex VMP (`observed`) + +The reference file's Dex-VMP test is "the dump's method bodies are intact but decode +as private opcodes — rare-opcode density off the charts, unknown opcode slots". On this +target **zero method bodies fail to decode**, and every `native` declaration has a +matching exported `Java_*`. The dex is not carrying an encoded program; the code left +the dex for the `.so`, which is `java2c-and-jni-sinking.md`'s territory, not this +file's. The two must not be conflated: an agent that reads "bodies decode as nonsense" +here will go hunting a private disassembler for code that is ordinary ARM64 in a +206 KB library. + +The six `assets/` files are a separate, **unresolved** observation: they are not +referenced by any dex string, so they cannot be load-bearing at the dex layer. Whether +the native island reads them by constructing the path at runtime, and what they +contain, is **unverified** here — it needs a dynamic run (§4.3), not more static +searching. + +### 4.3 What static analysis structurally cannot see (the B4 boundary) + +`java2c`'s independent pass reached the same classification on this sample and recorded +that the hardening takes effect **at runtime only**. That is the general shape, and it +is the half of B4 that no dex probe can settle: + +| Question | Answerable statically? | Why | +|---|---|---| +| Are the bodies real dalvik or private opcodes? | yes | `dexdump` decodes them or does not; measured 0 failures here | +| Did the code leave the dex for native? | yes | `native` access flags + `Java_*` exports / `JNI_OnLoad` | +| Does the native island check the signature / root / debugger? | **no** | the logic is ARM64 behind a vtable; the check may not exist in any dex | +| Do those `assets/` blobs get decrypted and used? | **no** | the path may be assembled at runtime; grep for the literal names finds nothing | +| Does a repack actually fail, and why? | **no** | needs the repack installed and run against a control (G4) | + +**Rule this pass proposes for the reference file:** a static dex probe may return +"this is *not* dex-VMP" when the bodies decode cleanly and every native declaration has +an export. It may **not** return "this is not hardened" from the same evidence — +those are different claims, and the second one is the misdiagnosis +`java2c-and-jni-sinking.md` was written to stop. + +### 4.4 Repack control — `observed` up to the install, `unverified` beyond + +G4 requires a zero-change control through the same pipeline. Built with the +repository's own repacker against an explicit toolchain (this host has no uber-apk-signer; +`keytool` lives in the JDK that `ENV.md` does not list): + +``` +$ python skills/apk-reverse/scripts/repack.py \ + --apk tools/_work/bench/unpack/b4/ezAndroid.apk \ + --out tools/_work/bench/unpack/b4/ez-ctrl.apk \ + --ks tools/_work/bench/unpack/b4/ctrl.jks --signer apksigner \ + --apksigner E:\tools\android-14\apksigner.bat \ + --zipalign E:\tools\android-14\zipalign.exe \ + --keytool "C:\Program Files\Java\jdk-17.0.4.1\bin\keytool.exe" +[in ] E:\apk-reverse\tools\_work\bench\unpack\b4\ezAndroid.apk (2984841 bytes) +[zip] - META-INF/1N0R.SF / 1N0R.RSA / MANIFEST.MF (signature artifact) +[zip] unsigned written: 2828919 bytes +[zip] AndroidManifest.xml STORED offset=0x31 +[zip] classes.dex 3960320 bytes method=8 crc=037e9cb3 +[zip] resources.arsc STORED offset=0x202f14 aligned +[sign] route: apksigner +[verify:zipalign] rc=0 +[verify:v2block] present +[cert] SHA1 C9:89:C5:75:A4:83:AE:E0:A6:4B:F4:F6:0D:E3:D2:73:C2:8B:65:69 +[result] OK +``` + +Zero-change repack, v2 block present, alignment verified, `classes.dex` carried through +at its original CRC (`037e9cb3`). **Install and launch on `` are `unverified` +in this record**: the shared device was held by a parallel benchmark row (`B1 L1 repack +control + patched install`, `B2 L3 frida`) for the whole window available to this pass. +No claim is made here about whether the control launches; the artifact and the command +above are the reproducible half, and the device half is the open item. + +Consequently **"repacking is truly refused on this target" is `inferred`, not +`observed`** — it follows from the native island being the only home of the app's own +logic (`Stringfjni`, `cf`, `GMS` are all native declarations), which makes a +dex-level patch unable to reach the behaviour, but no device run confirms a refusal. +The honest deliverable form for this target is the G1 row **annotated analysis report +with a stated boundary**, with `lsposed_scaffold.py` / `frida_rpc_serve.py` named as +the routes *if* a device run shows the build is refused. + +## 5. `dex_mem_scan.py` — the cut half is measured, the in-process half is not + +The script's job splits cleanly into *find-and-cut* (locate `dex\n03x` in a blob and +write exactly the declared `file_size`) and *reach the blob at all* (do it against a +live process's anonymous mappings through `/proc//mem`). Only the first half was +measurable in this pass, and it was measured with a synthetic high-entropy capture +built to have the shape a heap buffer presents: + +``` +$ python … # 2 MiB of random bytes with a 4,124-byte dex written at offset 0x9a000 +fixture dex: 4124 bytes, magic=b'dex\n039\x00' +capture written: …/memscan/anonregion.bin 2097152 bytes dex at 0x9a000 + +$ python skills/apk-reverse/scripts/dex_mem_scan.py …/anonregion.bin --dump …/scanout +== dex memory scan: 1 capture(s), 1 hit(s) == +capture offset ver file_size classes state +anonregion.bin 0x9a000 039 4124 1 ok + wrote …/scanout/anonregion+0x0009a000.dex + +$ python skills/apk-reverse/scripts/dex_dump_validate.py …/scanout +name size sha256[:12] ver cksum sig class noco code stub% +anonregion+0x0009a000.dex 4124 d3dedcd2b75e 039 ok ok 1 0 5 0.0% +``` + +Exact offset, zero false hits in 2 MiB of noise, the extracted image byte-identical +by construction and accepted by the judge including its checksum and signature. That +is the **whole find-and-cut path, observed** — the halves compose correctly. + +**What is not measured.** The live-process half: the fixture that would exercise it is +built and staged (§6), but the shared device was held by a parallel benchmark row for +the entire window this pass had, so `dex_mem_scan.py` has still never been run against +a real anonymous mapping here. The one related number that exists is from the earlier +root-dump pass, and it is a **zero**: 142 anonymous regions scanned, no hits, against +17 dex images found in *named* ART mappings with no search at all. A zero-hit scan is +weak evidence for the reasons listed in the script's own header, and the honest +summary is "the finder works on a shape that looks right; its in-process reach is +unverified". + +To close it (all commands staged, none run in this pass): + +``` +adb push tools/_work/bench/unpack/memscan/out2/memfixture.jar /data/local/tmp/ +adb push tools/_work/bench/unpack/b3/src-r2pay.dex /data/local/tmp/payload.dex +adb shell 'su -c "CLASSPATH=/data/local/tmp/memfixture.jar app_process /system/bin \ + com.revprobe.memfixture.Main /data/local/tmp/payload.dex"' & +# the fixture prints its own pid, truncates the on-disk payload so the heap copy is the +# only remaining source, and writes /data/local/tmp/memfixture.maps +adb shell 'su -c "grep -c . /proc//maps"' # confirm cheap maps first +adb shell 'su -c "grep -v \"/\" /proc//maps"' # the anonymous set to export +``` + +Note the fixture deliberately truncates its source file: a scan that finds the dex +after that point cannot have found a file copy, which is what makes the result +attributable to memory rather than to disk. + +## 6. Not done — and why + +- **No device run of the B4 control build.** The shared device was locked by another + benchmark row for the whole window; `devlock.py acquire unpack` timed out twice + (900 s and 1,800 s waits). Install, launch, time-to-death and refusal/success are + therefore `unverified`. +- **No in-process run of `dex_mem_scan.py`.** The fixture is built and staged + (`tools/_work/bench/unpack/memscan/`, §5), the arm64 `app_process` route is chosen + because this host has **no `android.jar`** and therefore cannot compile a real APK + at all, and the closing commands are listed — but the device was never available, so + the anonymous-mapping hit rate remains unmeasured. The only related measurement is + the earlier pass's **0 of 142** anonymous regions. +- **Whether the `assets/` blobs are live.** Needs a dynamic run (read the files out of + the process after the native island runs, or hook the path construction). +- **No end-to-end FART splice.** The reference file's FART loop needs a live + extraction-shell target; nothing in `repos/` is one (§1), and building an actively + resisting shell is out of scope for this pass. +- **A real APK build on this host.** `E:\tools\android-14` ships `d8`, `aapt2`, + `apksigner` and `zipalign` but **no `android.jar`**, and no `android.jar` exists + anywhere on the machine (searched `E:\tools`, `C:\Program Files`, `E:\Android`). Any + `javac` against `android.*` fails with "package android.os does not exist". APKs can + still be *repacked* here (`repack.py` does not recompile), which is why §4.4 works + and the self-built-APK route in the task brief does not. + +## 7. Reproduction + +``` +python tools/_work/bench/unpack/b3/make_skeleton.py tools/_work/bench/unpack/b3/src-r2pay.dex tools/_work/bench/unpack/b3/skel +python tools/_work/bench/unpack/b3/analyze_pair.py +python tools/_work/bench/unpack/b3/crosscheck.py tools/_work/bench/unpack/b3/skel/*.dex +python skills/apk-reverse/scripts/dex_dump_validate.py tools/_work/bench/unpack/b3/skelall +python skills/apk-reverse/scripts/dex_dump_validate.py tools/_work/t2-unpack --find 'Lcom/stub/' +python tools/_work/bench/unpack/b4/vmp_boundary.py tools/_work/bench/unpack/b4/orig/classes.dex +E:\tools\android-14\dexdump.exe -d tools/_work/bench/unpack/b4/orig/classes.dex +python skills/apk-reverse/scripts/dex_mem_scan.py tools/_work/bench/unpack/memscan/anonregion.bin --dump tools/_work/bench/unpack/memscan/scanout +``` + +Raw captures live beside the fixtures: `evidence/prefix-validate.txt` (before the fix, +including the inverted ranking) and `evidence/postfix-validate.txt` (after). The +`memscan/` tree carries the staged (unrun) in-process fixture: `plain/com/revprobe/memfixture/Main.java` +(no-`android.jar` form), `build.ps1` (the APK build that cannot complete here, kept as +the record of *why*), and `out2/{classes.dex,memfixture.jar}`. + +To re-verify the fix from scratch, the two commands that matter are +`dex_dump_validate.py …/skelall` (expect `nopfilled.dex` flagged, not ranked first) and +`dex_dump_validate.py tools/_work/t2-unpack --find 'Lcom/stub/'` (expect +`01_original_shell.dex` still the winner). diff --git a/docs/tool-verification/EXTENSION-java2c.md b/docs/tool-verification/EXTENSION-java2c.md new file mode 100644 index 0000000..1b24f9e --- /dev/null +++ b/docs/tool-verification/EXTENSION-java2c.md @@ -0,0 +1,316 @@ +# Extension verification — Java2C identification and routing + +Covers the 2026-09 extension pass that added `references/java2c-and-jni-sinking.md` and +`scripts/java2c_probe.py`. Strength labels as defined in this directory's README: +**observed** (exact command + output here), **inferred** (follows from observed facts, +step not executed), **unverified** (assumed or externally reported, not independently +confirmed). + +Reference environment for this pass: Windows 11 host, Python 3.14.0, JDK 17, +build-tools (`d8`/`aapt2`) from a local Android 14 SDK, `gcc` from MinGW only. +**No Android NDK, no host `clang`, no `wsl` (HCS unavailable), and no device-side +`clang`/Termux.** A rooted Android 11 / API 30 / arm64-v8a `` was attached and +reachable but **was not used at all** in this pass: every result below is static. No +package name, project name or device serial appears in this record. + +Samples are identified by hash rather than by name. They are public MASTG training apps +(a no-native baseline, and two carrying a JNI sink), one VMP-labelled app taken from +`tools/_work/bench/repos/`, one real third-party aarch64 ELF already on the host, and one +locally synthesised Java2C-shaped dex. + +--- + +## 1. The compile chain could not be closed — observed + +The intended route was to build a real Java2C `.so` with `amimo/dcc` and probe it. `dcc` +is present at `tools/_work/bench/repos/dcc/` and its Python dependencies are satisfied. +The build stops at the C compiler, and every alternative was tried exactly once before +being written off (two-strike rule — no third variant was attempted): + +``` +$ which gcc clang +gcc.exe C:\MinGW\bin\gcc.exe # MinGW targets PE/COFF, not ELF +gcc/clang for Android: none + +$ wsl -l -q ; wsl -e bash -lc "which gcc" +Ubuntu-E +(empty) -> exit 1 with: Wsl/Service/CreateInstance/CreateVm/HCS/HCS_E_SERVICE_NOT_AVAILABLE + +$ adb shell "which clang clang++ gcc ; ls /data/data/com.termux" +(no output for which) +ls: /data/data/com.termux: No such file or directory +``` + +Related **observed** fact: the `dcc` checkout ships **no prebuilt binary at all** — a +recursive search for `*.so`, `*.apk`, `*.a` over the whole repo returns zero files. There +is no ready-made Java2C library to fall back on. + +Consequence: **no ELF file anywhere in this pass was produced by a Dex-to-C compiler.** +Every Java2C-specific criterion in the reference document is therefore **inferred**. + +## 2. Degenerate route A — `dcc` generates the C without an NDK — observed + +`dcc.py` takes `--no-build`, and `dcc_main(..., do_compile=False)` writes the generated +sources and the JNI project *before* `build_project(project_dir)` would be called. With +`--no-build` and no `-o`, the apktool repack path is skipped entirely (it is guarded by +`if is_apk(apkfile) and outapk`). So the translation stage runs to completion on a machine +with no NDK. + +``` +$ cd tools/_work/bench/repos/dcc +$ python dcc.py /baseline.apk --no-build --dynamic-register \ + --source-dir /dcc-out +# (emits only Python 3.14 SyntaxWarnings from the bundled androguard) + +$ find /dcc-out -type f +jni/Android.mk +jni/Application.mk +jni/nc/Dex2C.cpp jni/nc/Dex2C.h +jni/nc/DynamicRegister.cpp jni/nc/DynamicRegister.h +jni/nc/ScopedLocalRef.h jni/nc/ScopedPthreadMutexLock.h +jni/nc/well_known_classes.cpp jni/nc/well_known_classes.h +jni/nc/compiled_methods.txt +jni/nc/Java_____.cpp +``` + +This is the mapping evidence the pass needed, and it is exact rather than imagined: + +`compiled_methods.txt` is a one-line dex signature, and it produced exactly one C file +whose name is the same identifier in JNI long form: + +``` +L/MainActivity;onCreate(Landroid/os/Bundle;)V + -> Java__MainActivity_onCreate__Landroid_os_Bundle_2.cpp +``` + +The generated function opens with the parameter shape the reference document cites as the +`.so`-side signature of a translated method: + +```c +/* L/MainActivity;->onCreate(Landroid/os/Bundle;)V */ +void Java__MainActivity_onCreate__Landroid_os_Bundle_2(JNIEnv *env, jobject thiz, jobject p2){ +jobject v0 = NULL; +... +jclass cls0 = NULL,cls1 = NULL,cls2 = NULL,cls3 = NULL; +``` + +And the registration half is generated as real text, which is the strongest single piece +of evidence produced in this pass that the "no `Java_*` symbol to search for" path is not +hypothetical: + +```c +// DynamicRegister.cpp +const char *dynamic_register_compile_methods(JNIEnv *env) { +jclass clazz; +clazz = env->FindClass("/MainActivity"); +if (clazz == nullptr) + return "Class not found: /MainActivity"; +const JNINativeMethod export_method_0[] = { +{"onCreate", "(Landroid/os/Bundle;)V", (void *)Java__MainActivity_onCreate__Landroid_os_Bundle_2} +}; +env->RegisterNatives(clazz, export_method_0, 1); +env->DeleteLocalRef(clazz); +return nullptr; +} +``` + +Two further facts read out of the generated `jni/Application.mk`, both **observed** and +both load-bearing for the JNI-boundary section of the reference document: + +``` +APP_STL := c++_static +APP_CPPFLAGS += -fvisibility=hidden +APP_ABI := armeabi-v7a arm64-v8a +``` + +`-fvisibility=hidden` means the shipped library can present as "exports no `Java_*` +symbol" **even when the translation used static naming** — the second, independent reason a +symbol search returns nothing. `c++_static` is the origin of the `libc++` string criterion, +and is also why that criterion is weak: it is an ordinary NDK setting. + +## 3. The probe against real JNI-sinking targets — observed + +This is the discriminating measurement the reference document is built on. The tool was +run unmodified: + +``` +$ python skills/apk-reverse/scripts/java2c_probe.py --apk /jni-a.apk + classes.dex: 720084 B, 625 classes, 5081 methods, native 2 (0.04%), + native-dominated classes 0, fully native 0 + libfoo.so: 14176 B, aarch64, dynsym 20 entries, Java_* 2, JNI_OnLoad False + libfoo.so: 13948 B, arm, dynsym 26 entries, Java_* 2, ... + libfoo.so: 13788 B, x86, dynsym 16 entries, Java_* 2, ... + libfoo.so: 14440 B, x86_64, dynsym 16 entries, Java_* 2, ... + verdict : jni-sinking + confidence : medium +``` + +``` +$ python skills/apk-reverse/scripts/java2c_probe.py --apk /jni-b.apk + classes.dex: 1934860 B, 1396 classes, 11182 methods, native 3 (0.03%), + native-dominated classes 0, fully native 0 + libfoo.so (arm/arm64: 3 Java_* symbols, x86/x86_64: 3 Java_* symbols) + verdict : jni-sinking + confidence : medium +``` + +Two independent real targets, and both give the same two results: **native density three +orders of magnitude below the Java2C shape**, and **an exact 1:1 between the dex native +declarations and the exported `Java_*` symbols** (2:2 and 3:3, across four ABIs each). + +That 1:1 is the point. It is *not* a Java2C signal on its own — both shapes produce it — +which is why the density measurement and the native layer must be read together. A tool +that reported on either alone would misroute one of these two samples. + +Baseline control, same command, no native code at all: + +``` +$ python skills/apk-reverse/scripts/java2c_probe.py --apk /baseline.apk + classes.dex: 5528 B, 7 classes, 15 methods, native 0 (0.00%), ... + verdict : no-hardening-signal + confidence : medium +``` + +## 4. The probe against a Java2C-shaped dex — observed + +No Java2C library could be built, so the dex half of the shape was synthesised. A Java +source declaring two classes whose methods are almost entirely `native` (plus one mixed +class, standing in for an ordinary JNI boundary) was compiled with `javac --release 8` and +assembled with `d8 --min-api 19`: + +``` +$ javac --release 8 -d classes src//Calc.java # exit 0 +$ d8 --min-api 19 --output out classes//*.class # exit 0 +out/classes.dex 1752 B sha256 59e0b1d34f00578e1343ee5f344d0e58829f482b643a7843c8ad280311ce9457 + +$ python skills/apk-reverse/scripts/java2c_probe.py --dex out/classes.dex + classes.dex: 1752 B, 3 classes, 29 methods, native 24 (82.76%), + native-dominated classes 2, fully native 1 + verdict : java2c-suspect-needs-so + confidence : low +``` + +The low confidence is the correct answer, not a shortcoming: the tool refuses to claim +Java2C from the dex alone, and says so in its route block. That refusal is the guard +against the misdiagnosis this pass exists to prevent. + +Combining that dex with a real library that registers dynamically makes the `java2c` branch +reachable, which is what proves the branch is not dead code — the input is composite and is +labelled as such: + +``` +$ python skills/apk-reverse/scripts/java2c_probe.py --dex out/classes.dex \ + --so /libandroid.so + verdict : java2c + confidence : medium + evidence: (strong) native layer: dynamic registration in libandroid.so + (JNI_OnLoad exported, zero Java_* symbols) +``` + +## 5. The JNI boundary — why `RegisterNatives` cannot be found — observed + +The reference document claims a symbol search fails silently and that looking for a +`RegisterNatives` symbol is structurally unable to work. Both are measured here. + +A real 273,120 B aarch64 system library pulled from one of the samples exports a JNI entry +point and no `Java_*` symbols, and the probe flags it as dynamic registration — the +affirmative check that does work: + +``` + libandroid.so: 273120 B, aarch64, dynsym 504 entries, Java_* 0, + JNI_OnLoad True, RegisterNatives-import False + (strong) native layer: dynamic registration in libandroid.so + (JNI_OnLoad exported, zero Java_* symbols) +``` + +Independent confirmation that the absence is real and not a parser gap — the same check run +outside the script: + +``` +$ python -c "d=open('gadget.so','rb').read(); [print(k, d.count(k.encode())) for k in + ['JNI_OnLoad','RegisterNatives','Java_','Dex2C','libc++_shared.so','frida']]" +JNI_OnLoad 0 +RegisterNatives 0 +Java_ 0 +Dex2C 0 +libc++_shared.so 0 +frida 311 +``` + +A 25.8 MB real third-party aarch64 ELF with 311 occurrences of its own name contains zero +`JNI_OnLoad` and zero `Java_`. The probe reported the same (0 `Java_*`, `JNI_OnLoad False`), +so the two readings agree and the probe is not reporting a parse failure as an absence. + +**Why the `RegisterNatives` search cannot fire** `[observed]`: the library above registers +dynamically and contains **zero literal `RegisterNatives` symbols**. The cause is a +documented property of the NDK C++ headers `[inferred]`: `jni.h` declares `RegisterNatives` +as an inline member of `JNIEnv` that dispatches through the function table, so a call site +emits no external symbol. The observed half is the missing symbol; the explanation is +inferred from the mechanism. + +## 6. Coverage of the probe's code paths — observed + +| Path | Exercised by | Result | +|---|---|---| +| dex header + class_defs + class_data (ULEB128) | 5 real dex images + 1 synthetic | Parsed; header size match reported | +| `direct_methods` / `virtual_methods` decoding | the synthetic fixture (has both) | Correct after the defect in §7 | +| native density, native-dominated classes | all six | 0.03–0.04 % real vs 82.76 % synthetic | +| no-`code_item` and trivial-body ratio | all six | 0 on the fixture, 4–5 % on the real apps | +| ELF header, arch, section table | 10 real ELFs (4 ABI variants of two `libfoo.so`, 2 app libs, 1 gadget) | aarch64/arm/x86/x86_64 all identified | +| `.dynsym` parse, `Java_*` extraction | real `libfoo.so` ×8 | 2 and 3 symbols, matching the dex exactly | +| `JNI_OnLoad` export detection | real system library | True | +| dynamic-registration verdict | same library | Fired | +| string-signature scan | 25.8 MB ELF, all samples | Weak hits only (`dcc` ×6 on the gadget); no false strong hit | +| `--json` | synthetic fixture | Valid JSON, metrics block parses | +| `--dir`, `--apk`, `--dex`, `--so` | all four modes | OK | +| `--help` | direct | OK | + +The string scan produced exactly one class of hit across every sample: the weak `dcc` +substring (6 occurrences inside a 25.8 MB binary, i.e. coincidental). **No strong marker was +ever produced by any real sample**, which is why the Java2C verdict rests on structure +rather than on strings. + +## 7. Not exercised — and why the Java2C verdict stays inferred + +- **A real Java2C `.so`.** Never produced; the compile chain is dead (§1). Everything about + how a *compiled* Java2C library presents — whether the generated functions keep their + names under `-fvisibility=hidden`, whether `Dex2C` runtime strings survive into `.rodata` + — is **inferred** from the generated sources and the build flags, not observed. +- **The runtime claim that no dex bytecode exists in memory.** This is the mechanism the + whole document turns on and it is **inferred from the translation model**. No live Java2C + app was run. It is falsifiable and cheap to test on a live sample; that test is named in + the reference document. +- **VMP identification by static dex metrics.** One VMP-labelled sample (sha256 + `76e3fa8119cf8b7c4b4768be557d71863433a8ae4d2ad40901eae6bb829349bc`, a 2,984,841 B APK + from the local bench repos, identified by hash only) + was measured and reported **`jni-sinking`**: its static dex has 23,603 methods of which 7 + are native, with a 4 % stub ratio, and the archive contains no additional encrypted + payload. **This is a recorded boundary, not a success**: whatever hardening that sample + carries is applied at runtime, and a static dex probe is structurally unable to see it. + Do not read this row as "the sample is unhardened". +- **Anything on a device.** The `` was not used; there is no dynamic evidence in + this record. + +## 8. Two defects this pass found in its own tool — observed + +Both were found by running the tool against the synthetic fixture and checking the numbers +by hand, and both are fixed in the shipped script. + +1. **`direct_methods` and `virtual_methods` were decoded as one continuous run.** They are + two independent encoded lists; each one's first `method_idx_diff` is relative to 0. + Walking them as one run mis-attributed every virtual method to the following class. The + fixture exposed it as `Codec: 5 methods` where the source declares 4. Fix: decode them as + two passes. After the fix the per-class counts match the source exactly (Calc 20/21 + native, Codec 3/3, Helper 1/2). + +2. **The "whole class is native" criterion was unreachable.** `javac` — and, per the + generated output, a translation pass — always leaves ``/`` behind, so no + class is ever literally all-native unless a constructor is counted as native. The + fixture showed `fully native 0` while two of its classes were plainly native-shaped. Fix: + constructors are excluded and the test is now ">= 70 % of >= 3 named methods native", + reported alongside the literal count. After the fix: `native-dominated classes 2, + fully native 1`. + +Both defects produced *plausible* output — a wrong count and a zero that reads as "nothing +found" — which is the failure shape this whole repository is organised against. Recording +them here because a reader relying on either number would have drawn a wrong conclusion. diff --git a/docs/tool-verification/EXTENSION-kernel-weapons.md b/docs/tool-verification/EXTENSION-kernel-weapons.md new file mode 100644 index 0000000..22f7a82 --- /dev/null +++ b/docs/tool-verification/EXTENSION-kernel-weapons.md @@ -0,0 +1,236 @@ +# Extension verification — kernel-level weapons (templates, gates, and a closed door) + +Covers the extension pass that added `scripts/kernelsu_syscall_mask.py` and the `§4a` section of +`references/kernel-and-environment-hardening.md`. Strength labels as defined in this directory's +README: **observed** (exact command + output here), **inferred** (follows from observed facts, +step not executed), **unverified** (assumed or externally reported, not independently confirmed). + +Reference environment: Windows 11 host, Python 3.14.0; `` = rooted Android 11 (API 30) +arm64-v8a, Magisk alpha, no APatch/KernelPatch. + +**Headline: the generator, the gate table and the userspace module skeleton are measured. No +kernel-side artifact was compiled or loaded anywhere, and every route is closed on the reference +device. That is the deliverable — a template plus a measured statement of the door.** + +--- + +## 1. Kernel gate, measured on the device — observed + +``` +$ adb shell 'uname -r; cat /proc/version' +4.14.186+ +Linux version 4.14.186+ (nobody@android-build) (Android (6443078 based on r383902) clang version +11.0.1 (https://android.googlesource.com/toolchain/llvm-project b397f81060ce6d701042b782172ed13bee898b79), +LLD 11.0.1 (/buildbot/tmp/tmp6_m7QH ...)) #1 SMP PREEMPT Wed Mar 30 23:32:42 CST 2022 +``` + +4.14.186 is below the 5.10 GKI gate for the eBPF route, and the device carries no +KernelPatch-patched image. Read-only query; no device lock was needed and no state was changed. + +## 2. Host toolchain probe — observed + +``` +$ python skills/apk-reverse/scripts/kernelsu_syscall_mask.py gates +== kernel-side routes and their gates == + + eBPF tracepoint/kprobe GKI kernel 5.10+ (Android 12+) + why: Android's BPF/tracing support as bpftrace-class tooling and + stackplz expect it is a GKI-era feature; BTF is required for + CO-RE style probes. + check: uname -r -> expect 5.10.x or newer; ls /sys/kernel/btf/vmlinux + + KernelPatch / APatch KPM KernelPatch-patched boot image + kpm toolchain + why: KPMs load through kpimg injected into the kernel image's + payload segment; building one needs a bare-metal ARM64 + compiler (aarch64-none-elf-gcc), not the NDK. + check: ls /data/adb/ | grep -i apatch ; which aarch64-none-elf-gcc + + Out-of-tree LKM Kernel source matching the device's exact version + vermagic + why: A module built against a different kernel revision is refused + at load time on a mismatched vermagic, and the device's kernel + source is frequently not published at all. + check: uname -r ; look for a matching kernel source tree for the device + + seccomp-BPF (for contrast) any modern kernel, but it is not this route + why: seccomp can make a syscall FAIL (SECCOMP_RET_ERRNO/TRAP); it + cannot rewrite the CONTENT a successful read returns. It closes + doors, it does not paint them -- which is the whole requirement + for a spoofed /proc read. + check: n/a -- this is a capability ceiling, not a version check + +== host probe (what can be checked from here) == + + bare-metal ARM64 compiler: NOT FOUND + the KPM route cannot be built from this host as configured + ndk-build: NOT FOUND + make: NOT FOUND + + A kernel source tree, a patched boot image and the device's own kernel headers cannot be + checked from the host; they are device facts. +``` + +The three `NOT FOUND` lines are the measured reason the KPM route cannot be built from here, and +they are also consistent with `ENV.md`'s record that the machine has no Android NDK. The LKM route +additionally needs kernel source that does not exist for this device. + +## 3. `generate` — observed + +``` +$ python skills/apk-reverse/scripts/kernelsu_syscall_mask.py generate \ + --out tools/_work/t5-vmp/sysmask --package '' +generated module scaffold in tools/_work/t5-vmp/sysmask + module.prop 165 B + customize.sh 166 B + post-fs-data.sh 830 B + service.sh 467 B + uninstall.sh 298 B + config/syscall_mask.json 1079 B + kpm/syscall-mask.c 8096 B + kpm/Makefile 880 B + lkm/syscall-mask.c 2434 B + ebpf/syscall_mask.bpf.c 4338 B + README.md 3752 B + +STATUS: the userspace module is installable; every kernel-side file is an UNVERIFIED template. + No kernel code was compiled or loaded. +``` + +`module.prop`, as generated (the package identity is the `` placeholder by design — this file +carries no real package name): + +``` +id=syscall-mask +name=Syscall Mask +version=0.1.0 +versionCode=1 +author=apk-reverse +description=config-driven syscall return masking (openat/read/stat on /proc/self/*) +``` + +The configuration table rendered into the KPM template from `config/syscall_mask.json`, so the +rule set is data rather than hand-edited C: + +``` +static const struct mask_rule rules[] = { + { "/proc/self/maps", 0, MASK_DENY, -2, 0 }, + { "/proc/self/maps", 0, MASK_FAKE_SIZE, -2, 0 }, + { "/proc/self/status", "TracerPid", MASK_FILTER_LINE, -2, 0 }, + { "/proc/self/status", "TracerPid", MASK_FILTER_LINE, -2, 0 }, +}; +``` + +And the KPM lifecycle surface, which is the part an interface drift would break first: + +``` +int kfunc_def(strncpy_from_user)(char *dst, const char __user *src, long count); +/* fp_hook_syscalln(__NR_openat, before_openat, 0, 0); */ +/* fp_hook_syscalln(__NR_newfstatat, before_newfstatat, 0, 0); */ +/* fp_hook_syscalln(__NR_read, before_read, 0, 0); */ +KPM_INIT(mask_init); +KPM_EXIT(mask_exit); +``` + +Those names (`KPM_NAME` / `KPM_INIT` / `KPM_EXIT`, `kfunc_def`, `fp_hook_syscalln`, +`hook_fargs4_t`, `syscall_argn`, `args->skip_origin`, `args->ret`) are taken from a public KPM +development write-up rather than invented — blackr0ck, *APatch KPM 开发：绕过 DirtySepolicy / +Duck Detector 的 SELinux Root 检测*, [bbs.kanxue.com/thread-291665.htm](https://bbs.kanxue.com/thread-291665.htm), +fetched 2026-02. **They are still `unverified` here**: nobody compiled against a `kpmodule.h` in +this pass, and the generated template says so in its own header for exactly that reason. + +The registration calls are left **commented out** in the template. A generated file that arms a +kernel hook by default is a file that arms a kernel hook the first time somebody runs `make` in a +hurry. + +## 4. `verify` — observed + +``` +$ python skills/apk-reverse/scripts/kernelsu_syscall_mask.py verify --dir tools/_work/t5-vmp/sysmask +== verifying tools/_work/t5-vmp/sysmask == + module.prop: author=apk-reverse, description=config-driven syscall return masking + (openat/read/stat on /proc/self/*), id=syscall-mask, name=Syscall Mask, + version=0.1.0, versionCode=1 + config: 4 rule(s), actions=['deny', 'fake_size', 'filter_line'] + kpm target present: Makefile, syscall-mask.c + status: UNVERIFIED TEMPLATE -- not compiled, not loaded + lkm target present: syscall-mask.c + status: UNVERIFIED TEMPLATE -- not compiled, not loaded + ebpf target present: syscall_mask.bpf.c + status: UNVERIFIED TEMPLATE -- not compiled, not loaded + kpm/syscall-mask.c: KPM_* lifecycle macros present, kfunc_def not extern + +== 0 problem(s): structure is consistent == + (this says nothing about whether the kernel side compiles or loads) +``` + +`verify` is deliberately read-only and deliberately narrow: it checks `module.prop` fields, the rule +JSON, and the one static trap in the KPM template that has a documented consequence (an `extern` +`kfunc_def` becomes `*UND*` and the loader refuses the module). It states on its last line what it +does **not** establish, because "0 problems" on a template is the easiest sentence in this file to +over-read. + +### 4a. A false positive this check produced, and its fix — observed + +The first run reported: + +``` +== 1 problem(s) == + - syscall-mask.c declares kfunc_def with `extern`, which the loader rejects as an unknown symbol +verify rc=1 +``` + +The template does not declare it with `extern` — its comment *explains* why `extern` is forbidden, +and the check's own regex ran from that prose across to the real declaration's semicolon. A linter +that reads comments as code produces reports nobody can act on. The fix strips `/* */` and `//` +comments before any token check (`_strip_c_comments`). Both the false positive and the fix are +recorded here rather than quietly repaired, because the failure mode — a check that fires on its +own documentation — is one a reader will meet again. + +## 5. The userspace module's ceiling — observed by construction, and worth stating plainly + +`post-fs-data.sh` and `service.sh` in the generated skeleton do not load anything. That is not an +omission: a KernelSU/Magisk/APatch-userspace module runs as root in the normal world and cannot +change what a syscall returns. The generated `README.md` says this in its first paragraph, and +`post-fs-data.sh` carries the reasoning in a comment: + +``` +# The kernel side (if you built and loaded one) owns the syscall table; this +# script's whole job is to make the configuration visible to it and to leave a +# record that the module actually ran. Do not put a syscall hook here: this is +# userspace, and nothing here can change what openat/read/stat return. +``` + +The uninstall hook likewise refuses to unload a kernel module, because a silent unload during +package removal is how a device ends up in a boot loop. + +## 6. eBPF route — gate table and the capability ceiling + +| Route | Gate | Check on the device | Status here | +|---|---|---|---| +| eBPF tracepoint/kprobe | GKI 5.10+ with tracing/BTF | `uname -r`; `ls /sys/kernel/btf/vmlinux` | **closed** — device is 4.14.186 | +| seccomp-BPF | any modern kernel | n/a | not this route: it can only fail a syscall, not rewrite a successful read | +| KernelPatch / APatch KPM | patched boot image + `aarch64-none-elf-gcc` | `ls /data/adb/ \| grep -i apatch`; `which aarch64-none-elf-gcc` | **closed** — neither present; patch-the-boot-image risk declined | +| Out-of-tree LKM | kernel source matching vermagic | `uname -r` + a matching source tree | **closed** — no source for this device | + +The probe template (`ebpf/syscall_mask.bpf.c`) is a `sys_enter_openat` tracepoint observer with a +uid filter map and a ringbuf event, and it documents two dead ends in the file itself: +`bpf_override_return()` only applies to `ALLOW_ERROR_INJECTION` functions, which raw syscall +entries are not, and a tracepoint cannot rewrite a result at all. **A tracepoint observes; it does +not spoof.** The honest eBPF shape for this job is therefore observation — which is still worth +having, because `§5` of the reference file requires you to establish whether the target uses a raw +`svc` or a libc call *before* designing any spoof. + +## 7. What was not done + +- **No kernel-side code was compiled.** Not the KPM, not the LKM, not the eBPF program. No + `kpmodule.h`, no kernel headers, no `bpftool`, no bare-metal ARM64 compiler on this host. +- **Nothing was loaded, flashed or activated on ``.** No boot-image patch, no module load, + no kernel modification. The task forbids it for good reason: a kernel module that faults takes + the device down before it can report why. +- **The KPM API surface is `unverified`.** The macro and helper names follow a public write-up; the + argument arity of `fp_hook_syscalln` in particular is marked `TODO(verify-on-device)` in the + generated source rather than guessed, because a wrong arity in a hook registration is a + load-time failure at best. +- **The struct-stat size offset is a TODO, not a value.** The `newfstatat` handler deliberately does + not carry a magic offset: it is ABI-specific and copying one from a different architecture is how + a spoof corrupts an unrelated field. +- **`verify` establishes structure, not capability.** Its own output says so. diff --git a/docs/tool-verification/EXTENSION-protobuf-raw.md b/docs/tool-verification/EXTENSION-protobuf-raw.md new file mode 100644 index 0000000..fe616f3 --- /dev/null +++ b/docs/tool-verification/EXTENSION-protobuf-raw.md @@ -0,0 +1,383 @@ +# Extension verification — schema-free protobuf decoding + +Covers the pass that added `skills/apk-reverse/scripts/protobuf_decode_raw.py` and the measured half +of `references/protocol-reverse.md` §1 ("The decoder that ships here"). Strength labels as defined in +this directory's README: **observed** (exact command + output recorded here), **inferred** (follows +from observed facts, step not executed), **unverified** (assumed or externally reported, not +independently confirmed here). + +Reference environment for this pass: Windows host, Python 3.14.0, `protobuf` **6.33.6**. No device +was used (see §7) — every result below is host-side, offline, and reproducible with the standard +library alone. Workbench artifacts live in `tools/_work/t5-proto/` (not part of the skill, and +`tools/` is gitignored). + +The delivered script imports **no** third-party module. `google.protobuf` is used here for one +purpose only: to *generate* reference bytes from a real schema, so the hand-written decoder is +checked against the reference implementation instead of against itself. + +--- + +## 0. Summary of what this pass established + +| Claim | Label | +|---|---| +| `protobuf_decode_raw.py` `--help`, `--selftest`, hex/file/stdin inputs, `--json`, `--max-depth`, `--reencode` all run on this host | observed | +| 26/26 built-in known-answer checks pass (field decode **and** byte-exact round trip) | observed | +| The decoder agrees field-by-field with a message serialized by `google.protobuf` 6.33.6, and its re-encode is byte-identical to `SerializeToString()` | observed | +| A real AndroidX DataStore file written by `scripts/datastore_inject.py` decodes to the correct structure and round-trips byte-identically | observed | +| Editing one value in the JSON tree and re-encoding produced a file that `datastore_inject.py` read back as the edited value | observed | +| proto3 zero semantics: an implicit (no-presence) field set to 0 writes **nothing**; a presence-carrying field set to 0 writes its tag plus a zero byte | observed | +| A packed field and a nested message are **both** legal readings of the same length-delimited bytes (measured on real encoded bytes, not argued) | observed | +| `--split varint-length` is protobuf's `writeDelimitedTo` framing; handed a gRPC frame it silently produced 5 frames instead of failing | observed | +| Decoding a **device-side** protobuf file (real app data read over adb) | **not done** (device lock held by another agent; §7) | +| Any protobuf payload recovered from the benchmark APKs | **not done** — the one APK scanned carries no `.pb`/`.proto` resource (§7) | +| `blackboxprotobuf` / `protobuf-inspector` behaviour | unverified (not installed on this host; the reference still says so) | + +--- + +## 1. Input forms, measured + +``` +$ python skills/apk-reverse/scripts/protobuf_decode_raw.py --help +usage: protobuf_decode_raw.py [-h] [--hex STR] [--json] [--max-depth N] + [--no-expand] [--split {none,varint-length}] + [--reencode JSON] [--out PATH] [--check PATH] + [--selftest] + [input] +... exit = 0 +``` + +Hex text, a binary file and stdin are all accepted, and the two stdin shapes are told apart without +a flag: + +``` +$ '0a220a176578616d706c655f6578706972795f65706f63685f6d7312072080b08fe6b277' \ + | python skills/apk-reverse/scripts/protobuf_decode_raw.py - +input: stdin (hex text), 36 byte(s) +frame 0 bytes 0..36 (36 B) status=end_of_buffer + f1 len-delimited(34) view=nested_message + ... + +$ cmd /c "python skills\apk-reverse\scripts\protobuf_decode_raw.py - < tools\_work\t5-proto\ds.preferences_pb" +input: stdin (raw bytes), 81 byte(s) +frame 0 bytes 0..81 (81 B) status=end_of_buffer +``` + +Separators and escapes, in one command (two frames, `0x` prefixes, spaces, a `\xNN` escape): + +``` +$ python skills/apk-reverse/scripts/protobuf_decode_raw.py --hex "0x08 0x96 0x01 | \x12\x07testing" +frame 0 bytes 0..3 (3 B) status=end_of_buffer + f1 varint 150 +frame 1 bytes 3..12 (9 B) status=end_of_buffer + f2 len-delimited(7) view=utf8_string + ... + string: 'testing' +``` + +`08,96,\n01` (commas and a newline) and the single token `0x089601` both decode to the same field. +**This command is the one that found the script's first real defect — see §6.** + +--- + +## 2. Known-answer fixtures: `--selftest` + +Twelve fixtures are built byte by byte inside the script (no external file, no schema), covering a +canonical varint, a multi-byte varint, `uint64` maximum, two-level nesting, a packed field, fixed64 +and fixed32, an explicit zero, a deprecated group, a non-canonical varint, a 10-byte overflow, two +top-level messages in one stream, and varint-length framing. Each is decoded with an expected value +**and** passed through the round trip. + +``` +$ python skills/apk-reverse/scripts/protobuf_decode_raw.py --selftest +== known-answer fixtures (bytes produced by hand in this script) == + canonical varint 150 089601 + field 1 varint value PASS 150 + unsigned max uint64 08ffffffffffffffffff01 + field 1 varint value PASS 18446744073709551615 + nested two levels 089601120774657374696e671a060801120210072206038e029ea705... + field 3 nested child count PASS 2 + deprecated group 3/4 43080544 + field 8 has children PASS 1 + non-canonical varint 08968100 + field 1 varint value PASS 150 + 10-byte varint overflow 08ffffffffffffffffff7f + field 1 varint overflow PASS True + ... +== round trip: decode -> re-encode -> byte comparison == + canonical varint 150 MATCH (1 fields, 1 identical) + nested two levels MATCH (7 fields, 7 identical) + non-canonical varint MATCH (1 fields, 0 identical) re-encode emits the minimal + form 089601, original 08968100, and the report states why + varint length framing MATCH (7 fields, 7 identical) + +== the two wire-format traps == + explicit zero bytes 2800 decodes to value 0; absent field decodes to no fields at all: PASS + packed payload 038e029ea705 offers live=['bytes', 'packed_varint'], rejected=['nested_message', + 'utf8_string'] -> view=packed_varint : PASS + +selftest: 26/26 PASS exit = 0 +``` + +The non-canonical fixture is the one deliberate exception: the re-encode is *expected* to differ +(`08968100` -> `089601`, the minimal form), and the tool reports the difference with its reason +rather than normalising it away silently. That is the honest form of "round trip": a difference that +is explained is evidence; a difference that is not is a bug. + +--- + +## 3. Cross-check against the official protobuf runtime + +`tools/_work/t5-proto/official_ref.py` builds a real `FileDescriptorProto` (`Outer` with int64 / +string / message / repeated int32 (packed) / fixed64 / float / enum / proto3-optional / bytes / +fixed32 fields, plus a nested `Inner`), serializes it with `SerializeToString(deterministic=True)`, +and then asks the hand-written decoder to reproduce it. + +``` +$ python tools\_work\t5-proto\official_ref.py +== official runtime bytes (protobuf 6.33.6) == + 58 B: 089601120774657374696e671a0508011201782206038e029ea705290807060504030201350000c03f380240004a06038e029ea70555efbeadde + +== field-by-field agreement with the schema == + field 1 count == 150 PASS decoded 150 + field 2 name == "testing" PASS decoded 'testing' + field 3 inner walked as a nested message PASS view=nested_message children=2 + field 4 nums read as packed [3,270,86942] PASS view=packed_varint values=[3, 270, 86942] + field 5 stamp fixed64 hex 0807060504030201 PASS 0807060504030201 + field 6 ratio fixed32 hex 0000c03f (1.5f) PASS 0000c03f + field 7 color enum == 2 PASS + field 8 explicit_zero is present and 0 PASS + field 9 blob hex 038e029ea705 PASS + field 10 crc fixed32 hex efbeadde PASS efbeadde + +== round trip against the official bytes == + re-encode is byte-identical to SerializeToString() PASS 10/10 fields identical + no field was rewritten PASS [] + +official-runtime cross-check: 21/21 PASS exit = 0 +``` + +Field 5 is the useful false-friend: the decoder offers `as_double_repr` for those 8 bytes, and the +official value is a `fixed64` integer. Nothing on the wire distinguishes them, which is why the +candidate list exists instead of a single typed value. + +### 3.1 The four-way tie, measured on real bytes + +A really-encoded nested message — `Inner{id:1, tag:"x"}` inside field 3, inner payload `0801120178` +— is reported as: + +``` + nested message payload 0801120178 -> {'nested_message': 0.5, 'packed_varint': 0.5, + 'utf8_string': 0.35, 'bytes': 0.2} +``` + +`0801120178` is simultaneously a valid two-field message **and** a valid five-element packed varint +array. The decoder refuses to choose; the `tie:` line states that the displayed `view` is a display +default. A real packed field (`2206038e029ea705`) yields `packed_varint 0.55` with +`nested_message` **rejected** — the rejection is printed with its reason instead of being omitted. + +### 3.2 proto3 zero — the reference sentence, corrected by measurement + +The reference previously stated that `28 00` is byte-identical to an absent field. The runtime shows +that the sentence needs splitting into two halves: + +``` + implicit int64 count = 0 : 120178 (3 B) + count never touched : 120178 (3 B) + implicit (no presence) zero is byte-identical to absent PASS + that byte string contains no tag for field 1 PASS + optional int32 explicit_zero=0: 1201784000 (5 B) + a presence-carrying field DOES put tag+0x00 on the wire PASS 1201784000 + the optional zero costs 2 bytes the implicit zero never costs PASS 5 B vs 3 B + the decoder sees that 0 and flags the ambiguity PASS ["proto3_explicit_zero"] +``` + +Read it as two separate facts: + +1. For a field **without** presence, `value = 0` and "never set" are the same bytes (no bytes at + all), so a field **missing** from a decode is not evidence that the value in use was 0. +2. A `0` that **is** on the wire was emitted on purpose — a presence-carrying field (`40 00` here, + field 8) or a hand-rolled writer. The decoder cannot tell which, so it flags the case. + +`references/protocol-reverse.md` §1 consequence 3 now carries that correction. + +--- + +## 4. A real AndroidX DataStore file + +`scripts/datastore_inject.py` writes the real AndroidX Preferences protobuf container +(`map`); the decoder has never seen that schema. + +``` +$ python skills/apk-reverse/scripts/datastore_inject.py --file tools\_work\t5-proto\ds.preferences_pb \ + --key example_expiry_epoch_ms --type long --value 4102444800000 \ + --extra example_flag=true:bool --extra example_name=hello:string +[ok] wrote tools\_work\t5-proto\ds.preferences_pb (81 bytes) + +$ python skills/apk-reverse/scripts/protobuf_decode_raw.py tools\_work\t5-proto\ds.preferences_pb +input: tools\_work\t5-proto\ds.preferences_pb, 81 byte(s) +frame 0 bytes 0..81 (81 B) status=end_of_buffer + f1 len-delimited(34) view=nested_message + ... + f1 len-delimited(23) view=utf8_string + string: 'example_expiry_epoch_ms' + f2 len-delimited(7) view=nested_message + ... + f4 varint 4102444800000 + f1 len-delimited(18) view=nested_message <- the second map entry + ... +``` + +The map-entry structure, the key strings and the `long` value all come out correctly with no schema; +every length-delimited field still shows its ties (`nested_message` vs `packed_varint` vs +`utf8_string`) because those ties are real. + +### 4.1 Round trip, and an edit confirmed by the other tool + +``` +$ python skills/apk-reverse/scripts/protobuf_decode_raw.py tools\_work\t5-proto\ds.preferences_pb --json > tools\_work\t5-proto\ds.tree.json +$ python skills/apk-reverse/scripts/protobuf_decode_raw.py --reencode tools\_work\t5-proto\ds.tree.json \ + --out tools\_work\t5-proto\ds.reenc.bin --check tools\_work\t5-proto\ds.preferences_pb +re-encoded 81 byte(s) from 3 field(s) + fields re-encoded byte-identically: 3/3 + written to tools\_work\t5-proto\ds.reenc.bin + check against tools\_work\t5-proto\ds.preferences_pb (81 B, the whole file): MATCH exit = 0 +``` + +Then one value is edited in the JSON tree and re-encoded: + +``` +editing field 4 from 4102444800000 +re-encoded 80 byte(s) from 3 field(s) + fields re-encoded byte-identically: 2/3 + CHANGED field 1: expected 0a22...80b08fe6b277, got 0a21...20ffc7afa025 (the tree was edited) + +$ python skills/apk-reverse/scripts/datastore_inject.py --in tools\_work\t5-proto\ds.edited.bin --list +[in] tools\_work\t5-proto\ds.edited.bin (80 bytes) + example_expiry_epoch_ms long=9999999999 (value=6 bytes) + example_flag bool=True (value=2 bytes) + example_name string='hello' (value=7 bytes) +``` + +Two independent facts in one run: the file shrank 81 -> 80 B because a 6-byte varint became a +5-byte one **and the enclosing length prefix was recomputed** (the re-encode rebuilds lengths, it +does not copy them), and a different tool of this repository read the edited bytes back as the new +value. A round trip that only agrees with itself would not have caught either. + +--- + +## 5. Framing: `writeDelimitedTo` is not a gRPC frame + +`tools/_work/t5-proto/framing_check.py`: + +``` +message (20 B): 089601120774657374696e672206038e029ea705 +writeDelimitedTo form (21 B): 14089601120774657374696e672206038e029ea705 + varint-length framing: exactly one frame PASS got 1 + a second message makes two frames PASS + gRPC frame (25 B): 0000000014089601120774657374696e672206038e029ea705 + decoded with --split varint-length: 5 frame(s): [(1, 1, 'end_of_buffer'), (2, 2, ...), + (3, 3, ...), (4, 4, ...), (5, 25, ...)] + gRPC frames are NOT varint-length framed (the split misfires) PASS got 5 frame(s) + stripping the 5-byte gRPC prefix decodes the message correctly PASS field 1=150, field 4=[3, 270, 86942] + '|' separates two independently framed messages PASS frames=2 + +framing check: 8/8 PASS exit = 0 +``` + +| Framing | Bytes in front of the message | `--split varint-length` | +|---|---|---| +| protobuf `writeDelimitedTo` | varint length | correct — 1 frame per message | +| gRPC | 1 compression-flag byte + **4-byte big-endian** length | **wrong, and it does not fail**: the 4 zero bytes become four empty frames | + +The same gRPC bytes decoded with **no** splitting reported `stray_end_group` (exit 1), and a +`writeDelimitedTo` file decoded with no splitting reported the same error — that error is a *refusal*, +whereas the wrongful split is a *silent reinterpretation*. Strip the 5-byte gRPC prefix yourself +(`references/protocol-reverse.md` §3), and check the frame count either way. + +A `--split varint-length` round trip compares **frame bodies only**, and says so: + +``` + check against tools\_work\t5-proto\two_frames.bin (26 B, frame bodies only: 2 of 26 byte(s) are + length prefixes (framing, not message data)): MATCH +``` + +--- + +## 6. Defects these checks found (in the code, not in the target) + +Four defects, three of them in the delivered script, all found by the checks above rather than by +reading the code. + +| # | Defect (observed) | How it surfaced | Fix | +|---|---|---|---| +| 1 | When two candidate readings tied, both were lowered to the same confidence as the `bytes` fallback, and the tie was then broken by **alphabetical order** — `bytes` sorts before `nested_message`, so a real nested message was never expanded and `--reencode` copied it as an opaque blob | `--selftest`: `nested two levels / field 3 nested child count` returned `0` instead of `2` | `bytes` now scores 0.2 and ties are broken by an explicit `VIEW_PRIORITY`, structural candidates first | +| 2 | In the `\xNN` path, whitespace and commas were kept as data bytes, so `"08 96 01 | \x12\x07testing"` decoded the space after the bar as `field 4 varint 18` and then hit `invalid_wire_type` | the input-form check in §1 | whitespace and commas are stripped before unescaping, in that branch too | +| 3 | Two fixtures were mis-designed, not the code: a bare packed stream and a length-prefixed frame were round-tripped as if each were one top-level message, which reported `FAIL` for correct behaviour | `--selftest` round-trip section | the packed fixture is now a field *value*; the framing fixture compares frame bodies | +| 4 | The cross-check's own assertion expected `0800` for the explicit zero, but the field under test is number 8, whose tag is `0x40` — the official bytes were `...4000`, and the assertion, not the decoder, was wrong | `official_ref.py` | assertion compares `4000` and the 2-byte cost | +| 5 | `--no-expand` did nothing: the expansion branch never consulted the flag, so a 7-level payload was still walked to the bottom | the deep-payload check below | the branch honours `--no-expand` and records `expansion: suppressed by --no-expand` on the field | +| 6 | Hitting `--max-depth` was invisible: the depth cut-off only appeared as a `rejected` note inside one candidate, so the report showed a shallow tree with no statement that anything had been left unexamined | the same check | a `depth_limit` ambiguity is emitted, naming the path and the limit | + +Defect 1 is worth its own line in any future reader's mind: a *heuristic* tie-break silently changed +which bytes the tool re-emitted, and it was caught only because a fixture asserted a child count. +Defects 5 and 6 are the other direction — the tool silently *did more* than asked, and silently +*stopped* without saying so; both are ways a report can be read as complete when it is not. + +### 6.1 The deep-payload check + +A 7-level nesting (`08 01` wrapped six times in field 1, 14 bytes) is the fixture: + +``` +$ python skills/apk-reverse/scripts/protobuf_decode_raw.py tools\_work\t5-proto\deep.bin +frame 0 bytes 0..14 (14 B) status=end_of_buffer + [depth_limit] frame0.1.1.1.1.1.1 @2: a nested reading of this field was not attempted: + --max-depth 5 reached. The bytes here may well be a message; re-run with a larger --max-depth + before concluding that they are not + +$ python ... tools\_work\t5-proto\deep.bin --max-depth 2 + [depth_limit] frame0.1.1.1 @2: ... --max-depth 2 reached. ... + +$ python ... tools\_work\t5-proto\deep.bin --no-expand + expansion: suppressed by --no-expand +``` + +The truncation is reported as an ambiguity with the path of the field it stopped at, so "no nested +message here" can never be read off a truncated tree. + +--- + +## 7. What was **not** done + +* **No device was used.** The pass intended to pull a real app's `.preferences_pb` from + ``: `python tools\_work\devlock.py acquire proto --timeout 1800` waited 120 s and timed out + with the lock held by another agent (`held by bench for 297s`, note "L1 repack control+patched + install & screen; L3 frida"). Per the two-strike rule the device source was dropped rather than + contended for, and the real-file evidence comes from the DataStore file this repository's own tool + writes (§4). Reading a **device-side** protobuf file with this decoder is therefore **unverified**. + Device facts used elsewhere in this repository (attach-by-name, the broken `logd` channel) were + not re-tested here. +* **No protobuf payload was recovered from the benchmark APKs.** Scanning the available sample + (``-style de-identified, 4,293,620 B, 435 zip entries) for `.pb` / `.proto` / `protobuf` + resources found none; the MASTG crackmes carry no protobuf at all. So no wild-captured payload was + decoded in this pass — `inferred` for any claim that a real traffic body looks like these fixtures, + though §3's bytes come from the reference implementation and §4's from a real AndroidX container. +* **No live traffic.** Nothing was proxied, so the gRPC section of the reference is exercised only + as *framing* (§5), not as captured HTTP/2. +* `blackboxprotobuf` / `protobuf-inspector` were not installed or compared; the reference still + labels them unverified. +* `--max-depth` is measured up to a 7-level fixture (§6.1); no real payload deeper than that was + decoded, and no payload was decoded whose *only* reading is a deep message. + +## 8. Artifacts produced by this pass + +`tools/_work/t5-proto/` (not committed): `official_ref.py` (official-runtime cross-check, 21 checks), +`framing_check.py` (8 checks), `ds.preferences_pb` (81 B, real DataStore container), +`ds.tree.json` / `ds.edited.json` (decoded trees), `ds.reenc.bin` (byte-identical round trip), +`ds.edited.bin` (80 B, edited value read back by `datastore_inject.py`), `two_frames.bin`, +`frames.tree.json`, `frames.reenc.bin`. + +Files changed in the repository by this pass: `skills/apk-reverse/scripts/protobuf_decode_raw.py` +(new), and `skills/apk-reverse/references/protocol-reverse.md` (a new measured subsection under §1, +the corrected proto3-zero wording in §1 consequence 3, and a new framing failure mode in §6). +`SKILL.md` and `README.md` were deliberately not touched; the index rows for the new script are +handed to the lead to register. diff --git a/docs/tool-verification/EXTENSION-split-apk.md b/docs/tool-verification/EXTENSION-split-apk.md new file mode 100644 index 0000000..487d1ce --- /dev/null +++ b/docs/tool-verification/EXTENSION-split-apk.md @@ -0,0 +1,337 @@ +# Extension verification — split APK / App Bundle sets + +Covers the pass that added split-set support to `scripts/repack.py` and wrote +`references/split-apk.md`. Strength labels as defined in this directory's README: +**observed** (exact command + output reproduced here), **inferred** (follows from observed +facts, step not executed), **unverified** (assumed or externally reported, not independently +confirmed). + +Reference environment for this pass: Windows 11 host, Python 3.14.0, Java 17.0.4.1 +(`keytool` NOT on PATH — it was passed explicitly as +`/bin/keytool.exe`), `apksigner` 0.9 run from `lib/apksigner.jar`, `zipalign.exe` from the +same build-tools layout. **No `uber-apk-signer` anywhere on the machine**, which is the condition +that made the signing-route fallback a requirement rather than a nicety. Target: one rooted physical +Android 11 (API 30) arm64-v8a device, written `` throughout. + +All package names, application directory names and file names are replaced with placeholders +(``, ``, ``, ``). The measurements, byte counts, entry counts and outputs +are unchanged; only the identifiers are. + +Fixtures: two **real split sets already installed on the device**, copied out read-only +(`adb pull` of the system directory; nothing on the device was uninstalled, replaced or modified). +They were chosen because one exercises each side of the merge/resign decision: + +| | Members | Shape | +|---|---|---| +| Set A | 8 | base + 7 density splits, every density split carrying its own resources | +| Set B | 9 | base + 1 ABI split (native libraries only) + 7 density splits that carry a placeholder table and nothing else | + +Neither set is third-party: `pm list packages -3` reported **zero** split sets among ~40 installed +third-party packages, while four preinstalled Google components were split. That distribution is +recorded because "pick a third-party split app off the device" is not reliably available. + +## 1. Reading a set: what the members actually contain — observed + +`pm path ` on each package returned one line per member (8 and 9 lines respectively), which is +how the targets were identified in the first place: + +``` +$ adb shell 'pm path ' +package:/system/priv-app//.apk +package:/system/priv-app//-arm64_v8a.apk +package:/system/priv-app//-hdpi.apk +... (one line per member) +``` + +`scripts/repack.py --split-mode analyze` over set A (identifiers replaced): + +``` +== split set: 8 apk(s), 6629510 bytes total + density -hdpi.apk 49623 B split=config.hdpi res=39 entries arsc=9392 + density -ldpi.apk 45501 B split=config.ldpi res=39 entries arsc=9392 + density -mdpi.apk 45501 B split=config.mdpi res=39 entries arsc=9392 + density -tvdpi.apk 74227 B split=config.tvdpi res=78 entries arsc=15124 + density -xhdpi.apk 53754 B split=config.xhdpi res=39 entries arsc=9432 + density -xxhdpi.apk 57878 B split=config.xxhdpi res=39 entries arsc=10324 + density -xxxhdpi.apk 57876 B split=config.xxxhdpi res=39 entries arsc=11200 + base .apk 6245150 B split=- dex=classes.dex res=581 entries arsc=1924488 assets=1 +[plan] base = .apk (package=) +[plan] MERGE BLOCKED: merging is not legal for 7 of 7 splits: config.hdpi, config.ldpi, config.mdpi, +config.tvdpi, config.xhdpi, config.xxhdpi, config.xxxhdpi carry their own resources, and folding those in +means merging resources.arsc (a resource-compiler job). Use --split-mode resign, or pass +--drop-split-resources to accept losing exactly those resources +[plan] resign = legal: one keystore for all 8 members, install the set with +[result] analyze only: nothing was written. Pass --split-mode merge|resign|auto to act on this set. +``` + +And over set B: + +``` +== split set: 9 apk(s), 1411272 bytes total + abi -arm64_v8a.apk 255861 B split=config.arm64_v8a libs=arm64-v8a + density -hdpi.apk 16797 B split=config.hdpi arsc=40 + ... (6 more density members, all 16797 B, all arsc=40) + base .apk 1037832 B split=- dex=classes.dex res=2 entries arsc=71316 assets=1 +[plan] base = .apk (package=) +[plan] note: every non-base split carries only code/native libraries, so a merged single APK is legal +[plan] note: config.arm64_v8a provides libs for arm64-v8a -- keep only the ABIs the target device runs, +selected with --abi +[plan] merge = legal: 8 member(s) fold into the base +``` + +Four facts this pass established with those numbers, and that the reference file now depends on: + +- **The ABI member carries no `resources.arsc` at all** (set B: `libs=arm64-v8a`, no arsc column). + Its two `.so` files are the whole payload. This is the member that folds cleanly. +- **Each density member carries its own table and its own `res/` entries** (set A: 39-78 entries, + 9.0-15.1 KB tables). `resources.arsc` is per member, not per set — the base's 1.88 MB table does + not contain the density variants. +- **A `resources.arsc` can be a 40-byte placeholder.** All seven density members of set B carry + `arsc=40` and zero `res/` entries, while still being legally required to carry a table. A + "has an arsc → owns resources" test would have blocked a merge that is entirely legal here, which + is why the script's test is *`res/` entries or a table over 1 KB*. +- **Base and split share one package name**, read from the binary AXML `package` attribute of every + member, and the base is identifiable as the member whose manifest carries **no `split` attribute** + (set A: `split=-`). + +## 2. Unified re-signing — observed + +`--split-mode resign` over set A with one keystore, run through the new `apksigner` route: + +``` +$ python skills/apk-reverse/scripts/repack.py --split-dir / --split-mode resign \ + --split-out-dir signed_set/ --ks key.jks \ + --keytool /bin/keytool.exe --apksigner /lib/apksigner.jar \ + --zipalign /zipalign.exe +[resign] signer route: apksigner +[resign] -hdpi.apk -> signed_set/-hdpi.apk (41298 bytes) +[resign] -ldpi.apk -> signed_set/-ldpi.apk (37202 bytes) +[resign] -mdpi.apk -> signed_set/-mdpi.apk (37202 bytes) +[resign] -tvdpi.apk -> signed_set/-tvdpi.apk (61642 bytes) +[resign] -xhdpi.apk -> signed_set/-xhdpi.apk (41337 bytes) +[resign] -xxhdpi.apk -> signed_set/-xxhdpi.apk (49563 bytes) +[resign] -xxxhdpi.apk -> signed_set/-xxxhdpi.apk (49577 bytes) +[resign] .apk -> signed_set/.apk (4172089 bytes) +[result] OK: 8 members, one certificate (sha256 1eb31d9c…2c44) +``` + +Every member's independent `apksigner verify --print-certs --verbose --min-sdk-version 21 +--max-sdk-version 34` returned `Verifies` with: + +``` +Verified using v1 scheme (JAR signing): true +Verified using v2 scheme (APK Signature Scheme v2): true +Verified using v3 scheme (APK Signature Scheme v3): true +Signer #1 certificate DN: CN=apkreverse, OU=dev, O=dev, L=NA, ST=NA, C=NA +Signer #1 certificate SHA-256 digest: 1eb31d9c9c81104367a66a0201125aa91d28a6a09eb1dcb5029c36bc0bde2c44 +``` + +All eight printed the **same** SHA-256 — that, not per-file `Verifies`, is the acceptance +criterion for a set, and it is the check the run performs explicitly. + +**The signing-route fallback is the load-bearing part of this section.** With no +`uber-apk-signer` on the machine, the pre-change script could not sign anything at all: it exited +with `signer jar not found`, naming a jar the user never had. The new `--signer auto` resolves to +`zipalign -p -f 4` + `apksigner sign --v1 --v2 --v3`, which produced the output above. `--apksigner` +accepts `apksigner.bat` (verified: `rc=0`, `--version` → `0.9`), a `lib/apksigner.jar` value run +through `java -jar` (verified: same), and the extensionless shell wrapper (which is a shell script +and does not run on Windows — that is a fact about the wrapper, not about this script). + +### The signing route this pass actually used — and why it had to change + +`uber-apk-signer` is `repack.py`'s default signer, and it is **not installed on this machine** +(no `uber-apk-signer.jar`, `APK_SIGNER_JAR` unset). Before this change that made every signed build +impossible here: the script exited with `signer jar not found` and named a jar the user never had. +What actually ran in all four measured routes: + +| Item | Value used | +|---|---| +| signer | `zipalign.exe` + `apksigner`, the latter run from `lib/apksigner.jar` through `java -jar` | +| `--apksigner` | `/lib/apksigner.jar`. A `.bat` value was also tested and works (`rc=0`, `--version` → `0.9`); the extensionless wrapper in the same layout is a shell script and does not run on Windows | +| `--zipalign` | `/zipalign.exe` | +| `--keytool` | `/bin/keytool.exe` — **not on PATH**, and signing aborts without a way to create the keystore, so this flag is mandatory on such a machine | +| scheme flags | `--v1-signing-enabled true --v2-signing-enabled true --v3-signing-enabled true` | + +Alignment is handled **per member, and in this order**: + +1. the script writes the archive itself, keeping `AndroidManifest.xml` and `resources.arsc` STORED + and 4-byte-aligning `resources.arsc` plus uncompressed `lib/**`, then re-reads the file and + prints the gate (`== alignment gate: resources.arsc STORED and 4-byte aligned (OK)`); +2. `zipalign -p -f 4` runs over that output (`-p` adds the page alignment the loader wants for + `extractNativeLibs="false"` libraries); +3. `apksigner sign` runs **last** — it adds signature entries without reshuffling the archive, + whereas a `jarsigner`-style rewrite after aligning destroys the alignment and the install is then + refused with `[-124]`. + +For a split set this is repeated for every member, which is why `resign_split_set()` de-signs and +re-aligns each member individually before signing it: a single member whose `resources.arsc` is +compressed or unaligned is refused by the installer on its own, and the resulting error names that +member's package rather than the set. + +## 3. Merging — observed, including the refusal + +Set B, where every non-base member is code or native libraries: + +``` +$ python skills/apk-reverse/scripts/repack.py --split-dir / --split-mode merge \ + --out merged.apk --abi arm64-v8a --ks key.jks --apksigner <...>/lib/apksigner.jar +[merge] + lib/arm64-v8a/lib_compat_libc++.so (18944 bytes <- -arm64_v8a.apk) +[merge] + lib/arm64-v8a/libutilsjni.so (206680 bytes <- -arm64_v8a.apk) +[merge] folded: dex=0 libs=2 assets=0 conflicts=0 +[merge] DROPPED 7 entry/ies. This is a DOWNGRADED build -- say so when you deliver it: +[zip] classes.dex 872160 bytes method=8 crc=<...> +[zip] resources.arsc STORED offset=0x581fc aligned +== alignment gate: resources.arsc STORED and 4-byte aligned (OK) +[sign:zipalign] rc=0 +[sign:apksigner] rc=0 +[out] merged.apk (603824 bytes) +[verify:apksigner] rc=0 Verifies / v1 true / v2 true / v3 true +[result] OK +``` + +`dex=0` is correct rather than suspicious: the base owns `classes.dex` and no member of this set +carries a second one, so no renumbering was needed. The `libs=2` fold is the whole point — a +one-file APK that now carries the native libraries the ABI member used to provide separately. + +Set A, where merging is **not** legal, refuses rather than producing a broken artifact: + +``` +$ python skills/apk-reverse/scripts/repack.py --split-dir / --split-mode merge \ + --out merged.apk --no-sign +error: refusing to merge -hdpi.apk -- it carries its own resources (39 res entries, +arsc=9392 bytes). + Folding it in would require merging resources.arsc, which this script does not do. + Either use --split-mode resign (recommended for a resource split), or pass + --drop-split-resources to accept that the resources it owns are lost. +exit code 1 +``` + +This is the enforcement the reference file promises: the refusal happens **before** any output is +written (the `--out` path does not exist afterwards), and the escape hatch is explicit, printed, and +required. The mechanism is **observed**; whether a build produced through that hatch merely loses +its density variants or misrenders is **inferred** — it was not exercised, because the only sets +available were a legal-merge set and a refused set, and manufacturing the third case would have +meant hand-editing a resource table. + +## 4. Single-APK behaviour is unchanged — observed + +Both pre-existing paths were re-run through the same new signing route, on an unmodified sample +APK from `tools/_work/bench/apks/`: + +- **zero-change roundtrip** (`--apk … --out … --ks …`, no `--dexdir`): the run dropped exactly the + three original signature artifacts (`META-INF/CERT.RSA`, `META-INF/CERT.SF`, `META-INF/MANIFEST.MF`), + wrote a 60964-byte unsigned archive, signed to 70647 bytes, reported `== alignment gate: + resources.arsc STORED and 4-byte aligned (OK)` and `[result] OK` with v1/v2/v3 all `true`; +- **explicit dex replacement** (`--dexdir` supplying the same `classes.dex`, 5528 bytes): + `[repl] classes.dex<-…/classes.dex`, the entry re-added at the same size and CRC, alignment gate + OK, `[result] OK`. + +Both were run with `--no-sign` as well, which exits 0 and reports `UNSIGNED (--no-sign)`. `--help` +works, exits 0, and carries the split examples and the route-decision text; `--split-mode analyze` +exits 0; a merge refusal exits 1; a failed verification would exit 2. + +The output fix worth recording: `AndroidManifest.xml` used to be printed as `NOT ALIGNED` in the zip +layout (it only has to be STORED, unlike `resources.arsc`). It now prints `-` for entries that need +no alignment, so the report cannot be misread as a failing gate. + +## 5. Install and launch on the device — **unverified** + +**Not executed.** The device was held by a concurrent device experiment for the whole window +available to this pass (`python tools/_work/devlock.py status` reported the lock held by another +teammate for 1417 s, note naming its own install/launch and Frida work), and the lead's decision was +to ship this gap labelled rather than wait for it. Heavy device work must be serialised +(`tools/_work/ENV.md`), so the honest label for everything below is **unverified** — a claim about +installability that was never installed is exactly the failure `references/verification.md` exists +to prevent. + +Prepared, and not run — the sequence is the one in `references/split-apk.md` §4, against the +re-signed renamed fixture: + +``` +adb install-multiple -r /*.apk +adb shell cmd package resolve-activity --brief +adb shell am start -n / +adb shell 'dumpsys activity activities | grep -m2 ResumedActivity' +adb exec-out screencap -p > launch.png +``` + +What the pass *did* establish without a device, and what each fact does not cover: + +- **The artifacts are installable in principle, not in fact.** All eight members of the re-signed set + verify with v1/v2/v3 `true` and one identical SHA-256 (§2), and the merged single APK verifies the + same way (§3, §4). Uniformity of the certificate and a passing `apksigner verify` are necessary + conditions for a set install; the platform installer accepting the set is the step that was not run. +- **The negative control was not run either.** The signature-conflict text quoted in + `references/split-apk.md` §6 (a pulled set reinstalled under its own package name) is the + platform's documented shape and the shape a competing tool reports; no captured output from + `` backs it. +- Every statement about **what the archives contain** — split roles, arsc ownership, the 40-byte + placeholder tables, the fold set, the refusal — is **observed** and device-independent. +- Every statement about **what the package manager does with them** is **inferred**. + +**What closing it needs:** one device window that is not shared, then the sequence above (about three +minutes of device time). The two things most likely to show up there, and already accounted for in +the script: an OEM installer interception on `adb install-multiple` (there is a root +`pm install-multiple` path for it) and a launcher activity that needs resolving rather than guessing. + +## 6. Maintenance gate — observed + +Run at the repository root with the change in place: + +``` +$ python check_repo.py +== result: 1 problem(s) + - skills/apk-reverse/scripts/dex_strpatch.py --help crashed: + +$ python check_refs.py +== section references: 242 reachable == +== result: 0 dangling, 0 warning(s) +``` + +`check_refs.py` is clean, which covers the three `references/repack-and-sign.md` §-references the new +file adds, and `check_repo.py` reports one problem in a **different** file belonging to a different +task; `repack.py` parses and answers `--help` with exit 0. + +`check_repo.py` requires every `references/*.md` to be named in `SKILL.md` or the root `README.md`, +and this task was not permitted to edit either, so the reachability step was pre-checked in a **copy** +of the repository (`tools/_work/repo-check/`, workbench only, never in the tree) by running the gate +twice on one snapshot with the index lines as the only variable: + +``` +without the index lines: == result: 5 problem(s) ... UNLISTED references/split-apk.md +with the index lines: == result: 4 problem(s) ... (the UNLISTED line is gone) +``` + +The new file therefore contributes nothing to the gate once listed. The listing itself was added to +`README.md` by the lead, and the run quoted above is the result. + +## 7. Deliberately not done — and what stays unverified + +- **No `.aab` was processed.** A bundle input would need `bundletool` (absent here, and it is the + tool that owns the correct resource merge). This pass only ever handled maps of an *installed* set. +- **No fixture was hand-built with `aapt2`/`d8`.** Those became available on this machine only after + the fixtures were chosen (a build-tools archive was unpacked late in the round), and the two real + sets already exercised both branches of the decision. Building a minimal split set by hand remains + the cheapest way to reproduce this topic on a machine with no installed bundle and no device, and + the route is **inferred** — it was not attempted here. It would not change the resource-merge + boundary either: `aapt2 link` merges resources when it *builds* an APK from sources, not when it is + handed two finished ones. +- **No resource merge was implemented or attempted**, by design + (`references/split-apk.md` §5). The consequence is that a bundle whose resources live in splits + cannot be reduced to one faithful APK by anything in this repository. +- **`apktool` was not exercised, because it is not usable on this machine.** Measured: + `apktool.bat d …` fails with `Error: Unable to access jarfile `, while + `lib/apksigner.jar` and `zipalign.exe` in the same layout work. Any manifest edit that needs + `apktool` is therefore **unverified** here, and AXML attributes are read-only in this pass (the + script parses the string pool and the root element's attributes; nothing writes AXML). +- **Feature splits (`isFeatureSplit`) were not measured.** Neither available set has one. The + handling of `isFeatureSplit` / `configForSplit` in the reference file is **inferred** from the + platform's documented semantics plus the fact that both attributes are readable through the same + AXML reader that was verified against real manifests. +- **Split APKs that are not App-Bundle-derived** (the older per-density `-hdpi.apk` naming style, + or an OEM's own module set) were not available; both sets here use the standard `config.` + split names, and the reader accepts either because it reads the manifest attribute rather than the + file name. +- **A XAPK/APKS archive** (a store bundle of bundles, usually a zip holding the split set plus + `manifest.json`) was not processed. The script's `--split-dir` recursion handles a *directory* of + APKs, so an unpacked XAPK works; the archive itself is not read. diff --git a/docs/tool-verification/EXTENSION-stalker-exclude.md b/docs/tool-verification/EXTENSION-stalker-exclude.md new file mode 100644 index 0000000..d901aca --- /dev/null +++ b/docs/tool-verification/EXTENSION-stalker-exclude.md @@ -0,0 +1,89 @@ +# Stalker `exclude` — what it fixes, and what it does not + +**Question.** Community practice says a followed thread that runs through `libc`/`libart` on arm64 +gives deadlocks, watchdog `SIGABRT`s and zero-event traces, and that `Stalker.exclude()` is the +remedy. This pass measured which of those two claims survives contact with a device. + +**Strength summary:** the *crash* half is `observed` with a clean control; the *zero-event* half is +`observed` as a **negative** — exclusion did not change it. Every command and its output are below. + +## Design + +Three arms, one device, one package, one module, one follow window. Exactly one variable moves +between `baseline` → `control` → `treatment`. + +| Arm | What it does | Why it exists | +|---|---|---| +| `baseline` | spawn, attach, inject the script, **resume, and never follow** | Without this arm a dead process cannot be attributed: "following killed it" and "the app dies under frida anyway" look identical | +| `control` | follow the main thread for 6 s with `excludeModules: []` | The shape that produced the earlier zero-event / `SIGSEGV` results | +| `treatment` | same follow, with the script's built-in list of 20 system modules excluded | The proposed remedy | + +The harness also neutralises one trap in the script itself: `CONFIG.autoStart` defaults to `true`, so +loading `stalker_trace.js` starts a follow with *default* options before a runtime configuration can +land. Left alone, that run is neither the control nor the treatment and contaminates both. The +harness patches `autoStart` to `false` in the injected source and asserts the patch count. + +For the same reason the harness resumes explicitly: `frida` spawns a process **suspended**, so +without `device.resume(pid)` the followed thread executes nothing and both arms report a zero-event +trace *manufactured by the harness*. That failure mode is indistinguishable from the one under +investigation, which is exactly why it has to be engineered out before the measurement means +anything. + +## Result + +``` +baseline alive=True (no follow installed) +control alive=False err=script has been destroyed + EXCL|excluded=0/0 [] +treatment alive=True DONE|reason=timeout blocks=0 blk=0 calls=0 truncated=0 + EXCL|excluded=20/20 [libm.so,libdl.so,libart.so,libartbase.so,libnativehelper.so,…] not-loaded=3 + WARN|zero events 1500ms after follow (blocks=0 blk=0 calls=0) -- the pipeline is NOT proven +``` + +## What this settles + +1. **A follow with no exclusion killed the target process, and the baseline arm rules out the + alternatives.** The process survives an attach plus resume with no follow, and dies only when the + follow is installed. That is `observed`, not inferred. +2. **Exclusion kept the process alive.** The treatment arm reached its normal `DONE|reason=timeout` + with the process still running. Excluding the system libraries from a follow is not a style + preference. +3. **Exclusion did not restore event delivery.** The treatment arm still reported + `blocks=0 blk=0 calls=0` after a full 6-second window. The zero-event trace is therefore a + **separate defect from the load problem**, and `Stalker.exclude()` must not be presented as its + fix. This partially contradicts the community framing, which lists "no events" among the symptoms + exclusion cures. +4. **The zero-event warning added to `stalker_trace.js` this pass fires correctly** — `treatment` + emitted `WARN|zero events 1500ms after follow`, while `control` could not, because its process was + already gone. A dead pipeline now announces itself instead of being read as "the code did not run". + +## What this does not settle + +- **Why events do not arrive.** This measurement says it is not the exclusion list. It does not + identify the cause, and it does not distinguish "no events" from "the followed thread genuinely ran + nothing in that window" — the per-arm control that would separate those (following a thread + executing a known loop) was not run. +- **Whether the crash is reachable without a follow.** Only one package and one module were used, and + the harness kills the process at the end of each arm, so nothing is claimed about a long-lived follow. +- **Anything about other ROMs, other frida versions, or armv7.** Single-device result. +- **The multiplier.** Nothing here measures the 20-50x cost figure; this is about survival and + delivery, not throughput. + +## Environment + +`` (Android 11 / API 30 / arm64-v8a, Magisk root), frida host 16.7.19, on-device +frida-server 16.7.19 reached over `adb forward tcp:27099`. Target: the MASTG `UnCrackable-Level1` +sample, package ``, module `libc.so` followed on the main thread for 6 s per arm. Device +`load average` was ~30 throughout — high for an 8-core phone, and worth recording because the +earlier incident in `native-dbi-and-deobfuscation.md` §6 happened on a loaded device. + +## Reproducing + +``` +python tools/_work/b6/stalker_ab.py --package --module libc.so \ + --mode baseline|control|treatment --seconds 6 --device 127.0.0.1:27099 \ + --out tools/_work/b6/r_.json +``` + +The driver is a work artefact under `tools/_work/` (git-ignored), not part of the skill. Its three +arms and the two harness traps it neutralises are the reusable part. diff --git a/docs/tool-verification/EXTENSION-vmp-diff.md b/docs/tool-verification/EXTENSION-vmp-diff.md new file mode 100644 index 0000000..92ecccb --- /dev/null +++ b/docs/tool-verification/EXTENSION-vmp-diff.md @@ -0,0 +1,293 @@ +# Extension verification — differential hardening for Dex-VMP + +Covers the extension pass that added `references/vmp-differential-analysis.md` and +`scripts/vmp_diff_harness.py`. Strength labels as defined in this directory's README: +**observed** (exact command + output here), **inferred** (follows from observed facts, +step not executed), **unverified** (assumed or externally reported, not independently +confirmed). + +Reference environment: Windows 11 host, Python 3.14.0, JDK 17.0.4.1 +(`C:\Program Files\Java\jdk-17.0.4.1\bin\javac.exe`), Android build-tools 2.19 / D8 +8.2.2-dev off-PATH. **No device and no network were involved in this pass** — unlike the +unpacking and rootdump passes, this one is deliberately machine-local, because the thing +worth measuring here is the comparison logic, not the platform. + +**Headline: the local half of the route is measured and closes exactly; the +platform half was not exercised at all, and the cost table says why it cannot be.** + +--- + +## 1. Environment discovery — observed, and one correction to `tools/_work/ENV.md` + +`ENV.md` records `E:\tools\android-14\` as a complete build-tools directory. At pass time +it held **three** entries: + +``` +$ Get-ChildItem 'E:\tools\android-14\' -Name +apksigner.bat +lib +zipalign.exe +``` + +`aapt2.exe`, `d8.bat` and `dexdump.exe` were absent, so the `build` subcommand had no +toolchain. The complete build-tools tree was sitting unextracted in `E:\tools\bt.zip`: + +``` +$ (bt.zip entries) | Where-Object { $_.Name -match '^(aapt2|d8|dexdump|split-select)' } +android-14/aapt2.exe 4566296 +android-14/d8.bat 3156 +android-14/dexdump.exe 986392 +android-14/split-select.exe 1589016 +android-14/lib/d8.jar 14929869 +``` + +After `Expand-Archive -Path 'E:\tools\bt.zip' -DestinationPath 'E:\tools\' -Force`: + +``` +$ E:\tools\android-14\aapt2.exe version +Android Asset Packaging Tool (aapt) 2.19-10229193 + +$ E:\tools\android-14\d8.bat --version +D8 8.2.2-dev (build facedf41bbd28b563d1e9e09c5f72d7c5ca598d8 from go/r8bot (luci-r8-custom-ci-focal-14-zjz7)) +``` + +Two further tool facts worth recording, both **observed**: + +- **`E:\agb_apktool.jar` does not exist**, so `E:\tools\bin\apktool.bat` (which is + `java -jar "E:\agb_apktool.jar" %*`) resolves to nothing. apktool is unavailable on this + machine, and a bare `apktool d` fails with a jar-not-found error rather than a + recognisable "tool missing" message. +- `javac` is on PATH only through the Oracle `javapath` forwarder; the real JDK is at + `C:\Program Files\Java\jdk-17.0.4.1`. The harness takes `--javac` explicitly for this + reason. + +## 2. `build` — observed + +``` +$ python skills/apk-reverse/scripts/vmp_diff_harness.py build \ + --build-tools E:\tools\android-14 --out tools/_work/t5-vmp/fixture \ + --javac "C:\Program Files\Java\jdk-17.0.4.1\bin\javac.exe" --apk +javac: C:\Program Files\Java\jdk-17.0.4.1\bin\javac.exe +d8: E:\tools\android-14\d8.bat +aapt2: E:\tools\android-14\aapt2.exe +generated: ...\src\OpCoverBig.java (37196 B) + +$ ... javac -g --release 11 -d ...\classes ...\OpCoverProbe.java ...\OpCoverBig.java +javac ok: 9 class files + +$ E:\tools\android-14\d8.bat --min-api 26 --no-desugaring --output ...\dex <9 .class paths> +d8 ok: ...\dex\classes.dex (35028 B) +== ...\classes.dex (35028 bytes) == + classes=9 methods_with_code=75 instructions=3065 desynced=0 + opcode coverage: 218/224 (97.3%) + not emitted by this fixture (6): + 0x09 move-object/16 move-object/16 (32x) needs both registers >= 256; the object + frame reaches 256 on the source side only, which d8 encodes as 0x08 + 0x1b const-string/jumbo needs a string_ids index > 65535, i.e. a fixture carrying + 65536+ string constants + 0x2a goto/32 needs a >32767-code-unit backward jump; javac refuses the method + first with 'code too large' (its own 64 KB per-method bytecode cap) + 0xfd invoke-custom/range d8 emitted the 35c form for every lambda shape tried, including + a six-parameter one + 0xfe const-method-handle no Java-language literal; reachable only by hand-written smali + or direct dex construction + 0xff const-method-type same as 0xFE + +$ E:\tools\android-14\aapt2.exe link --manifest ...\apk\AndroidManifest.xml -o ...\apk\base.apk +aapt2 ok: ...\probe.apk (20482 B, unsigned) +build rc=0 +``` + +Structural facts behind those numbers: **0 desynced methods** (every `decode_all` walk +ended exactly on `insns_off + insns_size*2`, i.e. the format table and the emitted stream +agree), 3,065 instructions across 75 method bodies, 9 classes. + +### 2a. The two toolchain traps this measured — observed + +Both are encoded in the script so they are not re-derived: + +- **`d8` refuses a non-existent `--output` directory.** First run: + `Error: Invalid output: ...\dex / Output must be a .zip or .jar archive or an existing + directory / Compilation failed` and rc=1. Same class of trap as `frida-dexdump -o`. + `build` creates the directory first. +- **`aapt2 link` needs `android.jar` for every `android:` attribute.** A manifest carrying + `android:versionCode` / `android:versionName` / `android:minSdkVersion` produced five + `error: attribute android:... not found` lines and rc=1. A manifest using only the + un-namespaced `package` attribute links cleanly with no `-I` flag at all — which is what + the harness ships, at the cost of an APK with no version attributes. That is acceptable + for a dex carrier and is stated in the output. + +### 2b. `goto/32` is unreachable from Java — observed + +Driving the generator to 33,000 loop-body statements to push the backward jump past the +20t range: + +``` +$ python tools/_work/t5-vmp/gen_big_probe.py 33000 +generated ...\OpCoverBig.java: 34442 lines, 766468 bytes +javac rc=1 +OpCoverBig.java:1427: error: code too large + public static int gotoFar(int x) { +1 error +``` + +The JVM's own 64 KB per-method bytecode limit fires before the dex branch-distance +threshold can be crossed. `goto/32` therefore cannot be covered by any javac-built +fixture, at any generator size — this is a structural ceiling, not a tuning problem. + +## 3. Closed loop — observed, and this is the load-bearing result + +`simulate` relabels every opcode through a bijection whose ground truth it writes to disk. +`compare` is then given only the two dex files and must re-derive that table. + +``` +$ python .../vmp_diff_harness.py simulate .../fixture/dex/classes.dex \ + --out tools/_work/t5-vmp/sim.dex --seed 7 --table tools/_work/t5-vmp/truth.json +simulated: ...classes.dex -> ...sim.dex + mode=relabel, 3065 method body(ies)/instruction(s) touched (seed 7) + header recomputed: checksum_ok=True signature_ok=True + ground-truth table written to ...truth.json (compare must re-derive it) + +$ python .../vmp_diff_harness.py compare .../fixture/dex/classes.dex tools/_work/t5-vmp/sim.dex + aligned=75 resized=0 stripped=0 missing=0 added=0 + aligned slots compared: 3065 + verdict: USABLE + (map: 218 rows, every one verdict=high) +compare rc=0 +``` + +Scoring the derivation against the ground truth: + +``` +ground-truth bijection size : 256 +opcodes the fixture emits : 218 +compare rows : 218 +compare high-confidence rows : 218 +compare undetermined : 6 +compare non-injective : 0 +compare run verdict : usable + +derived entries that are WRONG : 0 (none) +derived entries with no truth : 0 [] +emitted opcodes NOT derived : 0 [] + +EXACT RECOVERY: 218/218 emitted opcodes (100.0%) +CLOSED-LOOP VERDICT: IDENTICAL +``` + +Why the denominator is 218 and not 256: the ground-truth bijection covers all 256 byte +values, but 32 of those are unallocated dex slots and 6 are opcodes this fixture never +emits — and an opcode that never appears leaves no known plaintext. The six are reported +as `undetermined` with their reasons, never guessed. **Every entry the run did produce was +correct, and every opcode that could be read was read.** + +## 4. Negative shapes — observed + +The decision table's failure branches were driven on purpose, because a detector that has +never been seen to fire is not a detector. + +**Stub shape** (`--mode stub`: every body replaced by an equal-length `return-void` fill — +the extraction shape): + +``` +$ python .../vmp_diff_harness.py compare .../classes.dex tools/_work/t5-vmp/stub.dex --quiet + aligned=75 resized=0 stripped=0 missing=0 added=0 + aligned slots compared: 3065 + verdict: NOT-USABLE + ! 1 private byte(s) are claimed by up to 218 original opcodes each. No injective + relabelling can do that: it is the signature of method bodies replaced by one + common stub, and any substitution read off such a pair describes the stub, not a + private opcode space. +rc=1 +``` + +This is the failure mode worth reading twice. In the per-opcode direction the reading is +**perfect** — 218 rows, all `high`, zero conflicts — because every original opcode maps to +the same private byte. Only the reverse (`private byte → set of originals`) check exposes +it. An implementation without that check would have returned a confident, wholly +meaningless table, and the first version of this one did exactly that: measured before the +fix, the same input returned `verdict: PARTIALLY-USABLE` alongside a note reading +"0 of 218 opcodes saw more than one candidate byte" — a warning that contradicted itself. + +**Length-changing shape** (`--mode shrink`: `insns_size` rewritten without moving bytes, a +deliberately inconsistent image that exists only to drive the detector): + +``` +$ python .../vmp_diff_harness.py compare .../classes.dex tools/_work/t5-vmp/shrink.dex --quiet + aligned=0 resized=75 stripped=0 missing=0 added=0 + aligned slots compared: 0 + verdict: NOT-APPLICABLE + ! 75 method(s) kept a body but changed its length. Boundaries cannot be projected + across a length change; a private opcode stream with its own length table is not + reachable this way. +rc=1 +``` + +Regression after both fixes: the relabel loop still reports `USABLE` and the closed-loop +check still reports `IDENTICAL` — recorded above, re-run after the verdict-logic change. + +**Not driven:** the `stripped` shape (`code_off == 0`) needs a class-data rewrite rather +than a byte edit, so its detector path is **inferred** from the shape counters, not +observed firing. The counters themselves (`stripped=0` in every run above) are observed. + +## 5. `emit-smali` — observed + +``` +$ python .../vmp_diff_harness.py emit-smali tools/_work/t5-vmp/sim.dex \ + --table tools/_work/t5-vmp/cmp.json --class LOpCoverProbe --method arithInt +; instructions restored: 3065 ; unmapped-opcode stops: 0 +; THIS IS A READING SKELETON, NOT ASSEMBLABLE SMALI. +... +.class public LOpCoverProbe; +.super Ljava/lang/Object; +.source "OpCoverProbe.java" + +# ---- arithInt(II)I registers=4 insns_size=29 +.method public static arithInt(II)I + .registers 4 + # 0x0000 90000203 add-int + # 0x0004 d000e803 add-int/lit16 + # 0x0008 d8000007 add-int/lit8 + # 0x000c b130 sub-int/2addr + ... + # 0x0038 0f00 return +.end method +``` + +Two checks that the restore is real rather than plausible: the stream decodes to the +**original fixture's** `arithInt` body (compare it against `OpCoverProbe.java`'s 17 +operations), and the payload-bearing `switchPacked` method also restores cleanly — +`packed-switch` followed by its payload block, stepped over by its own layout rather than +mistaken for opcodes. `unmapped-opcode stops: 0` is the honest coverage measure, and it is +zero only because the table is complete for this fixture. + +Three defects were found and fixed by this run, each one a silent-wrong-answer shape: + +- `--class LOpCoverProbe;` matched **zero** classes because the filter accepted only one of + the three spellings (`LOpCoverProbe;` / `LOpCoverProbe` / `OpCoverProbe`) — output read + exactly like a class that is not in the file. +- `.source` printed `""`: `source_file_idx` sits at + `class_def_item + 16`, not `+ 20` (that is `annotations_off`). +- Methods rendered without access flags, so the skeleton was not readable as smali. + +## 6. What was not done + +- **No hardening platform was contacted.** The entire submission link of the chain is + `unverified`, and its cost estimate in the reference file is reasoning about how those + services are distributed (accounts, rate limits, download-page delivery, possible refusal + of a synthetic fixture), not a measured upload. This is the pass's largest gap and it is + not closeable from this machine. +- **No real VMP sample was used.** `tools/_work/bench/repos/DexPatcher/assets/ezAndroid.apk` + is recorded in `ENV.md` as a real Dex-VMP challenge sample and was **not** opened in this + pass; the differential route needs a *matching hardened counterpart of the same build*, + which no available artifact provides. +- **The engine-behaviour assumption is inferred**: that a real VMP substitutes per-opcode + at stable instruction length. `simulate` implements the weakest form of that assumption + deliberately — it is the only form this harness can check itself against. +- **Table stability across inputs is unverified.** Whether one fixture's table applies to + another build of the same app, or survives a vendor version change, was not tested and is + the reason the reference file says to re-derive per target. +- **The `stripped` detector path did not fire** (see §4). +- `--mode shrink` writes a structurally inconsistent dex by design; `dexutil.Dex.check()` + will not reject it because it does not range-check `insns` against the file. It is a + detector probe, is labelled as such in `--help`, and must not be treated as a build. diff --git a/docs/tool-verification/README.md b/docs/tool-verification/README.md index 5895008..fc75b54 100644 --- a/docs/tool-verification/README.md +++ b/docs/tool-verification/README.md @@ -46,6 +46,24 @@ evidence here, one file per topic, in the same three labels: | `EXTENSION-native-dbi.md` | Stalker: the zero-event boundary, the crash from following a hot libc export, and an offline protobuf round-trip that caught a defect in its own decoder | | `EXTENSION-kernel-ondevice.md` | The MT Manager APK MCP probe, the module-environment facts it could verify, and the kernel routes that this device's 4.14 kernel puts out of reach | +## Benchmark-pass record + +The benchmark pass put **public targets** under the documented routes — `tests/benchmark.md` is the +matrix, and these files are the evidence behind its rows. Same three labels. Several of these rows +came back negative, and four of them overturned a conclusion this repository had already recorded; +those are called out in the matrix rather than buried here. + +| File | Covers | +|---|---| +| `EXTENSION-benchmark-l1-l3.md` | The first two matrix rows: the L1 equal-length patch -> repack -> re-sign chain with its on-screen behaviour change and its zero-change control, and the L3 time-to-death baseline, the two spawn/patch/detach orderings that cannot work, and the terminate path that patching the obvious death site does not stop | +| `EXTENSION-extraction-shell-bench.md` | `dex_dump_validate.py` against an 11-variant skeleton set: the trivial-body ratio as a bimodal result rather than a threshold, the three script defects the variants exposed, and the ranking that pointed at a modified image until independent review caught it | +| `EXTENSION-stalker-exclude.md` | Stalker exclusion measured across three arms: the no-follow baseline, the follow that kills the process, and the follow with 20 modules excluded that survives but still delivers zero events | +| `EXTENSION-java2c.md` | Java2C against JNI sinking: native-declaration density roughly 2000x apart, why a `Java_*` symbol search fails silently, and the Dex-to-C build chain no toolchain on this host could complete | +| `EXTENSION-vmp-diff.md` | The known-plaintext differential: a 218-of-224-opcode coverage fixture, a closed-loop check that recovered 218/218 mappings with zero fabrications, and the one link that cannot be automated | +| `EXTENSION-kernel-weapons.md` | The generated KernelSU/APatch scaffold and its three kernel-side templates, all shipped unbuilt, plus the structural correction that a userspace module cannot change a syscall return value | +| `EXTENSION-split-apk.md` | Two real split sets through both branches: unified re-signing, the merge that is refused by design, the merge that succeeds, and the signer chain this host actually needs | +| `EXTENSION-protobuf-raw.md` | A schema-free decoder cross-checked against the official runtime and a real DataStore container, with the packed-boundary and proto3-zero ambiguities reproduced on real bytes | + ## Adding a run Record the tools, versions and exact commands, and label each claim. Keep target-specific facts out: diff --git a/skills/apk-reverse/SKILL.md b/skills/apk-reverse/SKILL.md index b127584..9c17b3b 100644 --- a/skills/apk-reverse/SKILL.md +++ b/skills/apk-reverse/SKILL.md @@ -107,6 +107,13 @@ vocabulary and a set of reassuring numbers. - Packers, custom loaders and code virtualization: identifying them, measuring the validation boundary, and the routes that survive it (`packers.md`, `code-virtualization-and-custom-linkers.md`). +- **Java2C** — recognising that the code left the dex entirely, and the evidence that separates it + from an extraction shell, a VMP and ordinary JNI sinking (`java2c-and-jni-sinking.md`, + `scripts/java2c_probe.py`). Measured: `native` declaration density was **0.03-0.04%** in three real + JNI samples against **82.76%** in a Java2C-shaped fixture — roughly 2000x — and the `Java_*` symbol + count matched the dex native-method count 1:1 per ABI. **No library produced by a Dex-to-C compiler + was built here** (no NDK, no host clang, no device clang, WSL unavailable), so every Java2C-specific + criterion is `inferred`; the JNI-sinking side and the discrimination itself are `observed`. - The native layer: `.so` hosts, tamper-triggered self-termination, forged ELF structure, and neutralising a terminate path without freezing the process (`native-and-so.md`, `native-tamper-and-suicide.md`). @@ -299,7 +306,8 @@ unread in this repository. | A `FORTIFY: pthread_mutex_lock called on a destroyed mutex` abort in a Flutter app, on the **main** thread, before the first frame completes | `dart-aot.md` — check `libapp.so` is actually being loaded; Flutter's engine bootstrap is the usual place a native lifecycle fault surfaces | | Log says a **Java-layer** signature/integrity check **passed**, yet the process dies | `code-virtualization-and-custom-linkers.md` §a Java-layer "signature killer" is a decoy | | Deleting a library fixes validation but yields `UnsatisfiedLinkError: dlopen failed: library "X" not found` | `code-virtualization-and-custom-linkers.md` §the deadlock that eats hours | -| Whole classes appear as bare `native` declarations with no body | `code-virtualization-and-custom-linkers.md` | +| Whole classes appear as bare `native` declarations with no body | `java2c-and-jni-sinking.md` — read it **before** dumping memory: if this is Java2C there is no DEX to find, at any point in the process lifetime. A handful of `native` methods in an otherwise ordinary dex is JNI sinking, not this | +| A `Java_*` search over a hardened library returns nothing at all | `java2c-and-jni-sinking.md` §The JNI boundary — why a symbol search fails silently — dynamic registration, or `-fvisibility=hidden`. The check that works is "exports `JNI_OnLoad` and zero `Java_*`" | | A library's **SONAME does not match its filename** | `code-virtualization-and-custom-linkers.md`, `native-and-so.md` | | Your edit had **no effect at all**, with no error | `server-config-and-updates.md` §3 (the value may be server-sent), then `packers.md` §map the validation boundary | | Process **hangs** with no crash record, or dies to a `uid 0` killer | `native-tamper-and-suicide.md` §the rule (you probably made a terminate path *not return*) | @@ -336,6 +344,9 @@ unread in this repository. | The traffic is protobuf/gRPC/QUIC, or a proxy sees TLS but requests still fail on a Flutter app | `references/protocol-reverse.md` — schema-less protobuf, frame capture, and native-side pinning | | Userspace hooks land and the app still dies: the check reads `/proc/self/status` through a raw `svc`, or runs before `JNI_OnLoad` | `references/kernel-and-environment-hardening.md` — what the next layer up and down can actually do, and when to stop | | You must edit, repack, sign or inspect the APK **from the phone itself** | `references/on-device-tooling.md`, `scripts/mt_mcp_probe.py` | +| A captured body decodes to nothing readable, or you cannot tell whether a length-delimited field is a string, a nested message or a packed array | `protocol-reverse.md` §1. Protobuf on the wire (measured) — run `scripts/protobuf_decode_raw.py`; the candidate list and its `tie:` lines are the answer, and `docs/tool-verification/EXTENSION-protobuf-raw.md` records exactly what a schema-free decode can and cannot settle | +| Method bodies are present but decode as **private opcodes**, and you need the mapping rather than an explanation of why VMP is hard | `vmp-differential-analysis.md`, then `advanced-unpacking.md` for the shape diagnosis | +| A store build arrives as `base.apk` + `split_config.*.apk`, or a rebuilt build is refused **as a set** although every file verifies on its own | `split-apk.md` — one keystore across every member for `pm install-multiple`, and check that a merge is legal before trusting a merged single APK | ## Gates — clear these before you patch, in order @@ -556,6 +567,8 @@ Load only what the current step needs. | `references/byte-level-patching.md` | You want to change behaviour by editing a few bytes rather than rebuilding a method — equal-length patches, locating an instruction's exact offset, dex header integrity fields, branch polarity, verifier legality | | `references/packers.md` | The app is packed/hardened, or an edit makes it die before your code runs. Also load before discarding any route as "blocked by the shell" | | `references/code-virtualization-and-custom-linkers.md` | **No packer, dex is readable, and a re-signed build still dies** — whole classes turned into `native` declarations, a private loader with a mismatched SONAME, an embedded self-decrypting payload, or a Java-layer "signature killer" that logs success while a native check kills you. Covers the keep-it/drop-it deadlock and the string-redirect escape | +| `references/java2c-and-jni-sinking.md` | **Whole classes are bare `native` declarations and you are about to hunt for a decrypted DEX** — Java2C has none, ever: the code was compiled into a `.so`, so the route is reading that library, not dumping memory. Also the JNI boundary itself — why a `Java_*` symbol search fails silently — and the evidence that separates Java2C from an extraction shell, a VMP and ordinary JNI sinking | +| `references/vmp-differential-analysis.md` | Method bodies are present but decode as **private opcodes** — a real Dex VMP: the known-plaintext differential, which links can and cannot be automated, how to *prove* a derived opcode table, smali generation, and when the route is closed | | `references/framework-runtimes.md` | The UI is not native (Flutter / React Native / Unity / Cordova), or Java-layer hooks fire zero times while the UI clearly works | | `references/dart-aot.md` | The logic lives in a Dart AOT snapshot (`libapp.so`): pinning the Dart version, building a matching decompiler, the object pool and reference indexing, register/boolean conventions, locating and patching Dart code | | `references/native-and-so.md` | Patching in a `.so`, needing code to run before the app's own code, hand-built native payloads that crash inside the linker, or **deciding which library/ABI is actually loaded and executing** | @@ -586,6 +599,7 @@ Load only what the current step needs. | `references/protocol-reverse.md` | The traffic is protobuf without a schema, gRPC, or QUIC/HTTP3; or a proxy sees TLS while the app still fails — schema recovery, frame capture, and native-side certificate pinning (Flutter/BoringSSL) with its boundaries | | `references/kernel-and-environment-hardening.md` | Userspace hooking provably cannot reach the check — raw `svc` syscalls, `init_array`-early detection, a ROM that hunts instrumentation: what each layer up and down can still do, the kernel-route map and its version gate, and when escalating is the wrong answer | | `references/on-device-tooling.md` | Working **from the phone itself**: MT Manager edit/repack/sign and its built-in APK MCP, LSPosed Manager, Termux+frida, and on-device data inspection | +| `references/split-apk.md` | The target is a **split APK / App Bundle set** (`base.apk` + `split_config.*.apk`), or `pm path ` returned several files: reading a set off a device, when merging into one APK is legal versus when only unified re-signing works, the install refusal each mistake produces, the manifest attributes a merge breaks, and ABI/density matching | ## Script index @@ -608,7 +622,7 @@ All scripts are parameterized and path-agnostic; pass paths explicitly. Run `--h | `scripts/dart_pool_strings.py` | Recover string literals from a Dart AOT snapshot: framed entries, the one-byte vs UTF-16 split, file offsets, and a run-length noise filter | | `scripts/dart_disasm.py` | Annotated windowed disassembly of Dart AOT code (pool + boolean annotations) plus a B/BL caller index | | `scripts/find_refs.py` | Count and list callers of a method/field (blast-radius check). Takes a smali tree, a `.dex`, a directory of either, or an `.apk`. Always prints what it scanned, so "the input was unreadable" cannot be mistaken for "nothing references it" | -| `scripts/repack.py` | Rebuild an APK with replaced dex, strip only signatures, keep `META-INF/services/`, **write a 4-byte-aligned archive** (`resources.arsc` STORED+aligned, which Android R+ refuses to install without), sign, and verify | +| `scripts/repack.py` | Rebuild an APK with replaced dex, strip only signatures, keep `META-INF/services/`, **write a 4-byte-aligned archive** (`resources.arsc` STORED+aligned, which Android R+ refuses to install without), sign, and verify. Also handles **split APK / App Bundle sets**: inventory one (`--split-dir`/`--split`), sign every member with a single keystore for `pm install-multiple` (`--split-mode resign`), or fold code/native members into one standalone APK (`--split-mode merge` — refused by design when a member carries its own `resources.arsc`). `--signer auto/jar/apksigner` covers hosts without uber-apk-signer; see `references/split-apk.md` | | `scripts/dexpatch/` | dexlib2 method-level rewriter (for changes that genuinely need new instructions) + build notes | | `scripts/devsh.py` | Quoting-safe ADB shell helper for rooted devices | | `scripts/usb_net_proxy.py` | Give an offline device network over USB (adb reverse + local proxy) | @@ -638,3 +652,7 @@ All scripts are parameterized and path-agnostic; pass paths explicitly. Run `--h | `scripts/stalker_trace.js` | Instruction-level tracing with Frida Stalker: configurable module/offset targets, trigger selection, the event stream, output-size rules, and `transform` customisation | | `scripts/stalker_report.py` | Reduce a `stalker_trace.js` log into block histograms and call sequences, and print an explicit diagnostic for the measured **zero-event** case | | `scripts/mt_mcp_probe.py` | Probe MT Manager's on-device APK MCP (Streamable HTTP, `127.0.0.1:8787/mcp`): JSON-RPC handshake plus grouped `mt_apk_*` tool inventory; prints start-it-by-hand instructions and exits 2 while the service is down | +| `scripts/java2c_probe.py` | Collect the evidence that separates **Java2C** from an extraction shell, a VMP and ordinary JNI sinking: per-class `native` density, native-dominated classes and stub ratio from the dex, plus `Java_*` / `JNI_OnLoad` / dynamic-registration / toolchain-string evidence from the `.so`. Every item carries a strong/medium/weak label and the weak ones print their own warning — reading a single criterion is how this gets misdiagnosed | +| `scripts/protobuf_decode_raw.py` | **Schema-free protobuf decode**: hex string, binary file or stdin to a JSON tree, with every length-delimited field reported as a candidate set (nested message / UTF-8 / packed array / opaque bytes) and ties labelled rather than guessed. Flags the proto3 explicit-zero and packed-boundary ambiguities; `--max-depth`, `--split`, and `--reencode --check` for a byte-exact round trip. Cross-checked field-for-field against the official runtime | +| `scripts/vmp_diff_harness.py` | Differential hardening for Dex-VMP: build a labelled opcode-coverage fixture (**218 of 224 opcodes, measured**), derive a candidate private-opcode map from an original/hardened dex pair with per-entry confidence and a run-level verdict, forge known-table fixtures to verify the comparison in a closed loop, and render a restored stream as a smali skeleton. The upload-to-a-hardening-service link is **not automatable** — see `references/vmp-differential-analysis.md` §2 | +| `scripts/kernelsu_syscall_mask.py` | Generate a configurable KernelSU/APatch syscall-masking scaffold: an installable userspace module skeleton plus KPM/LKM/eBPF kernel-side templates, each with its version gate and an explicit unverified label. **A KernelSU userspace module cannot change a syscall return value** — the kernel-side templates are the part that could, and they are shipped unbuilt | diff --git a/skills/apk-reverse/references/advanced-unpacking.md b/skills/apk-reverse/references/advanced-unpacking.md index 5b4520f..a3da7f4 100644 --- a/skills/apk-reverse/references/advanced-unpacking.md +++ b/skills/apk-reverse/references/advanced-unpacking.md @@ -14,11 +14,13 @@ extraction shell and no VMP, so nothing here was exercised end to end against a hardened sample by this skill's own verification pass. The FART/Youpk mechanisms and the Android 12-16 failure analysis are **inferred** from public work (FART/Youpk release notes and a 2026-08 Kanxue thread, `thread-292312`, on why classic -active-invocation hooks die on modern ART). The one **measured** element is the -detection metric: the trivial-body statistic was exercised by -`scripts/dex_dump_validate.py` against a surgical fixture derived from a real hardened -sample (see `docs/tool-verification/EXTENSION-unpacking.md`). Label your own results -the same way. +active-invocation hooks die on modern ART). The detection metric has since been +**measured against a purpose-built extraction-shell fixture set**, and that pass +invalidated the threshold this file used to state — see the calibration table in +§Measuring extraction instead of guessing and the commands behind it in +`docs/tool-verification/EXTENSION-extraction-shell-bench.md`. The older surgical fixture +that first exercised the script is in `docs/tool-verification/EXTENSION-unpacking.md`. +Label your own results the same way. Two parts of this file have since been exercised against live hardened targets on a rooted device. The **root-side memory route** (§Dumping when frida is refused) was run @@ -28,7 +30,9 @@ traps and the throughput numbers are recorded in `docs/tool-verification/EXTENSION-rootdump.md`. The `frida-dexdump` refusal that motivates that section is in `docs/tool-verification/EXTENSION-device-run.md`. The FART/active-invocation half of the file is still **inferred** — no extraction-shell -target has been spliced end to end here. +target has been spliced end to end here, and the two candidate samples shipped in +`repos/CyReverse` turned out not to be extraction shells at all (their two "shell" +assets differ by six bytes; row B3 in the bench evidence file). ## The four shapes a dump can be @@ -39,9 +43,18 @@ directory and read the `stub%` column — the four shapes separate on it. |---|---|---|---| | Parses, many classes, real bodies | near-baseline | First-generation landing shell; the whole-dex dump **is** the original | Done — filter (`recon.md` §Unpacking a dex-level packer) and proceed to patching | | Parses, many classes, most bodies are `return-void` stubs or nop fills | high | **Extraction shell skeleton** — bodies are decrypted only on invocation | FART loop, this file | -| Whole classes appear as bare `native` declarations | n/a | dex2c / JNI sink — the code left the dex entirely | `code-virtualization-and-custom-linkers.md` | +| Whole classes appear as bare `native` declarations, and the matching `Java_*` export exists | n/a | **JNI sinking** — the code left the dex for the `.so` | `java2c-and-jni-sinking.md` — read the function in the library, **not** this file | +| Method bodies look like a thin forwarder to a native routine, or a single native island holds the app's own logic | n/a | **Java2C / native island** — the body is a compiled `J`/`.so` function, so re-reading the dex finds only the call | `java2c-and-jni-sinking.md` — the dex is the wrong instrument | +| Whole classes appear as bare `native` declarations, and **no** `Java_*` export exists | n/a | Dynamic-registration native sink — the binding table is built at runtime | `java2c-and-jni-sinking.md` §the symbol search fails structurally | | Parses, bodies present but instruction streams are nonsense to a dalvik disassembler | baseline | **Real Dex VMP** — private opcodes behind a native interpreter | Last section of this file, and be honest about the cost | +The two `native`-looking rows are **not** one row. They are separated because the +correct next action differs: JNI sinking keeps the logic in a named, exported ARM64 +function that you read from the library, while a dynamic-registration sink has no +symbol to find and needs the binding table read instead. Lumping them together sends an +analyst to dump memory for code that is sitting in an ordinary `.so` — the misdiagnosis +`java2c-and-jni-sinking.md` was written to stop. + Two supporting signals worth recording while you are here (the second is measured on this repo's hardened sample): a shipped shell `classes.dex` can be megabytes in size yet define only a handful of classes — the bytes are payload, not code, so class count @@ -69,12 +82,67 @@ measured to differ in ~73% of their bytes), rejects structurally impossible imag (bad magic, `file_size` mismatch, ids out of range), counts classes and stub bodies, and ranks which surviving image is the most likely original. -Calibration, **inferred** from field reports rather than measured across a population: -ordinary app code carries a few percent of genuinely trivial bodies (constructors that -do nothing, interface stubs); a ratio in the tens of percent or higher is a skeleton, -not code. A ratio that is high *and* the image parses is the extraction-shell -signature — a first-generation dump of the same app would show the same classes with -real bodies. +**Calibration — measured, and it is not a threshold.** Do not look for a `stub%` number +to compare against. On a real 982 KB application dex (5,061 bodies) every emptying +shape the shell family uses was built into a fixture and measured against a +zero-change control (`docs/tool-verification/EXTENSION-extraction-shell-bench.md`, +which carries the commands and the full matrix): + +| What the shell left behind | `stub%` | vs. control (1.9 %) | +|---|---|---| +| bare `return-void` in every body | 100.0 % | sighted | +| body cut to `const/4 v0,#0; return v0` | 93.2 % | sighted | +| nop fill, no return — **the slot-clearing shape** | 0.0 % | **blind, and scores below the untouched original** | +| `new RuntimeException; throw` stub — **the 360/legu shape** | 1.9 % | **blind, indistinguishable from the control** | +| 25 / 50 / 75 % of bodies emptied | 1.7 / 1.7 / 2.1 % | blind | + +The distribution is **bimodal**: `stub%` lands either at the app's own baseline +(ordinary constructors and interface stubs, measured at 1.9 % here) or at ~100 %. +Nothing sits in between, so there is no band a threshold could live in, and **partial +extraction is invisible** — a dex with three quarters of its bodies removed reported +0.2 points above its untouched control. + +Two consequences the metric's name hides: + +- **`stub%` high is dispositive; `stub%` low is not evidence of anything.** A low + reading is compatible with a genuine dex, a fully nop-wiped skeleton, and a + `throw`-stubbed skeleton alike. +- **Read `emptied%`, not `stub%`, for the skeleton call.** `emptied%` unions the + `return*`-stub and the nop-wiped classes, which is what lifts the nop-filled + skeleton from 0.0 % to 100 %. A `throw`-stubbed skeleton still reads low; for that + shape the ranking cannot help you and only per-body inspection can. + +The trivial-body shapes `dex_dump_validate.py` counts, and which of them are skeleton +evidence, are documented in the script itself (`classify_body`); the split between +`stub` / `erased` / `minimal` exists because collapsing them produced a ranking that +named a fully-wiped image the most likely original. + +### The ranking's failure conditions — read these before trusting its winner + +The `ranking` block names a "best-supported candidate for the original", and it is +wrong in two measured situations. Both were reproduced, and both are the *opposite* of +the obvious reading: + +- **A partially extracted image outranks a heavily stubbed one.** Ranking on the + emptied share alone put a dex with 25 % of its bodies removed *ahead of* its own + untouched control — 1.7 % against 1.9 %, because replacing a body with + `const/4 + return` lowers the `return-void` count. Measured before it was fixed; + the current key uses the count of skeleton-shaped bodies first, which is monotone, + and evicts anything over 50 % emptied. +- **A `throw`-stub skeleton is invisible and ranks at the control's own baseline.** + Every body replaced by `new RuntimeException; throw` measured 1.9 % — identical to + the untouched image — and it still sorts third in the fixture set. No body statistic + in this kit sees that shape. If the target family is 360/legu-like, **do not use the + ranking to pick a baseline**; diff candidate images against each other by body hash + instead. + +The reason to state this in the reference file rather than only in the script: the +ranking's output is a *sentence with a winner in it*, and a wrong winner is worse than +no winner — an agent that adopts a modified image as its baseline builds every +subsequent diff on the wrong artifact. When `emptied%` is at the app's own baseline and +you cannot name a signal that separates the candidates, treat the ranking as +uninformative for that set. Commands and full matrix: +`docs/tool-verification/EXTENSION-extraction-shell-bench.md`. ## The FART loop: skeleton, invocation, splice @@ -319,9 +387,46 @@ The last shape is the one to say out loud: if the dump's method bodies are intac decode as private opcodes — rare-opcode density off the charts, unknown opcode slots, and a large native interpreter loop behind them — the code has been virtualized, and nothing in this repository will hand it back to you decompiled. Distinguish it from -the JNI sink first (`code-virtualization-and-custom-linkers.md`): there the methods -are `native` stubs and the logic lives as native code; here the bodies are present -and *re-encoded*. +the JNI sink first (`code-virtualization-and-custom-linkers.md`, +`java2c-and-jni-sinking.md`): there the methods are `native` stubs and the logic lives +as native code; here the bodies are present and *re-encoded*. + +### Establish "the bodies do not decode" with a decoder you did not write + +This is the one measurement in this file that has already been got wrong here, so it +gets its own rule. A hand-written opcode-width table silently desynchronises on large +dex files — one arm64 branch opcode missing from the table is enough — and the +desynchronisation is indistinguishable from the finding it fakes. Measured on the +`ezAndroid` sample: a workbench table reported **1,393 of 22,424 bodies as +non-decodable** and I read it as private opcodes; the platform decoder returned +**943,223 lines, 34,566 instruction decodes, and zero structural errors** on the same +file (`docs/tool-verification/EXTENSION-extraction-shell-bench.md` §3.4). The bodies +were ordinary dalvik the whole time, and one of the "desync" methods disassembles into +five clean instructions under `dexdump`. + +So: **before any VMP verdict, decode the same file with `dexdump`, +`baksmali`/`smali`, or `jadx`, and quote the failure count that came from *it*.** A +rare-opcode histogram is a lead, not evidence. Never let a private table's desync +inform a classification — "my decoder stopped" is a fact about the decoder. + +### What a static dex probe can and cannot decide + +Two different questions live in the same dump, and only the first one is answerable +from the bytes on disk: + +| Question | Statically decidable? | Why | +|---|---|---| +| Are the bodies real dalvik, or an encoded program? | **yes** | the platform decoder decodes them or reports a structural error | +| Did the code leave the dex for native? | **yes** | `native` access flags, then `Java_*` exports / `JNI_OnLoad` in the matching `.so` | +| Does a native island check signature, root or debugger? | **no** | the logic is ARM64 behind a vtable; the check may exist in no dex at all | +| Are unlabelled `assets/` blobs live payload? | **no** | an unreferenced blob is dead until a runtime path constructs its name — measured: six such files in `ezAndroid`'s `assets/`, **zero** dex strings referencing them | + +The trap is the asymmetry. A clean static reading licenses exactly one sentence — +*"this is not dex-VMP"* — and it does **not** license *"this is not hardened"*. Those +are different claims: the second one is the misdiagnosis that sends an analyst looking +for an unpacker when the problem is a native island, or for a private disassembler when +the code is ordinary ARM64 in a 206 KB `.so`. A static probe's silence about hardening +is silence, not a clean bill of health. What recovery looks like when someone does it, **all inferred**, no fixture exists in this repo, treat as a research plan rather than a recipe: @@ -348,10 +453,12 @@ for the island itself. |---|---|---| | Dump parses, stub% near baseline | Filter, verify with jadx, move on to patching | `recon.md` | | stub% high, image parses | FART loop: skeleton + active invocation + splice | this file | +| `emptied%` high but `stub%` low | nop-wiped skeleton; the old `stub%`-only reading called it the original | §Measuring extraction instead of guessing | +| The ranking names a winner but `emptied%` sits at the app's baseline | Treat the ranking as uninformative for this set; do not adopt that image as a patch baseline | §The ranking's failure conditions | | Splice produces a half-parsing dex | Re-check code_item length (padding, handler list) | §The code_item length trap | | Dumper runs, output dir empty | Storage path + `open()` check, then entry-type probe | §Why the classic hooks die on Android 12-16 | | App dies under the invoker | Shell counter-measures; narrow the force list | §Why the classic hooks die on Android 12-16, and `detection-and-anti-analysis.md` | | frida attach refused (`script has been destroyed`) | Root-side `/proc//mem` dump, atomic inside one process lifetime | §Dumping when frida is refused | | Sample self-exits and restarts with no instrumenter present | The packer's own check, not your hook; shorten every step to one lifetime | §Dumping when frida is refused | -| Methods are `native` stubs | dex2c / JNI sink | `code-virtualization-and-custom-linkers.md` | -| Bodies present, decode as nonsense | Real VMP — differential oracle or walk away | §The honest boundary: real Dex VMP | +| Methods are `native` stubs with a `Java_*` export | JNI sinking — read the function in the `.so` | `java2c-and-jni-sinking.md` | +| Bodies present, decode as nonsense | Real VMP — **but confirm with `dexdump` first**, then differential oracle or walk away | §Establish "the bodies do not decode" with a decoder you did not write | diff --git a/skills/apk-reverse/references/emulation-and-rpc.md b/skills/apk-reverse/references/emulation-and-rpc.md index b55a9c3..0d547a3 100644 --- a/skills/apk-reverse/references/emulation-and-rpc.md +++ b/skills/apk-reverse/references/emulation-and-rpc.md @@ -47,6 +47,7 @@ Emulation and RPC are for when the code that matters must keep *running*. | Maintenance cost | every missing JNI/syscall is a new stub | reconnect logic, process lifecycle | | Reproducibility | deterministic (you control time/random) | real time, real randomness | | Best for | signing algorithms, crypto, parsers | anything entangled with live app state | +| **Priority when either could work** | second — the environment bill below is real and routinely underestimated | **first** — the device already owns the environment | The two-strike rule applies across the table: if a library refuses to run under unidbg after two focused rounds of `DalvikVM` patching, stop emulating and move to RPC (or the reverse) instead of a @@ -54,6 +55,33 @@ third round — a library that checks its own loading path or decrypts itself ag may simply not be worth emulating, and that is a finding, not a failure (`SKILL.md` §Stop conditions). +### The environment bill — budget days, not hours + +"Filling the environment" reads like a checklist and behaves like a project. The mistake this +section prevents is starting an emulation because the table says *works against anti-injection*, and +then discovering that this library's environment includes everything emulation cannot fake: an +Android `Context` backed by a real package manager, Binder round-trips into another process, a +`KeyStore` attestation that only succeeds on real hardware, or a self-check against device state +captured at install time. For a commercial native algorithm of that shape, "stub it until it runs" +is measured in **days, not hours** — each missing piece is discovered one fault at a time, and the +rounds do not get shorter. + +Weigh it before starting: + +| Question | If yes | +|---|---| +| Does it call through `Context` (package name, files dir, signature, `PackageManager`)? | a stub is often enough — cheap, keep going | +| Does it talk to another process (Binder service, bound SDK, remote provider)? | the stub surface grows fast; **prefer RPC** | +| Does it verify hardware (KeyStore, TEE, an attestation chain)? | emulation is likely a dead end — the value it wants cannot be produced here | +| Does it check its own loading environment (paths, maps, root, debugger, installer)? | readable and patchable *inside* the emulator, but budget one round per check | +| Must it be called thousands of times a minute? | the RPC throughput ceiling is real — this is the case that justifies the bill | + +Default to **RPC first, emulation second**: the device already provides the environment, and +per-call cost only starts to matter once you need volume. The two claims in this subsection +(the day-scale cost of a Context/Binder/KeyStore-shaped library, and the RPC throughput ceiling) +are **inferred** from the mechanism and from community practice — this repository has not emulated a +commercial sample end to end, as `EXTENSION-emulation-rpc.md` states. + ## Part A — Unidbg ### What it is diff --git a/skills/apk-reverse/references/java2c-and-jni-sinking.md b/skills/apk-reverse/references/java2c-and-jni-sinking.md new file mode 100644 index 0000000..8aa2f84 --- /dev/null +++ b/skills/apk-reverse/references/java2c-and-jni-sinking.md @@ -0,0 +1,149 @@ +# Java2C and JNI Sinking — the dex is not where the code is + +Load this when the dex shows method bodies missing or replaced by `native` declarations, and +you are about to go looking for a decrypted DEX in memory. **Decide which of the two shapes you +have before you spend anything**, because one of them has no DEX to find at any point in the +process lifetime, and the search for it is unbounded. + +Two neighbouring routes have historically folded this shape into one: `advanced-unpacking.md`'s +dump-shape table and the SKILL.md symptom index both point "whole classes are bare `native` +declarations" at `code-virtualization-and-custom-linkers.md`. That file is about protection which +leaves the code **reachable** — a private container, a loader, a private opcode interpreter, all of +which you can dump and measure. **Java2C is not reachable, because it was never bytecode.** This +file splits that row and owns both halves of it; the virtualization file stays what it is. + +**Strength note, read this first.** The measurements below are **observed** — every number was +produced by `scripts/java2c_probe.py` against a real sample during this pass, recorded in +`docs/tool-verification/EXTENSION-java2c.md`. But no Java2C library exists in this repository and +none could be built here (no NDK, no clang, WSL unavailable — see that record), so **the +Java2C-specific identification criteria are inferred, not observed end to end**. What *is* +observed is the discriminating measurement that separates Java2C from JNI sinking, and the JNI +boundary behaviour. Treat the type table as a map; label your own conclusions the same way. + +## The five shapes — one table, because the wrong row costs days + +| Shape | Static shape in dex | What is in memory at runtime | Route | Cost of the wrong row | +|---|---|---|---|---| +| **Landing shell** | `Application` is a third-party class; dex is a small stub | A full, decrypted DEX | Dump memory, filter, patch | Cheap if you notice the stub `Application` | +| **Extraction shell** | Reads as a normal dex until you measure it; bodies are stubs | A DEX whose bodies are filled in **on invocation** | Dump, measure `stub%`, FART-style active invocation | Medium — you keep dumping, and every dump is a skeleton | +| **VMP** | Dex parses; bodies are *present* but decode as nonsense | A DEX containing **private opcodes** interpreted by a native VM | Measure, then usually stop and report | Medium — recovery cost usually exceeds task value | +| **Java2C** | Whole classes are `native`; **no code_item at all** for them | **Never a DEX.** The translated methods exist only as compiled C in a `.so` | Read the `.so`; rebuild the JNI call chain | **Highest in this table** — you hunt a decrypted DEX that is never produced, then blame your dump tooling, then your anti-detection assumptions | +| **JNI sinking** | A handful of `native` declarations; the rest of the dex is ordinary Java | A normal DEX | Locate the Java call site; reverse one or a few native functions | Low — but you may widen the job to the whole app if you read it as Java2C | + +The one question that decides the table: **does the artifact ever hold dex bytecode for the +methods you care about?** + +- Landing shell, extraction shell, VMP: **yes** — the bytecode is there, in some state, and the + work is recovering or interpreting it. +- Java2C: **no** — a translated method is a `native` declaration plus an entry in a registration + table. Nothing is decrypted, so nothing can be dumped. `[inferred]` as a runtime claim: it + follows from the translation mechanism, and this pass did not observe a live Java2C app. +- JNI sinking: **yes** — the dex was never damaged. + +## Identification + +### The discriminating measurement is native density + +This is the single most useful number and it separates the two shapes by roughly three orders of +magnitude. Measured on real samples with `scripts/java2c_probe.py` `[observed]`: + +| Sample | dex methods | `native` methods | native ratio | +|---|---|---|---| +| JNI-sinking target A (MASTG L2) | 5081 | 2 | **0.04 %** | +| JNI-sinking target B (MASTG L3) | 11182 | 3 | **0.03 %** | +| App carrying a real JNI library | 23603 | 7 | **0.03 %** | +| Java2C-shaped fixture | 29 | 24 | **82.76 %** | + +A JNI *sink* is by definition surgical: a few hot methods are moved out and the rest of the app +stays in Java. **A JNI sink is not "the app is native".** When the density is in the tens of +percent, and whole classes have zero Java methods left, you are looking at a translation pass. + +`[observed]` in the same runs: two of the fixture's three classes were native-dominated (>= 70 % +of >= 3 named methods), against **zero** such classes in any of the three real targets. Note that +constructors are excluded from that test — a translation pass leaves ``/`` in Java +in practice, so "every method native" is an essentially unreachable test and should not be used. + +### Native-layer signatures + +| Evidence | Strength | Why | +|---|---|---| +| `.so` exports `Java_*` symbols whose count is roughly 1:1 with the dex native-method count | **strong** | Measured 2:2 (sample A) and 3:3 (sample B) `[observed]`. It is the signature of a *static-linkage* translation or sink. Note it does **not** distinguish Java2C from JNI sinking — only the density does | +| The `.so` contains one C function per Java method, each beginning `JNIEnv *env, jobject thiz, ...` | **strong** | Observed in generated, uncompiled output: every translated method became exactly one function with those leading parameters | +| Toolchain strings: `Dex2C`, `dynamic_register_compile_methods`, `ScopedLocalRef`, `well_known_classes` | **strong** | These are the runtime header and entry-point names a Dex-to-C toolchain emits. `[observed]` in generated sources; `[inferred]` that they survive compilation into the shipped `.so` | +| `JNI_OnLoad` present | **weak** | Present in nearly every JNI library. Measured true on a system library with zero `Java_*` symbols `[observed]` — it told us registration was dynamic, not that anything was hardened | +| `RegisterNatives` **string** anywhere in the `.so` | **weak** | See the boundary section: NDK's C++ headers do not emit this symbol at all | +| `libc++_shared.so` / `c++_static` | **weak** | An ordinary NDK C++ setting. The Dex-to-C toolchain read here does ship `APP_STL := c++_static`, but so do countless unrelated libraries `[observed]` | +| `classes.dex` byte-count per class far below the app's own pre-hardening build | **weak, uncalibrated** | Bodies leaving the dex should shrink it. Measured 584 B/class on the fixture versus 1152–1386 B/class on the real Java targets `[observed]` — but there is no population baseline for "normal", so this only compares an app against **itself** before and after | + +`java2c_probe.py` prints every one of these with its strength attached, and prints the weak ones +as an explicit warning block. A weak hit is not a verdict and must not be reported as one. + +## The route: read the .so, not memory + +For Java2C the analyst's instinct — dump memory and find the dex — is not merely expensive, it is +**unbounded**, because the success condition can never occur. + +1. **Do not dump.** There is no target artifact. If you dump anyway, you will recover a dex that + still shows the same `native` declarations you started with, and the natural (wrong) reading is + "the dump failed". +2. **List the translated functions.** With static linkage, `Java_____` + names map the `.so` back onto the dex one for one. Use the dex as the index and the `.so` as + the library. +3. **Rebuild the chain backwards.** A translated function calls *out* to Java constantly. The + strings `FindClass` / `GetMethodID` / `GetStaticMethodID` / `Call*Method` / `NewObject` / + `GetFieldID` inside it are the reverse edges. Following them reconstructs what the original + Java method did, which is the only way the logic comes back. +4. **Expect a register-machine shape.** Generated code declares a slot per dex register up front + and assigns them in sequence. Reading it as if a human wrote it wastes hours; read it as a + linearizer output, matching each call site against the JNI signature it passes. + +The decompiler choice matters less here than for obfuscated code: the generated functions are +long but flat and unoptimized, which is precisely the shape a decompiler handles well. + +## The JNI boundary — why a symbol search fails silently + +Two independent mechanisms erase the symbol that a reader would search for. Both are real; the +second was measured this pass. + +**1. Dynamic registration.** Nothing named `Java_*` is exported. Binding is performed at runtime +by a registration table built in a JNI entry point. A grep over the symbol table returns nothing, +exits zero, and looks like a legitimate answer. + +**2. `-fvisibility=hidden`, and an inline `RegisterNatives`.** The Dex-to-C toolchain read here +ships `APP_CPPFLAGS += -fvisibility=hidden` in its `Application.mk` `[observed]`, which hides the +generated functions from the dynamic symbol table — so even a statically-named translation can +present as "no `Java_*` symbols". Compounding it: NDK's C++ `jni.h` implements `RegisterNatives` +as an **inline member that calls through the function table**, so calling it emits *no* +`RegisterNatives` symbol either `[observed]`: a real system library exporting `JNI_OnLoad` and +zero `Java_*` symbols contains zero literal `RegisterNatives` symbols. A rule that greps for that +name is structurally incapable of firing. + +**The check that does work** `[observed]`: a library that **exports a JNI entry point and exports +zero `Java_*` symbols** is registering dynamically. That combination is affirmative evidence, and +it fired on a real library in this pass. + +**Confirming it at runtime.** Hook the registration entry point (`JNI_OnLoad`) and read the table +it builds, or hook the class-load path and enumerate the class's methods' native backing. Do not +conclude "no JNI boundary" from an empty symbol table; conclude "the binding is not visible in the +symbol table" and go find where it happens. + +## Failure modes + +| What you see | What it is not | What it is | +|---|---|---| +| Whole classes are `native`, you dump memory and find the same `native` declarations | "The dump failed" / "the tool is broken" | The translation moved the code out of dex permanently. Re-read the dex shape and open the `.so` | +| `grep Java_ .dynsym` returns nothing | "There is no native implementation" | Dynamic registration, or hidden visibility — see the boundary section | +| A library exports `JNI_OnLoad` and nothing else you recognise | "It is a loader" | It may be an ordinary dynamically-registering JNI library. A loader normally also has an `init_array` entry and a payload to map | +| Native density is high but every class still has Java methods | Java2C | Could be a JNI sink applied widely, or an R8-stripped app. Check whether the `native` methods have matching `Java_*` symbols before concluding | +| The dex looks completely unremarkable on a sample you were told is hardened | "Nothing is hardened" | A VMP whose payload is decrypted at runtime has an ordinary static dex. Measured this pass: a VMP-labelled sample had 7 native methods out of 23603 and a 4 % stub ratio `[observed]`. **Static dex metrics cannot see runtime-only hardening** — that needs a runtime check, not more reading | + +## Where this file stops + +- **It does not decompile the generated C.** That is ordinary native reversing; `native-and-so.md` + and `native-dbi-and-deobfuscation.md` own it. +- **It does not recover a VMP.** Private-opcode interpretation is a different mechanism with a + different cost model (`advanced-unpacking.md`). +- **The Java2C runtime claim is inferred.** That no dex bytecode exists in memory for a translated + method follows from the mechanism, not from a measurement made here. If you get a live Java2C + sample, that is the first thing to verify, and it is cheap: attach, force the class to load, and + ask whether the method resolves to Java bytecode or to a registered native entry. diff --git a/skills/apk-reverse/references/kernel-and-environment-hardening.md b/skills/apk-reverse/references/kernel-and-environment-hardening.md index a42286f..db56e62 100644 --- a/skills/apk-reverse/references/kernel-and-environment-hardening.md +++ b/skills/apk-reverse/references/kernel-and-environment-hardening.md @@ -19,6 +19,13 @@ externally documented mechanisms with sources, labelled `inferred` — none of i That is deliberate: the purpose is to stop you from spending a day rediscovering the version gate (§4) or the conflict (§5), not to teach kernel hacking. +**Extension pass, read the next paragraph before relying on it.** This file now has a companion +generator — `scripts/kernelsu_syscall_mask.py` — which emits a loadable userspace module skeleton +plus kernel-side **templates** for the three routes in §4. The generator, its userspace output and +its own consistency check are `measured`; **the kernel-side code has never been compiled or loaded +anywhere**, and every generated kernel file says so in its own header. A template is a starting +point that removes the blank page, not a weapon that has been fired. + ## 1. The attack-surface timeline — why "just hook it" fails An app process initializes in a fixed order, and every anti-analysis check you will meet lives at @@ -147,6 +154,83 @@ the process earlier than any userspace tool can be, needs no ptrace (so it does rewrites one `/proc` read is a research project; a Zygisk module that hooks the target's constructor is a build task. +### 4a. From "no weapon" to "template + gate": what the extension pass added + +The gap this closes is specific. §4 above was a map with no artefact behind it: a reader told +"Kernel module hooking — inline hooks and syscall-table hooks in kernel space — the layer that *can* +rewrite what a `/proc` read returns" had nowhere to go next. `scripts/kernelsu_syscall_mask.py` +produces the artifact; this subsection states its boundary, because the boundary is the part that +matters. + +**The correction worth internalising first: a KernelSU module cannot do any of this.** A KernelSU +(or Magisk, or APatch-userspace) module is a *userspace* module whose scripts run as root in the +normal world. `module.prop`, `post-fs-data.sh` and `service.sh` cannot change what `openat` +returns — not because of a version gate, but because nothing in that format is ever in the kernel's +return path. Detectors of the kind §2 and §3 discuss are not defeated by a module of that shape. +The generator emits that skeleton anyway, because it is the right carrier for the configuration and +the metadata, and it says in its own README what the skeleton cannot do. + +What *can* reach the return path is one of three artifacts, and each carries a gate that has to be +checked on the actual device: + +| Artifact | What it is | Gate | Status of the shipped template | +|---|---|---|---| +| **KPM** (KernelPatch / APatch) | A relocatable `.kpm` loaded by kpimg injected into the kernel image; replaces syscall-table pointers (`fp_hook_syscalln`) or rewrites prologues (`hook_wrapN`) | A KernelPatch-patched boot image, plus a **bare-metal** ARM64 toolchain (`aarch64-none-elf-gcc`) — not the NDK | `unverified` — template only, never compiled | +| **Out-of-tree LKM** | An ordinary kernel module that reaches `sys_call_table` and swaps a pointer | Kernel source matching the device's exact vermagic, and on ≥5.7 a way around `kallsyms_lookup_name` no longer being exported | `unverified` — template only; rated last of the three on cost | +| **eBPF probe** | A BPF program on a syscall tracepoint or kprobe | GKI 5.10+ with the tracing/BTF machinery | `unverified` — template only; the version gate alone closes it on most older devices | + +The eBPF row carries a capability ceiling that is easy to miss and is stated in the generated file +as well: **a tracepoint can observe a syscall, not rewrite its result.** `bpf_override_return()` +only applies to functions flagged `ALLOW_ERROR_INJECTION`, which raw syscall entries are not — +so "eBPF to spoof a `/proc` read" is, on most kernels, an observation plan wearing a rewrite +plan's clothes. Observation is genuinely useful (it tells you which syscall the check uses, which +`§5` requires you to know anyway); it is not spoofing. + +**Field notes on the KPM route, from a public KPM development write-up** (`inferred` here — the +author's measurements, not this repository's; source: blackr0ck, *APatch KPM 开发*, +[bbs.kanxue.com/thread-291665.htm](https://bbs.kanxue.com/thread-291665.htm), 2026-06). These are +recorded because each one is a day of somebody's time: + +- **An inline hook can install cleanly and never fire.** On a GKI kernel with LTO, the exported + symbol is frequently not the call site — the callee was inlined into its only caller, so + replacing instructions at the symbol's address intercepts nothing. The reported symptom is + "module loaded, callback never ran". Pointer replacement in the syscall table does not have this + failure mode, because the syscall entry path must go through the table. +- **The instruction-sequence length in a module header is a trap of its own.** The reported cause + of a hard-to-diagnose load failure: declaring a resolved kernel function with `extern` makes the + compiler emit an undefined reference (`*UND*`) instead of allocating the slot, and the loader + refuses the module with `unknown symbol`. Letting it be a tentative definition allocates `.bss` + and the loader fills it in. The generated template states this rule in the code, and + `verify` checks for it. +- **Per-syscall callbacks run on every syscall of that number, system-wide.** The reported + incident: a `write` hook that did not first reject `fd <= 2` intercepted the framework's own log + output tens of thousands of times and the device rebooted. **Performance is correctness here** — + the first statement in every callback is a filter that rejects the common case. +- **Which hook point survives is kernel-specific, and the cheap way to find out is to start at the + syscall layer.** The same write-up reports that modifying the SELinux internal function they + first targeted crashed the kernel under every combination tried (before/after, argument, + return value, skip-origin), while a syscall-table hook on `write` proved stable and sufficient. + Treat the syscall layer as the default and move inward only with evidence. + +**Measured on the reference device, and why every template here is labelled the way it is.** +`` runs kernel **4.14.186+** (`adb shell 'uname -r; cat /proc/version'` → +`4.14.186+`, `Linux version 4.14.186+ (nobody@android-build) (Android (6443078 based on r383902) +clang version 11.0.1 ...) #1 SMP PREEMPT Wed Mar 30 23:32:42 CST 2022`). On that device the two +remaining gates answer themselves: + +- **eBPF**: 4.14 is far below the 5.10 gate. Closed, as §4 already recorded. +- **KPM**: no KernelPatch-patched image is present, and the host has no + `aarch64-none-elf-gcc` (`kernelsu_syscall_mask.py gates` reports `NOT FOUND`, along with no + `ndk-build` and no `make`). Installing APatch means patching the boot image — a bricking risk + that this repository does not take on an APK task. +- **LKM**: no kernel source for the device, so no matching vermagic is possible. + +So on this machine the honest answer to "do you have a kernel-level weapon" is: **you have a +template, a gate table and a measured statement that the gate is closed here.** That is a route +decision, not a failure — and it is the same shape of answer §6 gives for the whole escalation +ladder. Do not let the existence of the template change what you claim about it: `unverified` is +still `unverified`. + ## 5. The syscall-level realities that survive every hook Two facts decide a lot of "unexplainable" behaviour, and both belong to the kernel's design @@ -178,7 +262,7 @@ honest outcome (`observed`, from `tools/_phone-modules/README-*.md`): | LSPosed (Zygisk) module hooking | available — framework activation verified by log (`welcome to LSPosed!` lines, `lspd` daemon process) | | MT Manager on-device editing/repack/sign | available (`on-device-tooling.md`) | | eBPF tracing | closed (kernel gate) | -| KPM / kernel module route | closed (no kernel source/build for the device; out of scope regardless) | +| KPM / kernel module route | closed on this device — no KernelPatch image, no kernel source, no bare-metal ARM64 toolchain; the **template and gate table** are shipped (§4a) | | Frida with disguised server | available (a renamed server binary was present on the device from prior work) | The generalisable rule: **enumerate the ladder for *your* device once, write the one-line @@ -198,6 +282,8 @@ defeated with a custom kernel module is not a deliverable anyone can install. | Check uses raw syscalls | Static patch of the check; or LSPosed/Zygisk module hooking the consuming code | "Hook libc open/read" (does nothing) | | Check detects the *environment* (root/emulator), not your patch | `detection-and-anti-analysis.md` A/B/C — usually a different device or route | Kernel work | | You are about to write a kernel module / patch a kernel | Stop. State what is blocked and the evidence; propose the static/module route | Kernel development (out of scope for this skill) | +| You have the kernel template and want to load it on a 4.14/Magisk device | Read `§4a`'s gate table, confirm the gate is closed, and say so | Patching a boot image to obtain a kernel route, on an APK task | +| A kernel-side template exists, so it feels like a weapon | It is `unverified` until a device with the matching gate compiles and loads it | Presenting a template as a working capability | | Escalation effort exceeds the user's actual ask | `detection-and-anti-analysis.md` stop signal — switch to static | One more layer | ## Checklist @@ -207,5 +293,8 @@ defeated with a custom kernel module is not a deliverable anyone can install. - [ ] Root scheme's artefacts enumerated (manager/daemon/mounts) before blaming the target - [ ] Hiding module: exactly one of Shamiko / Zygisk-Assistant; denylist configured to its spec - [ ] Kernel rung checked against `uname -r` — one line recorded if closed +- [ ] Kernel-side template treated as `unverified` until a device with the matching gate + actually compiles and loads it (`§4a`); the userspace module skeleton is a carrier for + configuration, not a kernel capability - [ ] Raw-syscall vs libc-call distinction verified by disassembly before building any spoof - [ ] No kernel development undertaken as part of an APK deliverable diff --git a/skills/apk-reverse/references/native-dbi-and-deobfuscation.md b/skills/apk-reverse/references/native-dbi-and-deobfuscation.md index 752b87d..e9de56d 100644 --- a/skills/apk-reverse/references/native-dbi-and-deobfuscation.md +++ b/skills/apk-reverse/references/native-dbi-and-deobfuscation.md @@ -182,6 +182,53 @@ across runs only for code that is mapped from file; do not compare a `libart`-ma address between two runs; and never place a follow/unfollow pair on a hot function (measured failure in §6). +### The cost is not linear, and on arm64 it decides the tool choice + +Two costs decide whether Stalker is the right tool, and neither is visible in the API: + +- **Translation multiplier.** A followed thread pays a large constant factor on every translated + block. Community reporting for arm64 puts it at **20-50x**; the mechanism is not controversial + (every block is copied into the DBI's code cache and routed through a dispatcher), while the exact + number is workload-dependent. This one is `inferred` — this repository has not measured the + multiplier itself, only its consequences below. +- **What you let into the trace.** Every block of every library the thread touches is a candidate. + Following into `libc` / `libart` / `libhwui` is how a four-second window becomes a device-wide + event: measured here, one such run took the test phone's `load average` to `34.11` on 8 cores and + destroyed a co-running job's attach (§6). + +`Stalker.exclude()` is the control for the second cost, and it is not optional. +`scripts/stalker_trace.js` applies it before `follow()` from its `excludeModules` list and reports +what it actually excluded on one `EXCL` line — read that line before believing any trace, because a +module that was not loaded at follow time was **not** excluded. Exclude every system library you are +not studying; the target module is never excluded even if its name appears in the list. + +Three failure shapes follow from ignoring this: deadlock, a watchdog `SIGABRT`, and the zero-event +trace. The deadlock and the watchdog shape remain community reports (`inferred`); the other two were +measured, **and the measurement splits the claim in two** — the split matters more than the summary: + +| Arm — one device, one package, one module, 6 s follow | Outcome | +|---|---| +| attach + resume, **no follow** | process survives | +| follow, `excludeModules: []` | **process dies**, script destroyed | +| follow, 20 system modules excluded | process survives, **but `blocks=0 blk=0 calls=0`** | + +Exclusion is what keeps the target alive — now `observed`, with the no-follow arm ruling out "it +would have died under frida anyway". But exclusion does **not** restore event delivery: the +zero-event trace survived the treatment arm unchanged. Treat those as two problems, and do not offer +`exclude` as the fix for a follow that delivers nothing. Commands and outputs: +`docs/tool-verification/EXTENSION-stalker-exclude.md`. + +**arm64 also raises the floor.** PAC/BTI-bearing code gives a translator more ways to mis-handle a +block than armv7 did, which is one more reason a trace that works on an emulator is not evidence +about a device. Treat any arm64 result as needing a control run (a followed thread executing a known +loop, nonzero `BLK` lines) before it is interpreted. + +**Emulation is a legitimate alternative, not a consolation prize.** When the target is a pure +computation inside a `.so` and its environment can be faked, a Unidbg/Unicorn trace can be both +faster and more stable than Stalker on a real arm64 device: no device load, no watchdog, no PAC, and +a deterministic replay. That inversion of the usual intuition is half the reason +`emulation-and-rpc.md` exists — decide with its decision table rather than by habit. + ## 5. From a trace to a deobfuscated function (inferred) This is the part the community write-up above also describes, and it is **inferred** here — the diff --git a/skills/apk-reverse/references/protocol-reverse.md b/skills/apk-reverse/references/protocol-reverse.md index d9c5ff2..e8509a3 100644 --- a/skills/apk-reverse/references/protocol-reverse.md +++ b/skills/apk-reverse/references/protocol-reverse.md @@ -55,9 +55,13 @@ Three consequences that decide how you interpret any decoded body: 2. **A packed repeated field is invisible without a schema.** Field 4 above is three varints (`3, 270, 86942`) packed into one length-delimited value. A schema-free decode can only report it as opaque bytes — the length is known, the element boundaries are not. -3. **An explicit zero is byte-identical to an absent field** in proto3 (`28 00` versus no bytes at - all). A decoded `0` is not evidence that the sender set the field, and "the server sent 0" is not - a safe conclusion from a decode alone. +3. **A decoded `0` and a field that was never set are two different questions.** An implicit + (no-presence) proto3 field whose value is the default writes **nothing at all**, so `field = 0` + and "never touched" produce identical bytes; a field **with** presence (`proto2 optional`, + `proto3 optional`) set to `0` does put its tag and a zero byte on the wire. Both halves were + measured against the official runtime; a schema-free decoder can only flag the case, never + resolve it. So a decoded `0` was emitted on purpose by some writer, and a field *missing* from a + decode is not evidence that 0 was the value in use. ### Decoding without a schema @@ -77,6 +81,63 @@ Three consequences that decide how you interpret any decoded body: covers reading these, and `datastore_inject.py` can re-encode an edited DataStore value once you know the message. +### The decoder that ships here (measured) + +`scripts/protobuf_decode_raw.py` walks a payload with no schema and prints a tree. Input is a hex +string, a binary file, or stdin (hex text and raw bytes are told apart automatically); `0x` prefixes, +spaces, commas, newlines and `\xNN` escapes are all accepted, and `|` splits independent frames. +`--json` emits the same tree as data, `--max-depth` bounds the nesting, and `--reencode` writes a +decoded (possibly hand-edited) tree back to bytes and can diff the result against the original. + +``` +$ python skills/apk-reverse/scripts/protobuf_decode_raw.py --hex "08 96 01 12 07 74657374696e67" +frame 0 bytes 0..12 (12 B) status=end_of_buffer + f1 varint 150 + f2 len-delimited(7) view=utf8_string + candidate utf8_string 0.50 valid UTF-8 with no control characters + candidate packed_varint 0.50 the payload is exactly a sequence of 7 varint(s) ... + candidate bytes 0.20 opaque bytes: always consistent with the payload ... + tie: packed_varint and utf8_string are equally consistent with these bytes ... + string: 'testing' +``` + +The point of the tool is the candidate list. Fed a nested message that `google.protobuf` 6.33.6 +itself produced (inner payload `0801120178`), it answers: + +``` +nested_message 0.50 | packed_varint 0.50 | utf8_string 0.35 | bytes 0.20 +tie: nested_message and packed_varint are equally consistent ... the view is a display default +``` + +Both readings really are legal protobuf for those six bytes, and no heuristic closes the gap. The +confidence numbers are a documented ordering, not probabilities, and the `view` exists only so the +tree can be expanded and re-encoded -- it is not a claim about what the field is. + +**Round trip, which is the strongest evidence available without a schema.** Decode, re-encode, and +compare bytes: the message serialized by the official runtime re-encodes byte-identically to +`SerializeToString()`; a real AndroidX DataStore file (81 B, written by `scripts/datastore_inject.py`) +round-trips byte-identically; and editing one value in the JSON tree and re-encoding produced a file +that the *other* tool read back as the new value. Two of the built-in fixture families are exceptions +by design and are reported as such rather than hidden: a non-canonical varint re-encodes to the +minimal form, and a `--split varint-length` run compares frame bodies only, since the length prefixes +are framing and not message data. + +**What a schema-free decode cannot conclude -- write none of these down:** + +- that a length-delimited field *is* a string, a submessage or a packed array, rather than that it + could be any of them; +- the element boundaries of a packed array, or its element type; +- that a field's value was 0 because it is absent from the decode, or that a sender "set" a field + because a 0 appears (see the two traps above); +- which message a field number belongs to, or that the same field number in two payloads means the + same thing; +- that a payload which parses cleanly is a real message at all -- §6 has that failure mode, and the + fixes are a second sample and a round trip, not a longer look at the first one. + +The measured record for this tool -- every command, both input forms, the round trip, and the +framing mistake it does *not* protect you from -- is in +`docs/tool-verification/EXTENSION-protobuf-raw.md`. + ## 2. Recovering the schema from the APK ### What generated protobuf code looks like @@ -253,6 +314,13 @@ their parents, and (c) when you have a schema, re-encoding the decoded structure with the input — the same round-trip check used in §1, which caught a decoder bug in this repository's own fixture. +**A framing choice that is wrong without being refused.** `--split varint-length` means protobuf's own +`writeDelimitedTo` framing: a varint length, then the message. gRPC frames a message differently — one +compression-flag byte, then a **four-byte big-endian** length (§3). Handed a real gRPC frame, the +varint splitter reported **five** frames rather than failing, while the same bytes decoded with no +splitting reported `stray_end_group`. Strip the 5-byte gRPC prefix yourself, and check the frame count +either way: a wrong split still produces a decode, which is what makes it expensive. + **Assuming the proxy sees everything.** The proxy sees what the client sends through it. An app that uses QUIC (§4), a raw socket, a native client bypassing Java's HTTP stack, or a certificate-pinned connection shows up as silence. Silence is evidence about the transport, not about the app: check diff --git a/skills/apk-reverse/references/split-apk.md b/skills/apk-reverse/references/split-apk.md new file mode 100644 index 0000000..ee7104c --- /dev/null +++ b/skills/apk-reverse/references/split-apk.md @@ -0,0 +1,322 @@ +# Split APKs and App Bundles + +A store build is frequently **not one file**. Play and most OEM stores ship App Bundles, and the +device receives a set: `base.apk` plus one `split_config.*.apk` per ABI, screen density and language +(and, for a dynamic-feature bundle, the feature modules as well). Every member carries **the same +package name and its own signature**, and the package manager treats the set as one package. + +Two failures follow from treating such a set as an APK, and both look like a broken build when they +are nothing of the kind: + +- Signing only the base leaves the set disagreeing about its certificate; the installer rejects the + whole set and names the package, not the member you forgot. +- "Just merge them into one APK" is free *only* for members that carry code or native libraries. A + member that carries resources cannot be folded into the base without merging `resources.arsc`, + which is a resource-compiler job. A wrongly merged archive installs and then renders the wrong + thing, or dies on the first resource lookup. + +`scripts/repack.py` handles both routes and **refuses to guess**: it inventories the set, decides +which route is legal from what the members actually carry, prints the reason, and only then acts. + +**Strength labels** (same three this repository uses everywhere): **observed** = reproduced with an +exact command and output; **inferred** = follows from observed facts, step not executed; +**unverified** = reported or assumed, not independently confirmed. The measurements behind this +file are recorded in `docs/tool-verification/EXTENSION-split-apk.md`. + +## 1. What the set is + +| Member | Manifest `split` attribute | Carries | Foldable into the base? | +|---|---|---|---| +| base | *(absent)* | all `classes*.dex`, the main `resources.arsc`, every resource not moved out, the real `` | — it *is* the base | +| ABI split | `config.arm64_v8a`, `config.armeabi_v7a`, `config.x86_64`, … | `lib//*.so` only | **yes** — entry names do not collide | +| density split | `config.hdpi` … `config.xxxhdpi` | its own `resources.arsc` **and** `res/` entries | **no** — requires a resource merge | +| language split | `config.en`, `config.zh`, … | its own `resources.arsc` (strings) | **no** — same reason | +| feature split | `feature_x` (+ `isFeatureSplit="true"`, `configForSplit="base"`) | its own `classes*.dex`, possibly resources | **code only**; its resources hit the same wall | + +**Observed** on a real 9-member set pulled from a stock Android 11 device (arm64-v8a target): + +- the base carried `classes.dex` (872 KB) and `resources.arsc` (71 KB); +- the ABI member carried two `lib/arm64-v8a/*.so` files and **no `resources.arsc` at all**; +- each density member carried its own `resources.arsc` and its own `res/` entries; +- a second, 8-member set's density members carried 39-78 `res/` entries each and 9-15 KB tables. + +So "which member owns the resources" is answerable **from the zip entry list alone** — no device, no +resource compiler, no decompiler. Copy a set to disk and ask. + +**A member can own a nearly empty table.** A split is *required* to carry a `resources.arsc`, so a +placeholder one is normal: on the same real set, eight density members carried a **40-byte** table +and no `res/` entries at all — they are placeholders, not resources. Treating "has an arsc" as "owns +resources" would block a merge that is perfectly legal. `repack.py` uses *entries under `res/` or a +table bigger than 1 KB* as the test. + +## 2. Getting the set off a device + +`pm path` prints **one line per member**; a package with several lines is a split set. This is how +you find one to copy, and it is also how you tell, before anything else, that the thing you are about +to patch is not a single APK: + +```bash +adb shell 'pm path ' +package:/system/priv-app//.apk +package:/system/priv-app//-arm64_v8a.apk +package:/system/priv-app//-xxhdpi.apk +``` + +To enumerate every split set on a device (toybox shell — no `--time-style`, no bash arrays): + +```bash +adb shell 'for p in $(pm list packages | cut -d: -f2); do c=$(pm path $p | wc -l); \ + if [ $c -gt 1 ]; then echo "$c $p"; fi; done' +``` + +Copy the whole directory out; the members live side by side: + +```bash +adb shell 'su -c "ls -la /system/priv-app//"' +adb pull /system/priv-app/ / +``` + +Two traps, both **observed**: + +- `adb pull ` creates `//` — the set arrives one level deeper than you expect. + `repack.py --split-dir` searches recursively for exactly this reason. +- A **third-party** split set is rare on a stock ROM. On the device measured here, `pm list packages -3` + returned **zero** split sets among ~40 third-party apps, while four preinstalled Google components + were split. If you need a real set and the third-party list is empty, that is the normal + distribution, not a mistake on your side. + +**Do not test by reinstalling a pulled set under its own package name.** It is already installed on +that device, owned by the vendor certificate you do not have, so the install fails for a reason that +has nothing to do with your pipeline (§6). Either work on a set you own, or rename the pulled one +(§7) to get an installable fixture. + +## 3. The decision: merge or resign + +Run this first — it writes nothing and answers with a reason: + +```bash +python skills/apk-reverse/scripts/repack.py --split-dir / --split-mode analyze +``` + +| Situation | Route | Why | +|---|---|---| +| Every non-base member carries only code and/or `lib/` (a placeholder `resources.arsc` is fine) | **merge** | nothing that needs a resource table moves | +| Any member carries `res/` entries or a real table | **resign** | a resource merge rewrites the table; see §5 | +| You must keep the build installable **and** the app is distributed as a bundle | **resign** | this is what the store does; the set stays a set | +| You need one file to hand someone, sideload, or feed to a tool that takes an APK | **merge**, if legal | the only way to get a single artifact | +| The delivery must survive store updates, or the ROM's own installer refuses multi-APK installs | **resign** (`pm install-multiple`), or merge and accept a fixed build | — | +| A member declares `isFeatureSplit="true"` and ships `classes*.dex` **and** resources | **resign** | its code could fold, its resources cannot; a half-merged build is worse than either route | + +Failure modes, one per route: + +- **merge**, when a resource-carrying member is dropped to make it legal: the app starts and shows + the base's resources. Missing density variants show up as blurry or missing images; missing + language splits show as untranslated strings. This is a **downgraded** build and has to be + delivered as one — `repack.py` prints `DROPPED n entry/ies` and refuses unless you pass + `--drop-split-resources`, so the loss cannot be silent. +- **merge**, when the base still declares `isSplitRequired="true"`: install succeeds and start is + refused (`INSTALL_FAILED_MISSING_SPLIT`, §6). The attribute has to be cleared first, and that is a + binary-AXML edit. +- **resign**, when one member is missed: the set fails as a set, with a signature error naming the + package. Sign the directory, not a list you typed by hand. +- **resign**, when the target ROM's installer is not the platform installer: `adb install-multiple` + can be intercepted by an OEM security centre, exactly as single-APK installs are + (`references/repack-and-sign.md` §Vendor install interception (OEM shells)). Use the root + `pm install-multiple` path. + +## 4. Unified signing — the route that always works + +```bash +python skills/apk-reverse/scripts/repack.py \ + --split-dir / --split-mode resign \ + --split-out-dir / \ + --ks key.jks --apksigner +``` + +Each member is de-signed (only the signature entries), rewritten as a 4-byte-aligned archive, and +signed with **one** keystore and **v1+v2+v3**; the run then verifies every member and prints the +signer fingerprint per file. The acceptance criterion is not "each file verifies" — it is **one +certificate across the whole set**: + +``` +[result] OK: 8 members, one certificate (sha256 1eb31d9c…2c44) +``` + +Then install the whole set, base included: + +```bash +adb install-multiple -r /*.apk +adb shell "su -c 'pm install-multiple -r /data/local/tmp/set/*.apk'" # when an OEM installer refuses the multi-APK form +``` + +**The signature must cover every member**, and `--abi`/density selection happens at install time, not +at signing time — sign them all, let the installer pick (see §6). + +### Toolchain: `uber-apk-signer` is optional, `apksigner` is not + +`repack.py`'s original signing route went through `uber-apk-signer` only, so a machine with a full +Android build-tools directory and no uber-apk-signer could not produce a signed build at all. The +script now picks the route itself: + +- `--signer jar` — the uber-apk-signer path (used automatically when the jar exists); +- `--signer apksigner` — `zipalign -p -f 4` then `apksigner sign --v1 --v2 --v3`, which is the whole + of what the jar was doing for us; +- `--signer auto` (default) — jar if present, else apksigner. + +`--apksigner` accepts all three shapes a build-tools install ships: `apksigner.bat`, the `apksigner` +shell wrapper, or the bare `lib/apksigner.jar`. **On Windows pass the `.jar` or the `.bat`** — the +extensionless wrapper is a shell script and will not execute. A `.jar` value is run through +`java -jar`, so `--java` (or java on PATH) is required for it. + +Order is load-bearing: **align, then sign**. `apksigner` adds signature entries without reshuffling +the archive; the v1 (JAR) path emulates `jarsigner`, whose rewrite of the zip would destroy the +alignment Android R+ requires of `resources.arsc`. + +## 5. Merging into a single APK + +```bash +python skills/apk-reverse/scripts/repack.py \ + --split-dir / --split-mode merge --out merged.apk \ + --abi arm64-v8a --ks key.jks --apksigner +``` + +What folds: `classes*.dex` (renumbered so a member's `classes.dex` becomes `classes2.dex`, keeping +the base's own dex intact), `lib/**`, `assets/**`. What does not, and why: `res/**` and +`resources.arsc`. + +**Why resource merging is not attempted.** `resources.arsc` is an indexed table: a global string +pool plus type-spec and type chunks, with every resource identified by a `(package, type, entry)` +triple and every string referenced by pool offset. Folding a member's table into the base's means +rewriting pool offsets, type/entry counts and the alignment of every string, in both directions — +that is `aapt2 link`, not a byte-level edit. The failure it produces is the worst kind: the archive +is *structurally* valid, installs, and then resolves resource ids to the wrong entries or to nothing. +An APK that renders garbage is harder to diagnose than one that refuses to install. + +If you genuinely need one standalone file for a bundle whose resources are split, the honest routes are: + +1. **`bundletool build-apks --mode=universal`** (the vendor tool, needs the original `.aab`) — this is + what merges resources correctly, and it is the only route that does. +2. **Keep the split set** and install it as a set (§4). +3. **Accept the downgrade** with `--drop-split-resources`, and say so in the delivery. + +**Manifest constraints when you do merge — and when you patch any split at all.** These attributes +are the difference between "installs and runs" and an install or start refusal, and they are easy to +miss because they live in binary AXML: + +| Attribute | Lives on | Effect if you get it wrong | +|---|---|---| +| `split="config.xxhdpi"` | `` of a **split** | Identifies the member. A split whose `package` differs from the base's, or whose `split` name is duplicated, is refused as a set | +| `configForSplit="base"` | `` of a split | Ties a configuration split to a feature module. Changing package/module names without updating it silently detaches the split | +| `isFeatureSplit="true"` | `` of a feature split | Marks an install-time/on-demand module. A merge that folds code in but leaves the attribute declared describes a member that no longer exists | +| `isSplitRequired="true"` | `` of the **base** | The platform refuses to start a base that believes it is incomplete. This is the one that breaks merged builds (§3) | + +Read them without a decompiler through `scripts/repack.py`'s own AXML reader (`--split-mode analyze` +prints the `split` name per member; `inspect_apk()` returns `package`, `split`, `configForSplit`, +`isSplitRequired`, `isFeatureSplit`). Changing any of them means editing the binary string pool and +the attribute chunks around it, which is what `apktool` is for — and note that a full +`apktool b` rebuild rewrites the whole archive +(`references/repack-and-sign.md` §Repacking an unpacked (de-shelled) app). + +## 6. Install refusals, and what each one means + +| Message | What it actually says | Attribution | +|---|---|---| +| `Failure [INSTALL_FAILED_INVALID_APK: ... signatures do not match previously installed version]` / `Package signatures do not match` | The set you are installing is signed with a different certificate than the one already on the device | **Not** a malformed build. Either re-sign to the original key, or install to a package name that does not exist yet | +| `Failure [INSTALL_FAILED_ALREADY_EXISTS]` / `INSTALL_FAILED_UPDATE_INCOMPATIBLE` | Same package, different signer (or a downgrade without `-d`) | A pulled system component cannot be reinstalled over itself; see §7 | +| `Failure [INSTALL_FAILED_MISSING_SPLIT: Missing split for ]` | The base declares `isSplitRequired="true"` and the set is incomplete — or you merged it | The base is telling the truth: it expects members. Clear the attribute, or install the complete set | +| `Failure [INSTALL_FAILED_INVALID_APK: Failed to parse ... split ... has different signature]` | **One member** of the set is signed differently from the rest | This is the failure mode of signing only the base. Sign the whole directory | +| `Failure [-124: ... resources.arsc ... stored uncompressed and aligned on a 4-byte boundary]` | One member's `resources.arsc` is compressed or unaligned | Per-member, not per-set. `repack.py` reports it per member before signing (`references/repack-and-sign.md` §2a) | +| `Failure [-99]`, or a numeric-only code with no `INSTALL_FAILED_*` | An OEM installer interceptor rejected the request | Nothing to do with the set; use the root `pm install-multiple` path | +| `adb install-multiple` on a device that has no `install-multiple` support (very old `pm`) | Install each member with `pm install` in order, base last | Use `pm install-create` / `install-write` / `install-commit` explicitly | + +`INSTALL_FAILED_INVALID_APK` in particular is a **family**, not one error: it covers stale +signatures, malformed base/split relationships, and alignment violations. Read the text after the +colon before attributing anything — the constant alone will send you looking at the wrong file. + +## 7. Making an installable fixture from a pulled set + +**Observed, and the reason this section exists:** a set pulled from the device is installed *on that +device*, under a name owned by the vendor certificate. Re-signed copies cannot be installed back +over it, so "does our signed set install?" cannot be answered with the pulled set as it stands. + +The cheap, structure-preserving way out is an **equal-length package rename**: replace every +occurrence of the old package name in `AndroidManifest.xml`, `resources.arsc` and `classes*.dex` +with a new name of exactly the same length. Nothing moves — no length prefix, no string-pool offset, +no dex `string_ids` offset, no resource index — so the only field that must be recomputed is the dex +header's integrity pair (signature first, checksum last, see +`references/repack-and-sign.md` §Repacking an unpacked (de-shelled) app). Then run the ordinary +`--split-mode resign` over the renamed directory and install that. + +A detail worth knowing before you touch a manifest: **a member's Java classes need not live under its +manifest `package` at all**. On the real set measured here, the manifest package was +`com.google.android.` while all 677 class references in the dex were +`Lcom/android//...` — and every `` was fully qualified. Renaming the +manifest package therefore left every component resolvable, with no dex edit needed. Assume the +opposite (that the two must match), and you will edit a dex for nothing. + +## 8. ABI and density matching + +The device tells you what it can run: + +```bash +adb shell getprop ro.product.cpu.abilist # e.g. arm64-v8a,armeabi-v7a,armeabi +adb shell getprop ro.product.cpu.abi # the primary ABI +adb shell getprop ro.product.cpu.abilist32 # 32-bit list, on a 64-bit device +``` + +Match against the **split name**, not the file name, and mind the separator: split qualifiers use +**underscores** (`config.arm64_v8a`) while `lib/` directories use **hyphens** (`lib/arm64-v8a/`). The +mapping is a substitution, not a lookup — `arm64_v8a` → `arm64-v8a`, `armeabi_v7a` → `armeabi-v7a`; +`x86` and `x86_64` are unchanged. That is why `--abi arm64-v8a` matches a split named +`config.arm64_v8a`. + +Four rules that decide ABI questions: + +- **A 64-bit device still installs a 32-bit-only set**, and then runs it 32-bit. If the base carries + no native code and only the `armeabi_v7a` split does, the app is a 32-bit app on that device. +- **The ABI split is selected by the installer, not by `repack.py`.** `--abi` exists for *merging* + (where the choice is baked into one file); for `--split-mode resign`, sign every member and let + `pm` pick. +- **Merging without `--abi` folds in every ABI member**, which makes one APK with two or three copies + of the same library set. Legal, larger, and pointless on a single-device delivery. +- **`getprop` reports what the device claims**; what runs is a live mapping question + (`scripts/lib_map.py`). A device that reports `arm64-v8a` first can still execute a 32-bit process. + +Density is the same shape without the separator trap: `config.xxhdpi` matches +`res/drawable-xxhdpi/`, and the installer picks the closest variant the device's `ro.sf.lcd_density` +can use. There is nothing to select on your side. + +## 9. Traps worth paying for once + +- **`resources.arsc` belongs to whoever ships it.** In a split set it exists *per member*, and the + base's table does not contain the density member's entries. Reading the base's table and concluding + "the resource exists" is wrong; it exists in one configuration only. +- **`android:extractNativeLibs`.** If the base declares `extractNativeLibs="false"` (the default for + modern targetSdk), the `.so` files are mmapped straight out of the archive, and they must be + **uncompressed and page-aligned** — this is what `zipalign -p` exists for. Merging native members + into a base and then writing the archive without that alignment produces an install that succeeds + and a start that dies in the linker. `repack.py` writes uncompressed `lib/**` 4-byte aligned, and + `zipalign -p -f 4` adds the page alignment the loader wants for them. +- **A placeholder `resources.arsc` is not a resource split** (§1). Blocking on "has an arsc" refuses + legal merges. +- **Do not `zipalign` after signing to "fix" a member.** It rewrites the archive and invalidates the + signature. Align, then sign. +- **`apktool` may not be usable even when a wrapper exists.** Measured on the pass machine: the + `apktool.bat` wrapper pointed at a jar that was not on disk + (`Error: Unable to access jarfile .jar`), while `apksigner.jar` and `zipalign.exe` in the + same layout were fine. Check the wrapper's target before planning a manifest edit around it. +- **The AXML `split` attribute is a string-pool entry**, which is why an equal-length rename of it is + safe and why a length-changing edit is not: the pool's own length prefixes and the attribute chunk + sizes around it would both need recomputing. + +## 10. What this file does not claim + +- **It does not merge resources.** No route here reproduces `bundletool build-apks --mode=universal`. + A bundle whose resources live in splits stays a set, or becomes a downgraded single APK by explicit + choice. +- **It does not split a monolithic APK back into modules**, and it cannot turn a `.aab` into APKs — + that is `bundletool`, and the `.aab` is the input it needs. +- **`--split-mode merge` covers code and native libraries only** (§5). Feature-split *code* folds; + feature-split *resources* do not. +- The install-failure texts in §6 are the shape the platform produces; **the exact wording varies by + Android version and by OEM installer**. Read the text after the constant, not the constant. diff --git a/skills/apk-reverse/references/vmp-differential-analysis.md b/skills/apk-reverse/references/vmp-differential-analysis.md new file mode 100644 index 0000000..278a0ca --- /dev/null +++ b/skills/apk-reverse/references/vmp-differential-analysis.md @@ -0,0 +1,237 @@ +# Differential hardening (the known-plaintext route to a private opcode table) + +Load this when the dump is a **real Dex VMP** — bodies present, decoding as nonsense to a +dalvik disassembler, a native interpreter loop behind them — and you have decided the +private opcode space is worth attacking rather than walking away from. The neighbouring +file owns the diagnosis (`advanced-unpacking.md` §The honest boundary: real Dex VMP) and +the sentence that sends you here. This file owns what happens next, and — more of it than +is comfortable — what does not. + +**Strength note, read before trusting any number here.** Every mechanism below was +exercised on locally built fixtures, and the closed loop is measured: `simulate` relabels +a fixture through a known bijection and `compare` re-derives that table **exactly, 218 of +218 emitted opcodes, zero wrong entries, zero fabricated entries** (`docs/tool-verification/EXTENSION-vmp-diff.md`). +What was **not** exercised is the other half of the premise: **no third-party hardening +platform was contacted**. The assumption that a real engine performs a per-opcode +substitution at stable instruction length is **inferred** from public VMP write-ups and is +exactly the assumption this method can fail on. The cost table's platform column is +therefore reasoning from how those platforms are distributed, not from a measured upload. + +## 1. The method, and the one thing it needs to be true + +The hardened body cannot tell you where its instruction boundaries are: its opcodes are +private, so the format specification's length table does not apply to it. That is the +whole obstacle — a disassembler that cannot decide where instruction *n* ends cannot name +instruction *n+1*. + +The differential dissolves it with an oracle you control. Compile a fixture whose every +instruction you know; get the same platform to harden it; then **take the original's +instruction boundaries and project them onto the hardened bytes**. Original offset +`0x14` was a `mul-int` occupying 2 code units, so hardened bytes `0x14..0x17` are the same +instruction, re-encoded. Read the hardened opcode byte at each projection and you have a +`(original opcode → private byte)` sample. Aggregate over thousands of instructions and +the substitution table falls out. + +Three preconditions, and they are not negotiable: + +| Precondition | Why it must hold | What it looks like when it fails | +|---|---|---| +| **The hardened body keeps the same length** | The projection is a stencil; a length change slides every boundary after the first edit | `compare` reports `resized` for the whole method and refuses the run | +| **The substitution is per-opcode, not per-method** | One private byte per original opcode is what makes the table a table | Every original opcode claims the same private byte (`compare` → `NOT-USABLE`, stub shape) | +| **The body is still in the dex** | Alignment needs two byte streams | Method has `code_off == 0` (`compare` reports `stripped`) | + +The third is the one that kills the method most often in practice: a VMP that *extracts* +bodies to a native interpreter leaves nothing to align. The differential route then has no +input at all, and no amount of clever fixture design creates one. Check for it before +spending any time here. + +## 2. Cost table — which links can be automated, and which cannot + +This is the part the method's public descriptions leave out. The chain is six links; two +of them are unavoidably human, and pretending otherwise is how a "harness" ships that has +never once been run against a real target. + +| # | Link | Automation | Where a human must stand | Time cost | Failure mode | +|---|---|---|---|---|---| +| 1 | Compile the coverage fixture into a dex + APK | **Full** — `scripts/vmp_diff_harness.py build`, measured | none | ~40 s on this machine (javac + d8 + aapt2), ~1 s of it the script | toolchain absent or version-drifted; `d8` rejects a non-existent `--output` directory, `aapt2 link` rejects `android:` attributes without `android.jar` | +| 2 | **Submit to the hardening platform** | **None. Not automatable.** | 100 % — a browser, an account, sometimes a queue and a review | minutes to **days**; the platform decides | **the binding constraint of the whole route.** No public API; per-account rate limits; uploads are frequently gated behind registration; the artifact often comes back as a download page rather than a file path. A synthetic fixture may simply be **refused** (no real app shape, no certificate, no package identity) | +| 3 | Retrieve the hardened output and extract its dex | Partial — if the output is an APK, `unzip` + `dexutil.load_dex` handles it | download and hand-off | seconds, once the file exists | output is an APK whose dex is *also* encrypted, or the platform returns only a repacked APK with a native loader and no readable dex | +| 4 | Derive the correspondence | **Full** — `compare`, measured end to end | none | **sub-second** on a 3,065-instruction fixture | alignment slips (see §4); engine is not a pure relabelling | +| 5 | Prove the derived table | Partial — the mechanical half is scripted and measured; the judgement is not | decide whether the held-out check was passed or merely not attempted | minutes | a table that is *self-consistent* but describes a stub (§5) | +| 6 | Render Smali from the restored stream | Full — `emit-smali`, measured | none | sub-second | table has holes; restored stream decodes short of the method end | + +**Conclusion, stated plainly: do not build a fully automatic pipeline.** Link 2 is +irreducibly manual — the platform has no interface a script can drive, and its output +cadence is not one a pipeline can wait on. The right shape is what this repository ships: + +- links 1, 4 and 6 as **reliable local tools** (they are deterministic and were measured); +- link 5 as **a checklist plus one scripted test** (§5) — the judgement stays human; +- link 2 as **a documented manual step**, with its cost stated up front, so nobody writes + an orchestrator that waits forever on a web form; +- and links 4–5 **self-verifiable without the platform at all**, which is what `simulate` + exists for. You can prove your comparison logic against a known table today; you cannot + prove the engine's behaviour until a human uploads a fixture. + +The practical consequence: the deliverable of this route is an **analysis report with a +stated boundary**, not a pipeline. Say so before starting. + +## 3. The coverage fixture — what a compiled probe can and cannot reach + +A Java-compiled fixture reaches **218 of the 224 named opcodes in this repository's format +table (97.3 %)** — that number is measured, not estimated, and the six it misses are +structural rather than incidental. The reason to care is direct: **an opcode your fixture +never emits leaves no known plaintext, so its substitution is unreadable by construction.** + +| Missed opcode | Why a javac + d8 fixture cannot reach it | +|---|---| +| `const-string/jumbo` (0x1b) | needs a `string_ids` index > 65535, i.e. a fixture shipping 65,536+ string constants | +| `goto/32` (0x2a) | needs a >32767-code-unit backward jump; javac refuses the method first with `error: code too large` — its own 64 KB per-method bytecode cap. **Measured** on a generated 33,000-statement loop | +| `const-method-handle` (0xfe), `const-method-type` (0xff) | no Java-language literal exists for these constants; reachable only via hand-written smali or direct dex construction | +| `move-object/16` (0x09) | needs **both** registers ≥ 256; the object frame pushes the source past 256 but d8 encodes that as `move-object/from16` | +| `invoke-custom/range` (0xfd) | d8 emitted the 35c form for every lambda shape tried, including a six-parameter one | + +What the fixture **does** cover, by construction, and the design decisions that get it +there (each of these was a measured gap before it was closed): + +- **Three-register forms.** javac emits the `/2addr` form whenever the destination is one + of the operands, so `sub-int`, `div-int`, `rem-int`, `and-int`, `or-int`, `xor-int`, + `shl-int`, `shr-int`, `ushr-int` and their long/float/double counterparts are **absent** + from ordinary code. Three distinct locals per operation is what forces them into the + stream; the `/2addr` forms then need the accumulator spelling. Both are wanted. +- **Literal widths.** `div-int/lit16` needs a divisor in 128..32767; `and-int/lit8` needs a + mask inside −128..127. Writing `& 255` produces `and-int/lit16` and silently skips the + lit8 form. +- **Per-type field access.** `iget-wide`, `iget-object`, `iget-boolean`, `iget-byte`, + `iget-char`, `iget-short` and the `sget` set all appear only when the load targets a + **fresh local**. Accumulating into an existing register is where d8 narrows the access. +- **`/range` invokes.** The 35c invoke forms carry at most five argument registers, so a + seven-register call (`this` plus six ints) is what makes the `/range` encoding the only + legal one. +- **Wide registers and long jumps** — `move/16`, `move-wide/16`, `move-object/from16`, + `goto/16` — have **no hand-written form at all**. They exist only past a register-number + (≥256) or branch-distance (>127 code units) threshold, so the generator emits bulk + source: 300 int locals, 140 long locals, 160 object locals, a 200-statement loop body. + Measured yield: `move/16` ×46, `move-wide/16` ×20, `move-object/from16` ×161, `goto/16` ×1. +- **Payloads and structure**: `packed-switch`/`sparse-switch` with their payload blocks, + `fill-array-data`, `filled-new-array/range`, try/catch ranges with `move-exception`, + `monitor-enter`/`monitor-exit`, `instance-of`/`check-cast`, and `invoke-custom` / + `invoke-polymorphic` (which is why `d8 --min-api 26 --no-desugaring` is the default — + desugaring replaces lambdas with synthetic classes and the two opcodes disappear). + +**Report the coverage every time.** `build` prints it, `audit` recomputes it for any dex. +A differential run whose coverage is 60 % produces a table that is 40 % missing, and +nothing in the output will say so unless you look at this number. + +## 4. Extracting the correspondence + +`compare ORIGINAL HARDENED` does four things, and the first three are diagnostics that +decide whether the fourth is meaningful: + +1. **Pair methods** by `(class, name, descriptor)` and classify each pair. `aligned` means + both sides have a `code_item` of **identical `insns_size`** — the only state in which + projection is legal. `resized` means the length moved. `stripped` means the hardened + method has `code_off == 0`. `missing`/`added` mean the method set itself moved. +2. **Project and sample.** For each aligned pair, walk the original's decoded instruction + list; for instruction at original offset *o*, read the hardened byte at + `hardened.insns_off + (o − original.insns_off)`. Record `(original opcode → hardened + byte)` with a count. +3. **Aggregate and audit the aggregate.** Per original opcode: one candidate byte and the + row is `high`; more than one and the row is a `conflict`. Then the **reverse + direction** — build `private byte → set of original opcodes` and flag every private byte + claimed by more than one original. This reverse pass is not a nicety; it is the only + thing that catches the stub shape (§5). +4. **Emit a run-level verdict**, because a table is dangerous without one: `usable`, + `partially-usable`, `not-usable`, or `not-applicable` with the reason. A `not-applicable` + run exits non-zero. + +Unreadable opcodes are reported as `undetermined` with the reason — either the fixture +never emitted them (§3) or the dex slot is unallocated. They are **not** guessed at. + +## 5. Proving the table — the part that decides whether any of this is real + +A candidate table is a hypothesis. Four checks, in increasing order of what they can +falsify: + +**a. Closed-loop regression — measured, and the strongest available without a platform.** +`simulate` relabels a fixture's opcodes through a bijection whose ground truth it writes +out; `compare` must re-derive that table from the two dexes alone. Measured on this +repository's fixture: **218 rows derived, 218 of 218 emitted opcodes recovered, zero +entries wrong, zero entries fabricated, zero non-injective private bytes, verdict +`usable`.** Any change to the comparison logic that breaks this check is a regression, and +the check needs no platform, no device and no network. + +**b. Injectivity — the check that catches a confident wrong answer.** Every private byte +must be claimed by exactly one original opcode. Measured against the stub shape (bodies +replaced by an equal-length `return-void` fill): per-opcode reading looks *perfect* — 218 +rows, all `high`, zero conflicts — because every original opcode maps to the same byte. +Only the reverse direction exposes it: one private byte claimed by **218** originals → +`NOT-USABLE`. A table produced without this check would have been silently worthless. + +**c. Held-out consistency.** Derive the table from one subset of methods, then use it to +disassemble a **different** subset that contributed nothing to the derivation. A correct +table restores the instructions such that the walk lands exactly on +`insns_off + insns_size × 2` and every branch target lands on an instruction boundary. A +wrong or incomplete table desynchronises and says so. `emit-smali`'s +`instructions restored` / `unmapped-opcode stops` counters are the readout: on the measured +fixture, 3,065 restored and 0 stops. + +**d. Cross-check against a second, independent recovery.** The same private opcode space +observed through a *different* route — an in-memory dump of the original dex alongside the +hardened one, or the `/proc//mem` route in +`advanced-unpacking.md` §Dumping when frida is refused — is the only evidence that the +platform's transformation is the one you assumed. **Not done here**; it requires a real +VMP sample and a real platform, and is the honest gap in this file. + +Note what none of these checks establish: that the engine's substitution is *stable across +inputs*. Fixture-derived tables have been reported in public work to drift per +vendor-version and per hardening pass. Re-derive per target; never cache a table. + +## 6. Generating Smali from a restored stream + +`emit-smali HARDENED_DEX --table TABLE.json` restores opcode bytes through the table and +renders method bodies as an annotated smali **reading skeleton**: + +- Boundaries come from the table itself — once a byte maps back to a standard opcode the + standard length table applies to that instruction, so the walk is self-bootstrapping. + A byte with no table entry stops that method's walk, and the count of such events is + printed as the honest coverage measure. +- `.class` / `.super` / `.source` / `.method` headers are real, with access flags decoded + from `access_flags`, so the skeleton is readable as smali rather than as a hex dump. +- Payload blocks (`packed-switch-payload` etc.) are stepped over by their own layout, not + treated as opcodes. + +**It is explicitly not assembliable, and the output says so in its first ten lines.** +Recovered: instruction boundaries and opcode names. Not recovered: register numbers, +field/method/string indices as the engine rewrote them, and any register re-allocation — +**no map for those exists in this method at all.** Feeding the skeleton to a smali +assembler produces an empty method, which looks like success and is not. + +## 7. When this route does not work — the honest retreat + +| What `compare` reports | What it means | Do this | +|---|---|---| +| `verdict: not-applicable`, `stripped` > 0 | bodies left the dex for a native interpreter | the differential has no input. Stop. Return to `advanced-unpacking.md` §The honest boundary: real Dex VMP and price the island honestly | +| `verdict: not-applicable`, `resized` > 0 | the engine's stream has its own length table | the projection premise is false. Stop rather than guessing boundaries | +| `verdict: not-usable`, one private byte shared by many originals | bodies replaced by a common stub | the engine did not relabel per-opcode here. Stop | +| `verdict: partially-usable`, conflicts ≤ 30 % | partial relabelling, or alignment slipping on some methods | usable only for the `high` rows; say so and stop at the unreadable ones | +| `usable`, but coverage was low | a complete table over an incomplete opcode set | usable, with the uncovered opcodes named as unreadable | + +And the retreat that outranks all of them: **the cheaper question is whether the behaviour +you need exists outside the virtualized island.** A hook at the method's Java boundary, an +emulated call to the routine rather than a reading of it +(`emulation-and-rpc.md`), or a server-side answer frequently reaches the goal without ever +paying for the island. Weeks of table recovery is the expensive way to ask a question a +boundary hook answers in an hour. + +## 8. Decision summary + +| Observation | Action | Where | +|---|---|---| +| Bodies present, decode as nonsense | Diagnose real VMP before anything else | `advanced-unpacking.md` §The honest boundary: real Dex VMP | +| Need the private opcode mapping | Build the coverage fixture, then submit it by hand | `scripts/vmp_diff_harness.py build`; §2 of this file | +| Asked to "automate the whole chain" | Say which link cannot be automated and what it costs | §2 — the platform step is manual, by construction | +| Fixture coverage below ~90 % | Add the missing opcode classes before submitting | §3, and the `audit` report | +| Two dexes in hand | Derive the candidate table, read the verdict first | `scripts/vmp_diff_harness.py compare` | +| About to trust a derived table | Run the closed loop, the injectivity check and a held-out subset | §5 — a/b/c are scripted | +| Table complete | Render the skeleton; expect operands, not registers | `scripts/vmp_diff_harness.py emit-smali` | +| `resized` / `stripped` / stub verdict | Stop and say so | §7 | diff --git a/skills/apk-reverse/scripts/dex_dump_validate.py b/skills/apk-reverse/scripts/dex_dump_validate.py index ebcea68..08db07d 100644 --- a/skills/apk-reverse/scripts/dex_dump_validate.py +++ b/skills/apk-reverse/scripts/dex_dump_validate.py @@ -9,9 +9,18 @@ structure before trusting); this script is the tool that rule was missing. Per file it reports: size, sha256, dex version, checksum/signature verdicts (`dexutil.verify_dex_header`), class count, method-body census (no-code / real / -trivial-stub / empty), the trivial-body ratio that flags an extraction-shell -skeleton, and --find pattern hits across the string table. Files are grouped by -sha256 and the survivors are ranked: most likely original first. +trivial-stub / nop-erased / minimal-form / empty), the stub and emptied ratios that +flag an extraction-shell skeleton, and --find pattern hits across the string table. +Files are grouped by sha256 and the survivors are ranked: most likely original first. + +**Read `emptied%`, not `stub%`, for the skeleton call.** The two differ, and the +difference is load-bearing: `stub%` counts only bodies left as a lone `return*`, while +`emptied%` also counts bodies wiped to nothing but nops. A shell that clears the slot +instead of writing a return scores `stub% = 0.0` on a fully emptied image — measured, +and it used to be ranked the *most likely original* because of it. `stub%` is also +bimodal in practice: it lands at the app's own baseline or at ~100 %, with no band in +between, and partial extraction (75 % of bodies removed) moves it by 0.2 points. Full +matrix and commands: `docs/tool-verification/EXTENSION-extraction-shell-bench.md`. Exit code 0 unless nothing in the input parses as a dex at all (exit 1). @@ -36,12 +45,34 @@ RETURN_OPS = {0x0E, 0x0F, 0x10, 0x11} # return-void / return / return-wide / re def classify_body(data, code_off): - """Classify one code_item's instruction stream: 'empty' | 'stub' | 'real'. + """Classify one code_item's instruction stream. - 'stub' = skip plain nop units (0x0000) and exactly one return* remains. That is - what an extraction shell leaves behind when it defers decryption to first - invocation, and what a repair fixture writes when mimicking one. Anything with a - second meaningful unit is real code, however small. + Returns one of 'empty' | 'stub' | 'erased' | 'malformed' | 'real'. + + 'stub' -- skip plain nop units (0x0000) and exactly one return* remains. + That is what an extraction shell leaves behind when it defers + decryption to first invocation, and what a repair fixture writes + when mimicking one. + 'erased' -- insns_size > 0 but every unit is 0x0000. Measured on a fixture: + a body emptied by nop fill lands here, and reporting it as 'real' + (the pre-2026-09 behaviour) inverted the ranking, because a fully + nop-filled skeleton scored stub% = 0.0 and was named the most + likely original. A real compiler never emits a whole body of + nops, so this is skeleton evidence, not code. + 'minimal' -- a body truncated to `const/4 vR, #0; return vR` and nop-padded, + with a non-zero high byte on the return. This is a *legal* minimal + non-void body, not a malformed one -- and it is a skeleton shape + the stub detector is blind to by design. Counted separately so the + blind spot is visible in the report instead of hiding inside + 'real'. Measured: a fixture with all 5,061 bodies cut to this form + reports stub% = 1.9%, the same as its untouched control. + 'real' -- anything else, including a body of a single non-return unit. + + A two-unit `const/4 v0, #0; return v0` is deliberately **not** a stub: it is what + a *legal* minimal body looks like, and treating it as skeleton evidence would + make every tiny accessor a false positive. That asymmetry is the detector's + designed boundary, and it is measured in + `docs/tool-verification/EXTENSION-extraction-shell-bench.md`. """ insns_size = u32(data, code_off + 12) if insns_size == 0: @@ -55,14 +86,27 @@ def classify_body(data, code_off): continue meaningful += 1 last_unit = unit - if meaningful > 1: - return "real" # early exit: two units can never be a stub + if meaningful > 2: + return "real" # three units can never be a stub or a minimal + if meaningful == 0: + return "erased" # every unit was 0x0000 if meaningful == 1: op = last_unit & 0xFF # first byte holds the opcode for 10x/11x - if op == 0x0E and last_unit == 0x000E: # return-void, no register - return "stub" + if op == 0x0E: + # 10x return-void has no operand: `0xNN0E` with NN != 0 is `return vNN`, + # i.e. a non-void return wearing a return-void opcode -- minimal form. + return "stub" if last_unit == 0x000E else "minimal" if op in (0x0F, 0x10, 0x11): # 11x returns: high byte is the register return "stub" + return "real" + # meaningful == 2: legal minimal body only when it is `const/4 vR,#0; return* vR` + first_unit = 0 + for k in range(insns_size): + if u16(data, base + k * 2): + first_unit = u16(data, base + k * 2) + break + if (first_unit & 0xFF) == 0x12 and (last_unit & 0xFF) in (0x0E, 0x0F, 0x10, 0x11): + return "minimal" return "real" @@ -161,7 +205,7 @@ def profile_dex(path, patterns, trim=False): prof["signature_ok"] = sig_ok prof["classes"] = header["class_defs_size"] - no_code = with_code = stubs = empties = 0 + no_code = with_code = stubs = empties = erased = minimal = 0 try: for code_off in walk_methods(data, header): if code_off == 0: # abstract / native declaration @@ -175,14 +219,26 @@ def profile_dex(path, patterns, trim=False): stubs += 1 elif kind == "empty": empties += 1 + elif kind == "erased": + erased += 1 + elif kind == "minimal": + minimal += 1 except Exception as exc: # desynced class_data: keep counts prof["walk_error"] = str(exc) prof["methods_no_code"] = no_code prof["methods_with_code"] = with_code - prof["bodies_real"] = with_code - stubs - empties + prof["bodies_real"] = with_code - stubs - empties - erased - minimal prof["bodies_stub"] = stubs prof["bodies_empty"] = empties + prof["bodies_erased"] = erased + prof["bodies_minimal"] = minimal + # trivial_ratio keeps its original definition (return*-stub share of bodies) so + # existing records stay comparable. The skeleton signal is the union of every + # measured emptied shape; a body wiped to nothing at all must never score lower + # than a body wiped to `return-void`, or the ranking inverts (see + # docs/tool-verification/EXTENSION-extraction-shell-bench.md). prof["trivial_ratio"] = (stubs / with_code) if with_code else 0.0 + prof["emptied_ratio"] = ((stubs + erased) / with_code) if with_code else 0.0 if patterns: try: @@ -207,7 +263,38 @@ def collect_files(paths): def rank_key(prof): - return (prof["trivial_ratio"], + """Sort key: most-likely-original first. + + Both body signals are needed, in the right order, and each was measured to fail + alone: + + * the `return*`-stub share alone let a **nop-filled** skeleton score 0.0 and be + named the most likely original, because a wiped body counts as neither a stub + nor real code; + * `emptied_ratio` alone still failed, because it is **bimodal** -- it sits at the + host app's own baseline or at ~100 %, with nothing in between. A dex with 25 % + of its bodies emptied measures *below* its own untouched control, 1.7 % against + 1.9 %, so the sort pointed at a modified image and called it the original; + * `minimal + erased` alone failed too, and in the opposite direction: a skeleton + that writes a bare `return-void` into every body has minimal = erased = 0 and + would sort first, despite `emptied%` = 100. + + So the key combines both, and the combination has to let *either* signal disqualify + an image rather than letting them tie: + + * level 1 -- `emptied_ratio >= 0.5`. Nothing that is mostly emptied is a candidate, + whatever else it scores. This is the level that evicts the two skeletons the + plain counts cannot see: an image with a bare `return-void` in every body has + minimal = erased = 0, so a pure-count key ranked it first. + * level 2 -- `minimal + erased`, the skeleton-evidence count, low-first. This is + what orders the **bimodal band**, where `emptied_ratio` cannot: the untouched + control carries 82, and the count is monotone as bodies are removed (1299 / + 2524 / 3736 / 4946 at 25 / 50 / 75 / 100 %). + * level 3 -- `emptied_ratio` low-first, to separate what is left. + """ + return (1 if prof.get("emptied_ratio", prof["trivial_ratio"]) >= 0.5 else 0, + prof.get("bodies_minimal", 0) + prof.get("bodies_erased", 0), + prof.get("emptied_ratio", prof["trivial_ratio"]), 0 if prof["checksum_ok"] else 1, 0 if prof["signature_ok"] else 1, prof["name"]) @@ -248,9 +335,9 @@ def main(argv=None): # ---- human-readable report ------------------------------------------------- print("== dex dump validation: %d file(s), %d parse, %d rejected ==" % (len(profiles), len(valid), len(profiles) - len(valid))) - fmt = "%-28s %9s %-12s %3s %-7s %-7s %6s %5s %6s %7s" + fmt = "%-28s %9s %-12s %3s %-7s %-7s %6s %6s %6s %7s %7s %7s" print(fmt % ("name", "size", "sha256[:12]", "ver", "cksum", "sig", - "class", "noco", "code", "stub%")) + "class", "noco", "code", "stub%", "erased%", "emptied%")) for p in profiles: if "error" in p: print("%-28s %9s %-12s %3s REJECTED: %s" @@ -261,12 +348,19 @@ def main(argv=None): "ok" if p["checksum_ok"] else "BAD", "ok" if p["signature_ok"] else "BAD", p["classes"], p["methods_no_code"], p["methods_with_code"], - "%.1f%%" % (100.0 * p["trivial_ratio"]))) + "%.1f%%" % (100.0 * p["trivial_ratio"]), + "%.1f%%" % (100.0 * (p["bodies_erased"] / p["methods_with_code"] + if p["methods_with_code"] else 0.0)), + "%.1f%%" % (100.0 * p.get("emptied_ratio", 0.0)))) if p.get("trimmed_from"): print("%-28s trimmed %d -> %d B (page-aligned dump)" % ("", p["trimmed_from"], p["trimmed_to"])) if p.get("walk_error"): print("%-28s walk stopped early: %s" % ("", p["walk_error"])) + if p.get("bodies_minimal"): + print("%-28s minimal-form bodies: %d (const/4+return truncation -- a " + "skeleton shape the stub%% column cannot see)" + % ("", p["bodies_minimal"])) for pat, n in sorted(p.get("find_hits", {}).items()): print("%-28s find %-24s %d" % ("", pat, n)) @@ -283,18 +377,32 @@ def main(argv=None): print("\nranking (most likely original first):") for i, p in enumerate(ranking, 1): note = "" - if p["trivial_ratio"] >= 0.5: - note = " <-- SKELETON: stub%%=%.0f, extraction-shell shape" \ - % (100.0 * p["trivial_ratio"]) - print(" %d. %-28s stub%%=%.1f cksum=%s sig=%s classes=%d%s" + if p.get("emptied_ratio", 0.0) >= 0.5: + note = (" <-- SKELETON: emptied%%=%.0f, extraction-shell shape" + % (100.0 * p["emptied_ratio"])) + elif p.get("bodies_erased", 0): + note = (" <-- %d body(ies) wiped to nops: skeleton evidence" + % p["bodies_erased"]) + print(" %d. %-28s stub%%=%.1f erased%%=%.1f min/erased=%d cksum=%s " + "sig=%s classes=%d%s" % (i, p["name"], 100.0 * p["trivial_ratio"], + 100.0 * (p["bodies_erased"] / p["methods_with_code"] + if p["methods_with_code"] else 0.0), + p.get("bodies_minimal", 0) + p.get("bodies_erased", 0), "ok" if p["checksum_ok"] else "BAD", "ok" if p["signature_ok"] else "BAD", p["classes"], note)) top = ranking[0] copies = len(groups.get(top["sha256"], [])) - print("\nverdict: %s is the most likely original (%d copy(ies) in this set)" - % (top["name"], copies)) + print("\nverdict: %s is the best-supported candidate for the original " + "(%d copy(ies) in this set)" % (top["name"], copies)) + print("ranked by: mostly-emptied images evicted first (emptied%% >= 50), then " + "the minimal+nop-erased body count low-first, then emptied%%.\n" + " Limits, stated rather than hidden: this cannot see a `throw`-stub " + "skeleton (it scores at the control's own baseline), and a\n" + " partially extracted image still outranks a heavily stubbed one. " + "Confirm the winner before using it as a patch baseline --\n" + " docs/tool-verification/EXTENSION-extraction-shell-bench.md") else: print("\nverdict: no image parsed as a dex -- nothing to rank") return 0 if valid else 1 diff --git a/skills/apk-reverse/scripts/dex_mem_scan.py b/skills/apk-reverse/scripts/dex_mem_scan.py index 87b5c70..ba340f2 100644 --- a/skills/apk-reverse/scripts/dex_mem_scan.py +++ b/skills/apk-reverse/scripts/dex_mem_scan.py @@ -16,10 +16,24 @@ Feeding the extracted images to `dex_dump_validate.py` is the next step: this script only *finds* and *cuts*; that one judges whether what came out is a real body or an extraction-shell skeleton. +**Measured hit rate on the reference device: zero for anonymous regions.** The +root-dump pass recorded in `docs/tool-verification/EXTENSION-rootdump.md` scanned +142 anonymous mappings and returned **no hits**, while the *named* ART dex mappings +in the same process yielded 17 dex images without any search at all. A negative +result from this script is therefore weak evidence: it is compatible with "the +payload is not resident", "the payload is resident but not as a contiguous +dex-magic image", and "the payload was paged out between capture and scan". Treat a +zero-hit scan as a reason to check the named mappings first +(`[anon:dalvik-classes*.dex extracted in memory from ]`), not as a conclusion +about the packer. A positive hit inside a genuinely unnamed mapping has not been +observed here yet. + Scope note: the header's `file_size` field decides the cut. When the declared size does not fit inside the capture, the hit is reported and skipped rather than guessed at -- a dex whose tail is missing is not recoverable by truncating the -wrong end. +wrong end. Page-aligned captures from a VMA routinely overrun the image they hold, +so a run against raw VMA exports usually needs `--keep-partial` or a trim step +before `dex_dump_validate.py` will accept the output. """ import argparse import hashlib diff --git a/skills/apk-reverse/scripts/java2c_probe.py b/skills/apk-reverse/scripts/java2c_probe.py new file mode 100644 index 0000000..1f1a9c9 --- /dev/null +++ b/skills/apk-reverse/scripts/java2c_probe.py @@ -0,0 +1,960 @@ +#!/usr/bin/env python3 +"""java2c_probe.py - collect the evidence that decides HOW a target was hardened. + +The question this answers is narrow on purpose. Faced with "the Java methods have +no bodies", four different mechanisms produce that observation and they need four +different routes: + + Java2C Java bytecode translated to C, compiled into a .so. No dex + bytecode exists at runtime, ever. Route: read the .so. + Extraction shell Real dex bytecode, encrypted at rest, decrypted at runtime. + Route: dump memory, then validate the dump. + VMP Real dex bytecode replaced by private opcodes fed to an + interpreter. Route: measure, then usually stop. + JNI sinking A few hot methods deliberately moved to native by hand. + Route: reverse the .so; the dex is still readable. + +The expensive mistake is reading the first shape as the second: the analyst goes +looking for a decrypted DEX that is never produced, because nothing was ever +encrypted. This script prints the discriminating evidence and labels each item +strong / medium / weak so the reading is not mistaken for a verdict. + +Usage (paths are always explicit; nothing is discovered implicitly): + + python java2c_probe.py --apk sample.apk + python java2c_probe.py --apk sample.apk --json + python java2c_probe.py --dex classes.dex --so lib/arm64-v8a/libnc.so + python java2c_probe.py --dir extracted/ # every *.dex and *.so below + +Exit codes: 0 = evidence collected, 1 = no usable input, 2 = argument error. +""" + +import argparse +import json +import os +import re +import struct +import sys +import zipfile + +# --------------------------------------------------------------------------- +# dex parsing (self-contained; deliberately does not import the repo's dexutil) +# --------------------------------------------------------------------------- + +ACC_NATIVE = 0x0100 +ACC_ABSTRACT = 0x0400 + +# Instruction length in 16-bit code units, for the subset that a stub or a +# decompiled-from-C body produces. Unknown opcodes return 1 so a counting error +# can never inflate the trivial-body ratio; see the note on that metric below. +_INSN_UNITS = { + 0x00: 1, 0x01: 1, 0x02: 1, 0x03: 1, 0x04: 1, 0x05: 1, 0x06: 1, 0x07: 1, + 0x08: 1, 0x09: 1, 0x0a: 1, 0x0b: 1, 0x0c: 1, 0x0d: 1, 0x0e: 1, 0x0f: 1, + 0x10: 1, 0x11: 1, 0x12: 1, 0x13: 2, 0x14: 3, 0x15: 2, 0x16: 2, 0x17: 3, + 0x18: 5, 0x19: 2, 0x1a: 2, 0x1b: 3, 0x1c: 2, 0x1d: 1, 0x1e: 1, 0x1f: 1, + 0x20: 1, 0x21: 1, 0x22: 1, 0x23: 1, 0x24: 3, 0x25: 3, 0x26: 3, 0x27: 1, + 0x28: 1, 0x29: 1, 0x2a: 1, 0x2b: 1, 0x2c: 1, 0x2d: 1, 0x2e: 1, 0x2f: 1, + 0x30: 1, 0x31: 1, 0x32: 1, 0x33: 1, 0x34: 1, 0x35: 1, 0x36: 1, 0x37: 1, + 0x38: 1, 0x39: 1, 0x3a: 1, 0x3b: 1, 0x3c: 1, 0x3d: 1, 0x3e: 1, 0x3f: 1, + 0x40: 1, 0x41: 1, 0x42: 1, 0x43: 1, 0x44: 2, 0x45: 2, 0x46: 2, 0x47: 2, + 0x48: 2, 0x49: 2, 0x4a: 2, 0x4b: 2, 0x4c: 2, 0x4d: 2, 0x4e: 2, 0x4f: 2, + 0x50: 2, 0x51: 2, 0x52: 2, 0x53: 2, 0x54: 2, 0x55: 2, 0x56: 2, 0x57: 2, + 0x58: 2, 0x59: 2, 0x5a: 2, 0x5b: 2, 0x5c: 2, 0x5d: 2, 0x5e: 2, 0x5f: 2, + 0x60: 2, 0x61: 2, 0x62: 2, 0x63: 2, 0x64: 2, 0x65: 2, 0x66: 2, 0x67: 2, + 0x68: 2, 0x69: 2, 0x6a: 2, 0x6b: 2, 0x6c: 2, 0x6d: 2, 0x6e: 3, 0x6f: 3, + 0x70: 3, 0x71: 3, 0x72: 3, 0x73: 1, 0x74: 1, 0x75: 1, 0x76: 1, 0x77: 1, + 0x78: 2, 0x79: 2, 0x7a: 2, 0x7b: 2, 0x7c: 2, 0x7d: 2, 0x7e: 2, 0x7f: 2, + 0x80: 2, 0x81: 2, 0x82: 2, 0x83: 2, 0x84: 2, 0x85: 2, 0x86: 2, 0x87: 2, + 0x88: 2, 0x89: 2, 0x8a: 2, 0x8b: 2, 0x8c: 2, 0x8d: 2, 0x8e: 2, 0x8f: 2, + 0x90: 2, 0x91: 2, 0x92: 2, 0x93: 2, 0x94: 2, 0x95: 2, 0x96: 2, 0x97: 2, + 0x98: 2, 0x99: 2, 0x9a: 2, 0x9b: 2, 0x9c: 2, 0x9d: 2, 0x9e: 2, 0x9f: 2, + 0xa0: 2, 0xa1: 2, 0xa2: 2, 0xa3: 2, 0xa4: 2, 0xa5: 2, 0xa6: 2, 0xa7: 2, + 0xa8: 2, 0xa9: 2, 0xaa: 2, 0xab: 2, 0xac: 2, 0xad: 2, 0xae: 2, 0xaf: 2, + 0xb0: 1, 0xb1: 1, 0xb2: 2, 0xb3: 2, 0xb4: 2, 0xb5: 2, 0xb6: 2, 0xb7: 2, + 0xb8: 2, 0xb9: 2, 0xba: 2, 0xbb: 2, 0xbc: 2, 0xbd: 2, 0xbe: 2, 0xbf: 2, + 0xc0: 2, 0xc1: 2, 0xc2: 2, 0xc3: 2, 0xc4: 2, 0xc5: 2, 0xc6: 2, 0xc7: 2, + 0xc8: 2, 0xc9: 2, 0xca: 2, 0xcb: 2, 0xcc: 2, 0xcd: 2, 0xce: 2, 0xcf: 2, + 0xd0: 2, 0xd1: 2, 0xd2: 2, 0xd3: 2, 0xd4: 2, 0xd5: 2, 0xd6: 2, 0xd7: 2, + 0xd8: 2, 0xd9: 2, 0xda: 2, 0xdb: 2, 0xdc: 2, 0xdd: 2, 0xde: 2, 0xdf: 2, + 0xe0: 2, 0xe1: 2, 0xe2: 2, 0xe3: 0, 0xe4: 0, 0xe5: 0, 0xe6: 0, 0xe7: 0, + 0xe8: 0, 0xe9: 0, 0xea: 0, 0xeb: 0, 0xec: 0, 0xed: 0, 0xee: 0, 0xef: 0, + 0xf0: 0, 0xf1: 0, 0xf2: 0, 0xf3: 0, 0xf4: 0, 0xf5: 0, 0xf6: 0, 0xf7: 0, + 0xf8: 0, 0xf9: 0, 0xfa: 0, 0xfb: 0, 0xfc: 0, 0xfd: 0, 0xfe: 0, 0xff: 0, +} + +_RETURN_OPS = (0x0e, 0x0f, 0x10, 0x11) + + +def _uleb128(buf, pos): + result = 0 + shift = 0 + while True: + if pos >= len(buf): + raise ValueError("uleb128 truncated") + byte = buf[pos] + pos += 1 + result |= (byte & 0x7F) << shift + if not (byte & 0x80): + return result, pos + shift += 7 + if shift > 35: + raise ValueError("uleb128 too long") + + +class Dex(object): + """Minimal structural reader: header, class_defs, class_data, code_item.""" + + def __init__(self, data, name): + self.data = data + self.name = name + self.header_ok = False + self.header_note = "" + self.methods = [] # dicts: class, name, flags, code_off, units + self.class_names = [] + self.native_methods = 0 + self.abstract_methods = 0 + self.body_methods = 0 + self.no_code_methods = 0 # neither native nor abstract, yet code_off == 0 + self.trivial_bodies = 0 + self.native_with_code = 0 + self.fully_native_classes = 0 + self.native_dominated_classes = 0 + self.parse_error = None + self._parse() + + def _u16(self, off): + return struct.unpack_from("= len(string_offsets): + return "" % idx + off = string_offsets[idx] + _, pos = _uleb128(d, off) # utf16 length, ignored + end = d.find(b"\x00", pos) + if end < 0: + return "" + return d[pos:end].decode("utf-8", "replace") + + try: + type_desc = [string_at(self._u32(type_ids_off + 4 * i)) + for i in range(type_ids_size)] + except Exception as exc: + self.parse_error = "type_ids read failed: %s" % exc + return + + try: + if method_ids_off + method_ids_size * 8 > len(d): + raise ValueError("method_ids out of bounds") + method_info = [] + for i in range(method_ids_size): + base = method_ids_off + 8 * i + class_idx = self._u16(base) + name_idx = self._u32(base + 4) + desc = type_desc[class_idx] if class_idx < len(type_desc) else "?" + method_info.append((desc, string_at(name_idx))) + except Exception as exc: + self.parse_error = "method_ids read failed: %s" % exc + return + + if class_defs_off + class_defs_size * 32 > len(d): + self.parse_error = "class_defs table out of bounds" + return + + per_class_native = {} + per_class_total = {} + + for ci in range(class_defs_size): + base = class_defs_off + 32 * ci + class_idx = self._u32(base) + class_data_off = self._u32(base + 24) + cname = type_desc[class_idx] if class_idx < len(type_desc) else "?%d" % class_idx + self.class_names.append(cname) + + if class_data_off == 0: + continue + try: + pos = class_data_off + static_fields, pos = _uleb128(d, pos) + instance_fields, pos = _uleb128(d, pos) + direct_methods, pos = _uleb128(d, pos) + virtual_methods, pos = _uleb128(d, pos) + + for _ in range(static_fields + instance_fields): + _, pos = _uleb128(d, pos) # field_idx_diff + _, pos = _uleb128(d, pos) # access_flags + + # direct_methods and virtual_methods are two independent encoded + # lists: each one's first method_idx_diff is relative to 0. Walking + # them as a single run silently mis-attributes every virtual + # method, so they are decoded as two passes. + for list_size in (direct_methods, virtual_methods): + method_idx = 0 + for _ in range(list_size): + diff, pos = _uleb128(d, pos) + flags, pos = _uleb128(d, pos) + code_off, pos = _uleb128(d, pos) + method_idx += diff + + if method_idx < len(method_info): + mcls, mname = method_info[method_idx] + else: + mcls, mname = "?", "?%d" % method_idx + + units = None + if code_off: + units = self._u16(code_off + 12) # insns_size + + self.methods.append({ + "class": mcls, "name": mname, "flags": flags, + "code_off": code_off, "units": units, + }) + # Constructors are excluded from the per-class shape: a + # Java2C pass leaves / in Java in practice, + # and counting them would make "all methods native" an + # essentially unreachable test. + named = mname not in ("", "") + if named: + per_class_total[cname] = per_class_total.get(cname, 0) + 1 + + if flags & ACC_NATIVE: + self.native_methods += 1 + if named: + per_class_native[cname] = per_class_native.get(cname, 0) + 1 + if code_off: + self.native_with_code += 1 + elif flags & ACC_ABSTRACT: + self.abstract_methods += 1 + elif code_off == 0: + self.no_code_methods += 1 + else: + self.body_methods += 1 + if self._is_trivial(code_off, units): + self.trivial_bodies += 1 + except Exception: + # One unreadable class_data must not void the whole measurement. + continue + + for cname, total in per_class_total.items(): + if total < 3: + continue + native_here = per_class_native.get(cname, 0) + if native_here == total: + self.fully_native_classes += 1 + if native_here >= 0.7 * total: + self.native_dominated_classes += 1 + + def _is_trivial(self, code_off, units): + """A body that only returns a constant: the classic stub shape. + + Approximation, and labelled as such: it walks the instruction stream with + the length table above and accepts only nop/move/const/return. Anything + that touches a field, calls out, or branches is not a stub. + """ + if not units: + return False + insns_off = code_off + 16 + if insns_off + units * 2 > len(self.data): + return False + try: + pos = 0 + seen = 0 + while pos < units and seen < 8: + word = struct.unpack_from(" (count, strength, meaning) + self.java_symbols = [] + self.has_jni_onload = False + self.imports_register_natives = False + self.jni_call_refs = [] + self.elf_note = "" + self._parse() + + def _parse(self): + d = self.data + if len(d) < 64 or d[:4] != b"\x7fELF": + self.elf_note = "not an ELF image" + return + self.is_elf = True + ei_class = d[4] + ei_data = d[5] + if ei_class not in (1, 2): + self.elf_note = "bad EI_CLASS=%d" % ei_class + return + endian = "<" if ei_data == 1 else ">" + is64 = ei_class == 2 + try: + e_type = struct.unpack_from(endian + "H", d, 16)[0] + e_machine = struct.unpack_from(endian + "H", d, 18)[0] + if is64: + e_shoff = struct.unpack_from(endian + "Q", d, 0x28)[0] + e_shentsize = struct.unpack_from(endian + "H", d, 0x3A)[0] + e_shnum = struct.unpack_from(endian + "H", d, 0x3C)[0] + e_shstrndx = struct.unpack_from(endian + "H", d, 0x3E)[0] + else: + e_shoff = struct.unpack_from(endian + "I", d, 0x20)[0] + e_shentsize = struct.unpack_from(endian + "H", d, 0x2E)[0] + e_shnum = struct.unpack_from(endian + "H", d, 0x30)[0] + e_shstrndx = struct.unpack_from(endian + "H", d, 0x32)[0] + except struct.error as exc: + self.elf_note = "header read failed: %s" % exc + return + + self.arch = { + 3: "x86", 8: "mips", 20: "ppc", 40: "arm", 62: "x86_64", + 183: "aarch64", 243: "riscv", + }.get(e_machine, "machine-%d" % e_machine) + if e_type not in (2, 3): + self.elf_note = "e_type=%d (not EXEC/DYN)" % e_type + + # Strings first: it is the check that works even when the section table + # has been stripped, which is the common case for a hardened library. + self._scan_strings() + + if e_shoff and e_shnum and e_shoff + e_shnum * e_shentsize <= len(d): + self._read_sections(endian, is64, e_shoff, e_shentsize, e_shnum, e_shstrndx) + else: + self.elf_note = (self.elf_note + "; " if self.elf_note else "") + \ + "no usable section table (stripped?) - symbol checks skipped" + + def _scan_strings(self): + d = self.data + for pattern, label, strength, meaning in _STRINGSIGS: + count = len(re.findall(pattern, d)) + if count: + self.sig_hits[label] = (count, strength, meaning) + for call in _JNI_CALLS: + if re.search(re.escape(call).encode(), d): + self.jni_call_refs.append(call) + + def _read_sections(self, endian, is64, shoff, shentsize, shnum, shstrndx): + d = self.data + sections = [] + for i in range(shnum): + base = shoff + i * shentsize + try: + if is64: + name_off = struct.unpack_from(endian + "I", d, base)[0] + sh_type = struct.unpack_from(endian + "I", d, base + 4)[0] + sh_offset = struct.unpack_from(endian + "Q", d, base + 0x18)[0] + sh_size = struct.unpack_from(endian + "Q", d, base + 0x20)[0] + sh_link = struct.unpack_from(endian + "I", d, base + 0x28)[0] + sh_entsize = struct.unpack_from(endian + "Q", d, base + 0x38)[0] + else: + name_off = struct.unpack_from(endian + "I", d, base)[0] + sh_type = struct.unpack_from(endian + "I", d, base + 4)[0] + sh_offset = struct.unpack_from(endian + "I", d, base + 0x10)[0] + sh_size = struct.unpack_from(endian + "I", d, base + 0x14)[0] + sh_link = struct.unpack_from(endian + "I", d, base + 0x18)[0] + sh_entsize = struct.unpack_from(endian + "I", d, base + 0x24)[0] + except struct.error: + return + sections.append((name_off, sh_type, sh_offset, sh_size, sh_link, sh_entsize)) + + if shstrndx >= len(sections): + return + _, _, str_off, str_size, _, _ = sections[shstrndx] + if str_off + str_size > len(d): + return + shstr = d[str_off:str_off + str_size] + + def sec_name(off): + end = shstr.find(b"\x00", off) + return shstr[off:end if end >= 0 else len(shstr)].decode("utf-8", "replace") + + for name_off, sh_type, sh_offset, sh_size, sh_link, sh_entsize in sections: + # SHT_DYNSYM == 11, SHT_SYMTAB == 2 + if sh_type not in (2, 11): + continue + if sh_link >= len(sections): + continue + _, _, dstr_off, dstr_size, _, _ = sections[sh_link] + if dstr_off + dstr_size > len(d) or sh_offset + sh_size > len(d): + continue + dstr = d[dstr_off:dstr_off + dstr_size] + entsize = sh_entsize or (24 if is64 else 16) + if entsize == 0: + continue + for off in range(sh_offset, sh_offset + sh_size, entsize): + try: + if is64: + st_name = struct.unpack_from(endian + "I", d, off)[0] + st_shndx = struct.unpack_from(endian + "H", d, off + 6)[0] + else: + st_name = struct.unpack_from(endian + "I", d, off)[0] + st_shndx = struct.unpack_from(endian + "H", d, off + 14)[0] + except struct.error: + break + if st_name >= len(dstr): + continue + end = dstr.find(b"\x00", st_name) + sname = dstr[st_name:end if end >= 0 else len(dstr)].decode("utf-8", "replace") + if not sname: + continue + self.dynsym.append((sname, st_shndx)) + if sname.startswith("Java_"): + self.java_symbols.append(sname) + if sname == "JNI_OnLoad": + self.has_jni_onload = True + # Substring match, so a C implementation (bare name) and a C++ + # one (mangled form) are both caught. In practice NEITHER is + # present: see the note on dynamic-registration in classify(). + if "RegisterNatives" in sname and st_shndx == 0: + self.imports_register_natives = True + + def summary(self): + return { + "name": self.name, + "size": len(self.data), + "is_elf": self.is_elf, + "arch": self.arch, + "elf_note": self.elf_note, + "dynsym_entries": len(self.dynsym), + "java_symbols": len(self.java_symbols), + "java_symbol_sample": self.java_symbols[:5], + "exports_JNI_OnLoad": self.has_jni_onload, + "imports_RegisterNatives": self.imports_register_natives, + "jni_call_refs": sorted(self.jni_call_refs), + "signature_hits": {k: {"count": v[0], "strength": v[1], "meaning": v[2]} + for k, v in sorted(self.sig_hits.items())}, + } + + +# --------------------------------------------------------------------------- +# verdict +# --------------------------------------------------------------------------- + +def classify(dexes, elves, so_bytes_total): + """Turn measurements into a type guess plus the reasoning behind it. + + Every rule below is a shape test, not a proof, and the confidence is reported + so a low-confidence Java2C reading is not mistaken for a confirmed one. + """ + evidence = [] + verdict = "no-hardening-signal" + confidence = "low" + + dex_metrics = [d for d in dexes if not d.parse_error] + total_methods = sum(d.total_methods for d in dex_metrics) + total_native = sum(d.native_methods for d in dex_metrics) + native_ratio = (float(total_native) / total_methods) if total_methods else 0.0 + fully_native = sum(d.fully_native_classes for d in dex_metrics) + native_dominated = sum(d.native_dominated_classes for d in dex_metrics) + no_code = sum(d.no_code_methods for d in dex_metrics) + trivial_bodies = sum(d.trivial_bodies for d in dex_metrics) + body_methods = sum(d.body_methods for d in dex_metrics) + trivial_ratio = (float(trivial_bodies) / body_methods) if body_methods else 0.0 + + java_syms = sum(len(e.java_symbols) for e in elves) + register_natives_ref = any(e.imports_register_natives for e in elves) + has_jni_onload = any(e.has_jni_onload for e in elves) + # The reliable static signature of dynamic registration is the ABSENCE of + # Java_* symbols from a library that does export a JNI entry point. Looking + # for a literal RegisterNatives symbol does not work: NDK's C++ jni.h + # implements it as an inline member that calls through the function table, + # so no such symbol is emitted (measured: a real library exports JNI_OnLoad + # with zero Java_* and zero RegisterNatives symbols). + # Judged per library, not across the set: one library registering + # dynamically says nothing about another that exports static symbols. + dynamic_register_shape = any(e.has_jni_onload and not e.java_symbols for e in elves) + tool_markers = [] + for e in elves: + for label, (count, strength, meaning) in e.sig_hits.items(): + if strength == "strong": + tool_markers.append("%s(x%d) in %s" % (label, count, os.path.basename(e.name))) + medium_markers = [] + for e in elves: + for label, (count, strength, meaning) in e.sig_hits.items(): + if strength == "medium": + medium_markers.append("%s in %s" % (label, os.path.basename(e.name))) + + # --- dex-layer shape ------------------------------------------------- + java2c_dex_shape = (native_dominated >= 2) or (native_ratio >= 0.35 and total_native >= 10) + jni_sink_dex_shape = 0 < total_native <= 20 and native_ratio < 0.15 + stub_shape = no_code > 0 or (body_methods >= 8 and trivial_ratio >= 0.5) + + if dex_metrics: + evidence.append({ + "claim": "dex layer: native declaration density %.2f%% (%d/%d)" + % (native_ratio * 100, total_native, total_methods), + "strength": "medium" if java2c_dex_shape else "strong", + "supports": "java2c" if java2c_dex_shape else "baseline", + "note": "High density with whole classes turned native is the Java2C shape. " + "It is NOT sufficient on its own: a hand-written JNI class looks " + "identical in dex, so the .so layer decides.", + }) + evidence.append({ + "claim": "dex layer: %d classes are native-dominated (>=70%% native of >=3 " + "named methods), %d fully native" % (native_dominated, fully_native), + "strength": "medium", + "supports": "java2c" if native_dominated >= 2 else "baseline", + "note": "A hand-written JNI boundary class normally keeps its Java half, so " + "whole classes turned native are the Java2C shape. Constructors are " + "excluded: they stay in Java even after a translation pass.", + }) + if no_code: + evidence.append({ + "claim": "dex layer: %d non-native, non-abstract methods have no code_item" + % no_code, + "strength": "strong", + "supports": "extraction-shell/vmp", + "note": "Structurally invalid for plain Java; expect a loader that " + "fills bodies at runtime, or a private-opcode converter.", + }) + if trivial_ratio: + evidence.append({ + "claim": "dex layer: %.0f%% of method bodies are return-a-constant stubs " + "(%d/%d)" % (trivial_ratio * 100, trivial_bodies, body_methods), + "strength": "medium", + "supports": "extraction-shell/vmp" if stub_shape else "baseline", + "note": "Approximate metric (see --help). Empty bodies are the signature " + "of an extraction shell or a VMP, not of Java2C.", + }) + + # --- native-layer shape ---------------------------------------------- + if elves: + evidence.append({ + "claim": "native layer: %d exported Java_* symbols across %d library(ies)" + % (java_syms, len(elves)), + "strength": "strong" if java_syms else "medium", + "supports": "java2c" if java_syms else "none", + "note": "Static-linkage form. Compare the count against the dex native " + "method count: a rough 1:1 map is the Java2C signature.", + }) + if dynamic_register_shape: + dyn_libs = sorted(os.path.basename(e.name) + for e in elves if e.has_jni_onload and not e.java_symbols) + evidence.append({ + "claim": "native layer: dynamic registration in %s " + "(JNI_OnLoad exported, zero Java_* symbols)" + % ", ".join(dyn_libs), + "strength": "strong", + "supports": "dynamic-registration", + "note": "Binding happens at runtime, so a symbol search comes back empty " + "and fails SILENTLY. Do not expect a RegisterNatives symbol " + "either: NDK's C++ jni.h makes it an inline member that calls " + "through the function table, so none is emitted.", + }) + if register_natives_ref: + evidence.append({ + "claim": "native layer: a RegisterNatives symbol is imported", + "strength": "weak", + "supports": "dynamic-registration", + "note": "Only a C-side implementation leaves this name; its absence " + "proves nothing.", + }) + if has_jni_onload: + evidence.append({ + "claim": "native layer: JNI_OnLoad exported", + "strength": "weak", + "supports": "jni-boundary-exists", + "note": "Present in nearly every JNI library. Holds no information " + "about which hardening was applied.", + }) + if tool_markers: + evidence.append({ + "claim": "native layer: toolchain markers found: %s" % ", ".join(tool_markers), + "strength": "strong", + "supports": "java2c", + "note": "A Dex-to-C toolchain leaves its own runtime symbols behind.", + }) + if medium_markers: + evidence.append({ + "claim": "native layer: runtime markers found: %s" % ", ".join(medium_markers), + "strength": "medium", + "supports": "java2c", + "note": "dcc runtime header names; suggestive, and easy to rename.", + }) + + # --- verdict ---------------------------------------------------------- + if tool_markers and (java2c_dex_shape or java_syms): + verdict, confidence = "java2c", "high" + elif java2c_dex_shape and java_syms and total_native and \ + (float(java_syms) / total_native) >= 0.5: + verdict, confidence = "java2c", "high" + elif java2c_dex_shape and medium_markers: + verdict, confidence = "java2c", "medium" + elif java2c_dex_shape and dynamic_register_shape: + # A Dex-to-C toolchain that hides its symbols (dcc's Application.mk ships + # -fvisibility=hidden) leaves exactly this: a JNI entry point and a + # registration table, with no Java_* name to search for. + verdict, confidence = "java2c", "medium" + elif stub_shape: + verdict, confidence = "extraction-shell-or-vmp", "medium" + elif java2c_dex_shape: + verdict, confidence = "java2c-suspect-needs-so", "low" + elif jni_sink_dex_shape: + verdict, confidence = "jni-sinking", "medium" + elif dex_metrics or elves: + verdict, confidence = "no-hardening-signal", "medium" + + routes = { + "java2c": [ + "Do NOT look for a decrypted DEX: none exists, at any point, in memory.", + "Read the C in the .so: each translated method is one function with a " + "JNIEnv*/jobject prefix.", + "Rebuild the call graph through the JNI reverse calls " + "(FindClass/GetMethodID/CallXxxMethod).", + ], + "java2c-suspect-needs-so": [ + "Supply the matching .so (--so) before acting: the dex shape alone " + "cannot separate Java2C from a hand-written JNI class.", + "If no .so carries the logic, re-read the dex shape as ordinary Java.", + ], + "extraction-shell-or-vmp": [ + "Dump memory and measure the dump with scripts/dex_dump_validate.py.", + "If the dump is mostly stubs, the bodies are filled at invocation time: " + "references/advanced-unpacking.md.", + "If bodies decode as private opcodes, that is a VMP - recovery cost " + "usually exceeds the task value.", + ], + "jni-sinking": [ + "The dex is still readable: locate the Java call site, then reverse " + "the one or few native functions.", + "references/native-and-so.md, then the algorithm tooling.", + ], + "dynamic-registration": [ + "Hook or find RegisterNatives to read the binding table at runtime.", + ], + "no-hardening-signal": [ + "Proceed with ordinary dex-level analysis.", + ], + } + + return { + "verdict": verdict, + "confidence": confidence, + "metrics": { + "dex_total_methods": total_methods, + "dex_native_methods": total_native, + "dex_native_ratio": round(native_ratio, 4), + "dex_fully_native_classes": fully_native, + "dex_native_dominated_classes": native_dominated, + "dex_no_code_methods": no_code, + "dex_trivial_ratio": round(trivial_ratio, 4), + "so_java_symbols": java_syms, + "so_dynamic_registration": dynamic_register_shape, + "so_bytes_total": so_bytes_total, + }, + "evidence": evidence, + "route": routes.get(verdict, []), + "weak_criteria_warning": [ + "JNI_OnLoad / RegisterNatives / libc++_shared.so / FindClass appear in " + "ordinary JNI libraries: a hit is not a hardening verdict.", + "A high native ratio is also produced by hand-written JNI code and by R8 " + "stripping Java bodies; only the .so layer separates these.", + "The trivial-body ratio is approximate and flags extraction shells and " + "VMPs, never Java2C.", + ], + } + + +# --------------------------------------------------------------------------- +# input collection +# --------------------------------------------------------------------------- + +def _looks_like_dex(data): + return data[:4] == b"dex\n" + + +def _looks_like_elf(data): + return data[:4] == b"\x7fELF" + + +def collect(args): + dex_blobs = [] + elf_blobs = [] + sources = [] + + def add_file(path): + try: + with open(path, "rb") as fp: + data = fp.read() + except OSError as exc: + print("warning: cannot read %s (%s)" % (path, exc), file=sys.stderr) + return + if _looks_like_dex(data): + dex_blobs.append((os.path.basename(path), data)) + sources.append(path) + elif _looks_like_elf(data): + elf_blobs.append((os.path.basename(path), data)) + sources.append(path) + else: + print("warning: %s is neither a dex nor an ELF image, skipped" % path, + file=sys.stderr) + + for path in args.dex or []: + add_file(path) + for path in args.so or []: + add_file(path) + + apks = list(args.apk or []) + for directory in (args.dir or []): + for root, _dirs, files in os.walk(directory): + for fn in files: + full = os.path.join(root, fn) + low = fn.lower() + if low.endswith(".apk"): + apks.append(full) + elif low.endswith((".dex", ".so")): + add_file(full) + + for apk in apks: + if not os.path.exists(apk): + print("warning: apk not found: %s" % apk, file=sys.stderr) + continue + sources.append(apk) + try: + with zipfile.ZipFile(apk) as zf: + for info in zf.infolist(): + low = info.filename.lower() + if not (low.endswith(".dex") or low.endswith(".so")): + continue + if info.file_size > 256 * 1024 * 1024: + print("warning: skipping oversized entry %s (%d B)" + % (info.filename, info.file_size), file=sys.stderr) + continue + data = zf.read(info) + label = os.path.basename(info.filename) + if _looks_like_dex(data): + dex_blobs.append((label, data)) + elif _looks_like_elf(data): + elf_blobs.append((label, data)) + except zipfile.BadZipFile: + print("warning: %s is not a readable zip/apk" % apk, file=sys.stderr) + + return dex_blobs, elf_blobs, sources + + +def main(argv=None): + parser = argparse.ArgumentParser( + description="Collect the evidence that separates Java2C from an extraction " + "shell, a VMP and ordinary JNI sinking.", + epilog="Weak criteria (hits that do NOT establish a verdict on their own): " + "JNI_OnLoad, RegisterNatives, libc++_shared.so, FindClass/GetMethodID, " + "and a bare 'dcc' substring. The trivial-body ratio is advisory: it " + "flags extraction shells and VMPs, never Java2C.", + ) + parser.add_argument("--apk", action="append", help="APK to inspect (repeatable)") + parser.add_argument("--dex", action="append", help="dex image (repeatable)") + parser.add_argument("--so", action="append", help="ELF library (repeatable)") + parser.add_argument("--dir", action="append", + help="directory scanned recursively for *.apk/*.dex/*.so") + parser.add_argument("--json", action="store_true", help="emit machine-readable JSON") + args = parser.parse_args(argv) + + if not any([args.apk, args.dex, args.so, args.dir]): + parser.error("at least one of --apk/--dex/--so/--dir is required") + + dex_blobs, elf_blobs, sources = collect(args) + if not dex_blobs and not elf_blobs: + print("error: no usable dex or ELF input found", file=sys.stderr) + return 1 + + dexes = [Dex(data, name) for name, data in dex_blobs] + elves = [Elf(data, name) for name, data in elf_blobs] + result = classify(dexes, elves, sum(len(d) for _n, d in elf_blobs)) + + report = { + "inputs": sources, + "dex": [d.summary() for d in dexes], + "native": [e.summary() for e in elves], + "classification": result, + } + + if args.json: + print(json.dumps(report, indent=2, sort_keys=False)) + return 0 + + print("=" * 72) + print("java2c_probe: hardening classification evidence") + print("=" * 72) + print("inputs: %d file(s)" % len(sources)) + for src in sources: + print(" - %s" % src) + + print("\n[dex layer]") + if not dexes: + print(" (no dex supplied: the dex-layer metrics are unmeasured, so a") + print(" Java2C reading cannot be supported or excluded from this input)") + for d in dexes: + s = d.summary() + if s["parse_error"]: + print(" %s: PARSE ERROR: %s" % (s["name"], s["parse_error"])) + continue + print(" %s: %d B, %d classes, %d methods, native %d (%.2f%%), " + "native-dominated classes %d, fully native %d" + % (s["name"], s["size"], s["classes"], s["total_methods"], + s["native_methods"], s["native_ratio"] * 100, + s["native_dominated_classes"], s["fully_native_classes"])) + print(" no-code methods %d, trivial bodies %d/%d (%.0f%%), header %s" + % (s["no_code_methods"], s["trivial_bodies"], s["body_methods"], + s["trivial_ratio"] * 100, + "OK" if s["header_size_matches"] else "MISMATCH: " + s["header_note"])) + + print("\n[native layer]") + if not elves: + print(" (no .so supplied)") + for e in elves: + s = e.summary() + if not s["is_elf"]: + print(" %s: not ELF (%s)" % (s["name"], s["elf_note"])) + continue + print(" %s: %d B, %s, dynsym %d entries, Java_* %d, JNI_OnLoad %s, " + "RegisterNatives-import %s" + % (s["name"], s["size"], s["arch"], s["dynsym_entries"], + s["java_symbols"], s["exports_JNI_OnLoad"], + s["imports_RegisterNatives"])) + if s["elf_note"]: + print(" note: %s" % s["elf_note"]) + if s["signature_hits"]: + for label, info in s["signature_hits"].items(): + print(" [%s] %s x%d - %s" + % (info["strength"], label, info["count"], info["meaning"])) + + print("\n[classification]") + print(" verdict : %s" % result["verdict"]) + print(" confidence : %s" % result["confidence"]) + print("\n evidence:") + for item in result["evidence"]: + print(" (%s) %s" % (item["strength"], item["claim"])) + print(" note: %s" % item["note"]) + print("\n route:") + for line in result["route"]: + print(" - %s" % line) + print("\n weak criteria - a hit here is NOT a verdict:") + for line in result["weak_criteria_warning"]: + print(" ! %s" % line) + print("=" * 72) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/skills/apk-reverse/scripts/kernelsu_syscall_mask.py b/skills/apk-reverse/scripts/kernelsu_syscall_mask.py new file mode 100644 index 0000000..a8514bc --- /dev/null +++ b/skills/apk-reverse/scripts/kernelsu_syscall_mask.py @@ -0,0 +1,937 @@ +#!/usr/bin/env python3 +"""kernelsu_syscall_mask.py -- generate a configurable kernel-side syscall-masking scaffold. + +What this generates, and the correction it exists to make +--------------------------------------------------------- +There is a widespread misreading worth killing early: **a KernelSU module is a +Magisk-compatible userspace module and cannot change what a syscall returns.** +Its `post-fs-data.sh` / `service.sh` run as root in the normal world; nothing in +that format reaches the kernel's return path. Everything a module of that shape +can do is userspace work -- bind mounts, setting props, starting a daemon, +writing config. + +Actually rewriting what `openat`/`read`/`stat` return for `/proc/self/maps` or +`/proc/self/status` needs one of exactly three things: + + kpm KernelPatch / APatch Kernel Patch Module -- pointer replacement in the + syscall table (`fp_hook_syscalln`) or inline hooks (`hook_wrapN`). + Built with a bare-metal ARM64 toolchain (aarch64-none-elf-gcc) into a + relocatable .kpm ELF. Needs a KernelPatch-patched boot image. + lkm An out-of-tree kernel module. Needs kernel source matching the device's + exact version and vermagic, and a loader path. + ebpf A BPF program attached to a syscall tracepoint or kprobe. Needs a GKI + kernel with the tracing/BTF machinery -- 5.10+ in Android practice. + +So this script emits a **module directory that is honest about which half is +loadable**: the userspace skeleton (module.prop + scripts + config) is real and +installable, and each kernel-side target is a template with its toolchain gate +stated in its own header. None of the kernel-side code is compiled or loaded +anywhere by this script. + +What is measured vs unverified +------------------------------ +`generate`, `gates` and `verify` are exercised on the host (see +docs/tool-verification/EXTENSION-kernel-weapons.md). **Whether the kernel-side +code compiles or loads is unverified** -- that needs a device with the matching +kernel and the matching toolchain, and the reference device is a 4.14.186 Magisk +build where all three routes are closed. + +Examples + python kernelsu_syscall_mask.py gates + python kernelsu_syscall_mask.py generate --out work/sysmask --package + python kernelsu_syscall_mask.py verify --dir work/sysmask +""" +import argparse +import json +import os +import re +import subprocess +import sys + +# --------------------------------------------------------------------------- +# Kernel gate table. Every number here is either a toolchain fact or a +# documented platform threshold; the "how to check" column is what turns it +# from trivia into a test you run on your own device. +# --------------------------------------------------------------------------- +GATES = [ + { + "route": "eBPF tracepoint/kprobe", + "gate": "GKI kernel 5.10+ (Android 12+)", + "why": "Android's BPF/tracing support as bpftrace-class tooling and " + "stackplz expect it is a GKI-era feature; BTF is required for " + "CO-RE style probes.", + "check": "uname -r -> expect 5.10.x or newer; ls /sys/kernel/btf/vmlinux", + "ref": "https://source.android.com/docs/core/architecture/kernel/bpf", + }, + { + "route": "KernelPatch / APatch KPM", + "gate": "KernelPatch-patched boot image + kpm toolchain", + "why": "KPMs load through kpimg injected into the kernel image's payload " + "segment; building one needs a bare-metal ARM64 compiler " + "(aarch64-none-elf-gcc), not the NDK.", + "check": "ls /data/adb/ | grep -i apatch ; which aarch64-none-elf-gcc", + "ref": "https://github.com/bmax121/KernelPatch", + }, + { + "route": "Out-of-tree LKM", + "gate": "Kernel source matching the device's exact version + vermagic", + "why": "A module built against a different kernel revision is refused at " + "load time on a mismatched vermagic, and the device's kernel " + "source is frequently not published at all.", + "check": "uname -r ; look for a matching kernel source tree for the device", + "ref": "https://source.android.com/docs/core/architecture/kernel/android-common", + }, + { + "route": "seccomp-BPF (for contrast)", + "gate": "any modern kernel, but it is not this route", + "why": "seccomp can make a syscall FAIL (SECCOMP_RET_ERRNO/TRAP); it " + "cannot rewrite the CONTENT a successful read returns. It closes " + "doors, it does not paint them -- which is the whole requirement " + "for a spoofed /proc read.", + "check": "n/a -- this is a capability ceiling, not a version check", + "ref": "references/kernel-and-environment-hardening.md", + }, +] + +DEFAULT_RULES = [ + { + "syscall": "openat", + "path": "/proc/self/maps", + "action": "deny", + "errno": "ENOENT", + "note": "hides the mapping list a userspace instrumentation agent " + "advertises itself in", + }, + { + "syscall": "newfstatat", + "path": "/proc/self/maps", + "action": "fake_size", + "size": 0, + "note": "a stat that reports 0 bytes is as good as absent for a reader " + "that sizes its buffer first", + }, + { + "syscall": "openat", + "path": "/proc/self/status", + "action": "filter_line", + "prefix": "TracerPid", + "note": "the write side owns this field; see " + "references/kernel-and-environment-hardening.md section 5", + }, + { + "syscall": "read", + "path": "/proc/self/status", + "action": "filter_line", + "prefix": "TracerPid", + "note": "line filter applied to the read once the fd is open", + }, +] + +MODULE_PROP = """id={id} +name={name} +version={version} +versionCode=1 +author=apk-reverse +description={description} +""" + +POST_FS_DATA = """#!/system/bin/sh +# Runs at post-fs-data, as root, in the normal world. +# The kernel side (if you built and loaded one) owns the syscall table; this +# script's whole job is to make the configuration visible to it and to leave a +# record that the module actually ran. Do not put a syscall hook here: this is +# userspace, and nothing here can change what openat/read/stat return. +MODDIR=${{0%/*}} +CONF=$MODDIR/config/syscall_mask.json +LOG=/data/adb/{id}.log + +if [ ! -f "$CONF" ]; then + echo "$(date) missing $CONF" >> "$LOG" + exit 0 +fi +echo "$(date) post-fs-data: config present, $(wc -c < "$CONF") bytes" >> "$LOG" +# The KPM/LKM loader path is intentionally NOT invoked from here. Loading a +# kernel module is a boot-image-level decision with a bricking risk; do it once, +# by hand, and read the evidence file first. +exit 0 +""" + +SERVICE = """#!/system/bin/sh +# Runs in late_start service mode, as root. Kept minimal and non-blocking on +# purpose: a module that spins here delays boot on every start, and this one +# only reports state. +MODDIR=${{0%/*}} +CONF=$MODDIR/config/syscall_mask.json +LOG=/data/adb/{id}.log + +if [ -r /proc/sys/kernel/version ]; then + echo "$(date) service: kernel $(uname -r)" >> "$LOG" +fi +echo "$(date) service: rules=$(grep -c '"syscall"' "$CONF" 2>/dev/null)" >> "$LOG" +exit 0 +""" + +UNINSTALL = """#!/system/bin/sh +# Uninstall hook. Deliberately does not attempt to unload a kernel module: if one +# is loaded, unloading it is a separate deliberate act with its own risk, and a +# silent unload during package removal is how a device ends up in a boot loop. +rm -f /data/adb/{id}.log +exit 0 +""" + +CUSTOMIZE = """#!/system/bin/sh +# KernelSU/Magisk install-time script. Kept as a no-op so the module always +# installs; see the module README for why nothing is set up here. +exit 0 +""" + +KPM_C = r'''// SPDX-License-Identifier: GPL-2.0 +/* + * syscall_mask.c -- config-driven syscall return masking, KernelPatch/APatch KPM. + * + * Generated by skills/apk-reverse/scripts/kernelsu_syscall_mask.py + * + * STATUS: TEMPLATE, NOT COMPILED AND NOT LOADED. Every hook below follows the + * KernelPatch KPM interface as documented in public KPM development write-ups + * (KPM_NAME/KPM_INIT/KPM_EXIT sections, kfunc_def symbol resolution without + * `extern`, fp_hook_syscalln pointer replacement, hook_fargs4_t callbacks with + * args->skip_origin / args->ret). VERIFY EVERY NAME AND SIGNATURE against the + * kpmodule.h of the KernelPatch revision you build against -- this file was not + * compiled anywhere, and an interface that has drifted produces a module that + * fails to load, or worse, loads and faults. + * + * Build (bare-metal ARM64, NOT the NDK): + * aarch64-none-elf-gcc -O2 -fno-stack-protector -c syscall_mask.c + * ... link into a relocatable ELF per the KernelPatch module docs + * Load: push the .kpm and add it through the APatch app's KPM manager. + * + * Design notes that are load-bearing rather than stylistic: + * - Callbacks run on EVERY syscall of their number, system-wide. The first + * thing each one does is reject the overwhelmingly common case, because a + * hook that does real work on every write is how a device locks up. + * - Inline hooks (hook_wrapN) rewrite the target's prologue. On an LTO kernel + * the exported symbol is frequently NOT the call site (the call got inlined), + * so an inline hook installs cleanly and never fires. Pointer replacement in + * the syscall table (fp_hook_syscalln) does not have that failure mode, + * because the syscall entry path must go through the table. + * - Never sleep in these callbacks except where the syscall itself runs in + * process context and the copy helper is the sleeping variant. + */ +#include +#include +#include +#include + +KPM_NAME("{id}"); +KPM_VERSION("{version}"); +KPM_LICENSE("GPL"); +KPM_AUTHOR("apk-reverse"); +KPM_DESCRIPTION("{description}"); + +/* kfunc declarations must NOT carry `extern`: an extern declaration compiles to + * an undefined reference (*UND*) and the loader rejects the module with + * "unknown symbol". Letting it be a tentative definition allocates the slot in + * .bss and the loader fills it in. Measured and documented in public KPM work. */ +int kfunc_def(strncpy_from_user)(char *dst, const char __user *src, long count); + +/* ------------------------------------------------------------------ rules -- + * Generated from config/syscall_mask.json. Keep this table in sync by + * re-running the generator rather than by hand-editing it. + */ +#define MASK_DENY 1 /* return -err before the real syscall runs */ +#define MASK_FAKE_SIZE 2 /* openat succeeds, stat reports a bogus size */ +#define MASK_FILTER_LINE 3 /* strip lines with this prefix from read() */ + +struct mask_rule { + const char *path; /* userspace path to match, NULL = any */ + const char *prefix; /* line prefix for MASK_FILTER_LINE, else NULL */ + int action; + int err; /* -errno for MASK_DENY */ + long value; /* size for MASK_FAKE_SIZE */ +}; + +static const struct mask_rule rules[] = { +{RULE_TABLE} +}; + +#define NRULES (sizeof(rules) / sizeof(rules[0])) + +/* A bounded, allocation-free matcher. The path is copied in first, so the + * comparison never touches userspace after the initial copy -- touching + * userspace memory twice is how a hook races with the caller's own unmapping. */ +static int copy_and_match(const char __user *upath, char *kbuf, unsigned long n, + const char **matched) +{{ + long copied; + unsigned long i; + + if (!upath || n == 0) + return 0; + if (!kfunc(strncpy_from_user)) + return 0; /* symbol unresolved: fail open */ + copied = kfunc(strncpy_from_user)(kbuf, upath, (long)(n - 1)); + if (copied <= 0) + return 0; + kbuf[copied] = '\0'; + + for (i = 0; i < NRULES; i++) {{ + if (!rules[i].path) + continue; + if (strstr(kbuf, rules[i].path)) {{ + *matched = rules[i].path; + return rules[i].action; + }} + }} + return 0; +}} + +/* ------------------------------------------------------------------ openat -- + * The path argument is arg2 on arm64 (dfd, path, flags, mode). If the path is + * unreadable, the rule is skipped rather than guessed at. + */ +static void before_openat(hook_fargs4_t *args, void *udata) +{{ + const char __user *upath = (const char __user *)syscall_argn(args, 1); + char kbuf[256]; + const char *matched = 0; + int action; + + (void)udata; + action = copy_and_match(upath, kbuf, sizeof(kbuf), &matched); + if (action == MASK_DENY) + args->ret = MASK_DENY_ERRNO; + if (action == MASK_FAKE_SIZE) + args->skip_origin = 0; /* let it open; the stat is spoofed */ +}} + +/* --------------------------------------------------------------- newfstatat -- + * arm64 has no plain `stat` syscall; readers go through newfstatat (or fstatat64 + * on 32-bit). A zero size is what makes a reader that sizes its buffer first + * decide there is nothing to read. + */ +static void after_newfstatat(hook_fargs4_t *args, void *udata) +{{ + const char __user *upath = (const char __user *)syscall_argn(args, 1); + char kbuf[256]; + const char *matched = 0; + + (void)udata; + if (copy_and_match(upath, kbuf, sizeof(kbuf), &matched) != MASK_FAKE_SIZE) + return; + /* The struct stat* is arg2; zeroing st_size is the whole point. Field + * offsets are ABI-specific -- resolve them against the target's headers + * rather than trusting an offset copied from a different architecture. */ + /* TODO(verify-on-device): zero the size field of the struct at arg2. */ +}} + +/* -------------------------------------------------------------------- read -- + * Line filtering happens on the read side, once the fd is already open. Doing + * it here rather than at openat is what lets a reader that opens the file + * before the filter is installed still be covered. + */ +static void before_read(hook_fargs4_t *args, void *udata) +{{ + long count = (long)syscall_argn(args, 2); + (void)udata; + if (count <= 0 || count > 8192) + return; /* first filter: the common case */ + /* TODO(verify-on-device): identify the fd's path, copy the buffer, rewrite + * it in place minus the filtered lines, and adjust the returned length. */ +}} + +/* ------------------------------------------------------------------ init ----- + * Registration order matters: install the syscall hooks first, then enable the + * rules. If the module is unloaded between the two, the hooks must already be + * gone. + */ +static long mask_init(const char *args, const struct kernel_patch_funcs *kp) +{{ + (void)args; + (void)kp; + + if (!kfunc(strncpy_from_user)) {{ + /* Failing loudly here is deliberate: a half-armed hook set that cannot + * read a path would pass every check through, which looks like success + * and is not. */ + return -EINVAL; + }} + + /* fp_hook_syscalln(__NR_openat, before_openat, 0, 0); */ + /* fp_hook_syscalln(__NR_newfstatat, before_newfstatat, 0, 0); */ + /* fp_hook_syscalln(__NR_read, before_read, 0, 0); */ + /* TODO(verify-on-device): the argument lists above follow the public + * examples; confirm arity against your kpmodule.h before enabling them. */ + return 0; +}} + +static long mask_exit(void *udata) +{{ + (void)udata; + /* Unhook in the exact reverse order of registration. */ + return 0; +}} + +KPM_INIT(mask_init); +KPM_EXIT(mask_exit); +''' + +LKM_C = r'''// SPDX-License-Identifier: GPL-2.0 +/* + * syscall_mask.c -- out-of-tree LKM variant of the same idea. + * + * STATUS: TEMPLATE, NOT COMPILED. Worse than the KPM variant in one specific + * way worth understanding before choosing it: to replace a syscall you must + * reach the syscall table, and `kallsyms_lookup_name` stopped being exported in + * Linux 5.7. On pre-5.7 kernels a direct call works; after that you need a + * kprobe to recover the address, or an explicit kernel export. The device's own + * kernel source must also match its exact vermagic, and an out-of-tree module + * built against a mismatched tree is refused at insmod. + * + * This is genuine kernel development with a bricking risk. It is the most + * expensive of the three routes and the one least likely to be worth it. + */ +#include +#include +#include +#include +#include + +MODULE_LICENSE("GPL"); +MODULE_DESCRIPTION("{description}"); +MODULE_AUTHOR("apk-reverse"); + +/* + * Pre-5.7: kallsyms_lookup_name() is exported, so the table pointer is + * recoverable directly. 5.7+: NOT exported -- you must either carry a kprobe to + * find the symbol or patch the kernel's export list. There is no portable + * version of this, which is the honest reason this route is rated last. + */ +#if LINUX_VERSION_CODE < KERNEL_VERSION(5, 7, 0) +static unsigned long *sys_call_table_addr(void) +{ + return (unsigned long *)kallsyms_lookup_name("sys_call_table"); +} +#else +static unsigned long *sys_call_table_addr(void) +{ + /* TODO(verify-on-device): recover via kprobe on kallsyms_lookup_name, or + * enable an explicit export in your own kernel build. */ + return NULL; +} +#endif + +static int __init mask_init(void) +{ + unsigned long *table = sys_call_table_addr(); + if (!table) { + pr_err("syscall_mask: cannot locate sys_call_table on this kernel\n"); + return -EINVAL; + } + pr_info("syscall_mask: sys_call_table at %px\n", table); + /* TODO(verify-on-device): save the original pointer, write CR0 (or use + * set_memory_rw) to make the table writable, swap in your handler. Getting + * the write-protect dance wrong takes the device down. */ + return 0; +} + +static void __exit mask_exit(void) +{ + /* TODO(verify-on-device): restore the saved pointers. */ +} + +module_init(mask_init); +module_exit(mask_exit); +''' + +EBPF_C = r'''// SPDX-License-Identifier: GPL-2.0 OR BSD-2-Clause +/* + * syscall_mask.bpf.c -- eBPF probe template: filter sys_enter_openat, then + * rewrite the result. + * + * STATUS: TEMPLATE, NOT COMPILED AND NOT ATTACHED. Not runnable on a 4.14 + * kernel: see the gate table in the module README (GKI 5.10+ in Android + * practice, plus BTF for CO-RE style probes). + * + * READ THIS BEFORE COUNTING ON THE RESULT REWRITE. There are two different + * attachment points and they are not interchangeable: + * + * - A *tracepoint* (syscalls/sys_enter_openat) is an observation point. It can + * read the arguments and it can emit events, and it CANNOT change the + * syscall's return value. A tracepoint-only program gives you visibility, + * not spoofing. + * - A *kprobe* on the syscall's entry (or a kretprobe on its exit) can modify + * a register with bpf_override_return() -- but only for functions flagged + * ALLOW_ERROR_INJECTION, which kernel syscall entry points are not, in + * general. On many kernels the helper is refused for exactly this target. + * + * So the honest eBPF shape for this job is: observe with a tracepoint, and do + * the rewrite in a place that actually owns the return value. What follows is + * the observation half done properly, plus the rewrite half marked for the + * device-side verification it needs. + */ +#include +#include +#include + +char LICENSE[] SEC("license") = "GPL"; + +#define MAX_PATH 256 +#define TARGET_LEN 15 /* strlen("/proc/self/maps") */ + +struct event { + __u32 pid; + __u32 uid; + __s32 dfd; + __s64 ret; + char path[MAX_PATH]; + char comm[16]; +}; +/* /proc/self/status is 17; kept as a separate constant so the two rules can be + * toggled independently. */ +#define TARGET_STATUS_LEN 17 + +struct { + __uint(type, BPF_MAP_TYPE_RINGBUF); + __uint(max_entries, 1 << 24); +} events SEC(".maps"); + +/* Filter the target package's processes. Populated from userspace with the + * uids the app can run as; an empty map means "all processes", which is a + * deliberate default because a filter that silently matches nothing looks + * exactly like a probe that is not firing. */ +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __uint(max_entries, 64); + __type(key, __u32); /* uid */ + __type(value, __u8); +} target_uids SEC(".maps"); + +static __always_inline int want(void) +{ + __u32 uid = bpf_get_current_uid_gid() & 0xffffffff; + __u8 *hit = bpf_map_lookup_elem(&target_uids, &uid); + /* Empty map -> observe everything. Populated map -> only listed uids. */ + return hit || (bpf_map_lookup_elem(&target_uids, &uid) == 0); +} + +SEC("tracepoint/syscalls/sys_enter_openat") +int tp_sys_enter_openat(struct trace_event_raw_sys_enter *ctx) +{ + struct event *e; + const char *upath = (const char *)ctx->args[1]; + + if (!want()) + return 0; + + e = bpf_ringbuf_reserve(&events, sizeof(*e), 0); + if (!e) + return 0; + e->pid = bpf_get_current_pid_tgid() >> 32; + e->uid = bpf_get_current_uid_gid() & 0xffffffff; + e->dfd = (__s32)ctx->args[0]; + e->ret = 0; + bpf_get_current_comm(&e->comm, sizeof(e->comm)); + bpf_probe_read_user_str(&e->path, sizeof(e->path), upath); + bpf_ringbuf_submit(e, 0); + return 0; +} + +/* + * The result-rewrite half. Two dead ends are recorded here on purpose so they + * are not re-derived: + * + * 1. bpf_override_return() only works on functions marked + * ALLOW_ERROR_INJECTION. A raw syscall entry is not one of them, so the + * helper is typically refused for this target -- check the return value + * and the verifier message rather than assuming the rewrite happened. + * 2. A tracepoint cannot rewrite at all. If the rewrite is essential, the + * route is a kprobe plus a kernel that permits the override, or one of the + * KPM/LKM routes; eBPF's strength here is observation. + */ +SEC("kprobe/do_sys_openat2") +int BPF_KPROBE(kp_do_sys_openat2, int dfd, const char *filename) +{ + char buf[64]; + if (bpf_probe_read_user_str(&buf, sizeof(buf), filename) <= 0) + return 0; + if (buf[0] == '/' && buf[1] == 'p' && buf[2] == 'r') { + /* TODO(verify-on-device): bpf_override_return(ctx, -ENOENT) here only + * after confirming ALLOW_ERROR_INJECTION on this target kernel. */ + } + return 0; +} +''' + +KPM_MAKEFILE = """# aarch64-none-elf-gcc is a BARE-METAL ARM64 compiler, not the Android NDK +# toolchain. The NDK targets Android userspace and will not produce a loadable +# KPM. STATUS: untested -- this toolchain is not present on a Magisk-only device. +CROSS ?= aarch64-none-elf- +CC := $(CROSS)gcc +OBJCOPY := $(CROSS)objcopy + +TARGET := {id} +KPM_DIR ?= ../KernelPatch + +CFLAGS := -O2 -fno-stack-protector -fno-pic -fno-builtin \\ + -I$(KPM_DIR)/kernel/include -I$(KPM_DIR)/include \\ + -Wall -Wno-unused-variable + +all: $(TARGET).kpm + +$(TARGET).kpm: src/{id}.c +\t$(CC) $(CFLAGS) -c $< -o $@.o +\t$(OBJCOPY) -O binary --only-section=.text $@.o /dev/null 2>/dev/null || true +\t@echo "NOTE: linking follows the KernelPatch module format; see the docs" +\t@echo " for the current link step before trusting this target." + +clean: +\trm -f *.o *.kpm + +.PHONY: all clean +""" + +MODULE_README = """# {name} -- syscall-return masking scaffold + +Generated by `skills/apk-reverse/scripts/kernelsu_syscall_mask.py`. + +**Read this first: the userspace module in this directory cannot change what a +syscall returns.** It is a KernelSU/Magisk-compatible module: its scripts run as +root in the normal world. Its only jobs are to carry the configuration, to +record that it ran, and to be a place to put the module metadata. The kernel +side is a separate artifact with a separate toolchain and a separate risk. + +## What is here + +| Path | Status | +|---|---| +| `module.prop`, `customize.sh`, `post-fs-data.sh`, `service.sh` | real, installable module skeleton | +| `config/syscall_mask.json` | the rule set, read by the kernel side | +| `kpm/{id}.c`, `kpm/Makefile` | APatch/KernelPatch template -- **not compiled, not loaded** | +| `lkm/{id}.c` | out-of-tree module template -- **not compiled, not loaded** | +| `ebpf/syscall_mask.bpf.c` | eBPF probe template -- **not compiled, not attached** | + +## Gate table -- check your device before choosing a route + +| Route | Gate | How to check | Why | +|---|---|---|---| +{gates} + +## Reference device, measured + +``` +$ adb shell 'uname -r; cat /proc/version' +4.14.186+ +Linux version 4.14.186+ (nobody@android-build) (Android (6443078 based on r383902) +clang version 11.0.1 ...) #1 SMP PREEMPT Wed Mar 30 23:32:42 CST 2022 +``` + +On that device **every kernel-side route above is closed**: + +- eBPF: kernel 4.14 is well below the 5.10 gate. +- KPM: no KernelPatch-patched image, and no `aarch64-none-elf-gcc` on the host; + installing APatch means patching the boot image, which is a bricking risk. +- LKM: no kernel source for the device, so no matching vermagic is possible. + +That is a **route decision, not a failure** -- and it is why every kernel-side +file here is labelled a template. Nothing in this directory was loaded on +anything. + +## If you do take a kernel route + +Do it once, deliberately, and read the evidence first: + +1. Confirm the gate in the table above on the actual device. +2. Build the kernel side on a machine that has the right toolchain. +3. **Back up the boot image and know the recovery path before loading anything.** + A kernel module that faults takes the device down before you can read why. +4. Load it by hand, watch `dmesg`, and keep the previous boot image in hand. +5. Only then wire it into boot, and only if the load is repeatable. + +`post-fs-data.sh` in this scaffold deliberately does **not** load anything. That +is not an omission. +""" + + +def _render(template, **kw): + """Fill a template's {placeholders} without str.format(). + + Braces are the common case in C, so using format() here would mean doubling + every one of them -- and the doubled-brace convention fails silently in the + other direction the moment somebody next edits the C by hand. `{{` is + unescaped and only the named placeholders are substituted, so ordinary C + braces pass through untouched. + """ + text = template.replace("{{", "{").replace("}}", "}") + for key, value in kw.items(): + text = text.replace("{%s}" % key, str(value)) + return text + + +def _rule_table_c(rules): + """Render the JSON rule set as a C table.""" + errno_map = {"ENOENT": 2, "EACCES": 13, "EINVAL": 22, "EPERM": 1} + action_map = {"deny": "MASK_DENY", "fake_size": "MASK_FAKE_SIZE", + "filter_line": "MASK_FILTER_LINE"} + lines = [] + for rule in rules: + path = rule.get("path") + prefix = rule.get("prefix") + action = action_map.get(rule.get("action"), "MASK_DENY") + err = errno_map.get(rule.get("errno", "ENOENT"), 2) + value = rule.get("size", 0) + lines.append(' { %s, %s, %s, -%d, %d },' + % ('"%s"' % path if path else "0", + '"%s"' % prefix if prefix else "0", + action, err, value)) + return "\n".join(lines) if lines else " /* no rules configured */" + + +def cmd_gates(args): + """Report the kernel-route gate table, and probe this host where it can be probed.""" + print("== kernel-side routes and their gates ==\n") + for g in GATES: + print(" %-28s %s" % (g["route"], g["gate"])) + print(" %-28s why: %s" % ("", g["why"])) + print(" %-28s check: %s" % ("", g["check"])) + print() + print("== host probe (what can be checked from here) ==\n") + cc = None + import shutil + for cand in ("aarch64-none-elf-gcc", "aarch64-linux-gnu-gcc", "clang"): + found = shutil.which(cand) + if found: + cc = (cand, found) + break + if cc: + print(" bare-metal ARM64-ish compiler: %s -> %s" % cc) + if cc[0] == "clang": + print(" note: clang can target aarch64-none-elf, so this is " + "potentially usable for the KPM route") + else: + print(" bare-metal ARM64 compiler: NOT FOUND") + print(" the KPM route cannot be built from this host as configured") + for name in ("ndk-build", "make"): + print(" %-12s %s" % (name + ":", shutil.which(name) or "NOT FOUND")) + print("\n A kernel source tree, a patched boot image and the device's own " + "kernel headers cannot be checked from the host; they are device facts.") + print(" Run the `check` command for each route on the device before " + "believing any of this applies to it.") + return 0 + + +def cmd_generate(args): + out = args.out + rules = DEFAULT_RULES + if args.rules: + with open(args.rules, encoding="utf-8") as fh: + rules = json.load(fh) + desc = args.description + os.makedirs(os.path.join(out, "config"), exist_ok=True) + + written = [] + + def put(rel, text, mode=None): + path = os.path.join(out, rel) + os.makedirs(os.path.dirname(path), exist_ok=True) + with open(path, "w", encoding="utf-8", newline="\n") as fh: + fh.write(text) + if mode is not None: + try: + os.chmod(path, mode) + except OSError: + pass + written.append(rel) + + put("module.prop", _render(MODULE_PROP, + id=args.id, name=args.name, version=args.version, + description=desc)) + put("customize.sh", CUSTOMIZE, 0o755) + put("post-fs-data.sh", _render(POST_FS_DATA, id=args.id), 0o755) + put("service.sh", _render(SERVICE, id=args.id), 0o755) + put("uninstall.sh", _render(UNINSTALL, id=args.id), 0o755) + put("config/syscall_mask.json", json.dumps( + {"version": 1, "id": args.id, "package": args.package, + "enabled": True, "note": "read by the kernel side, not by the shell " + "scripts -- see the module README", + "rules": rules}, indent=2) + "\n") + + want = set(args.targets.split(",")) + if "kpm" in want: + body = _render(KPM_C, id=args.id, version=args.version, description=desc, + RULE_TABLE=_rule_table_c(rules)) + put("kpm/%s.c" % args.id, body) + put("kpm/Makefile", _render(KPM_MAKEFILE, id=args.id)) + if "lkm" in want: + put("lkm/%s.c" % args.id, _render(LKM_C, id=args.id, description=desc)) + if "ebpf" in want: + put("ebpf/syscall_mask.bpf.c", EBPF_C) + + gate_rows = "\n".join("| %s | %s | `%s` | %s |" + % (g["route"], g["gate"], g["check"], g["why"]) + for g in GATES) + put("README.md", _render(MODULE_README, name=args.name, gates=gate_rows, + id=args.id)) + + print("generated module scaffold in %s" % out) + for rel in written: + size = os.path.getsize(os.path.join(out, rel)) + print(" %-28s %7d B" % (rel, size)) + print() + print("STATUS: the userspace module is installable; every kernel-side file is " + "an UNVERIFIED template.") + print(" No kernel code was compiled or loaded. Run `gates` and read the " + "device README") + print(" section before assuming any route is open on your target.") + return 0 + + +def _strip_c_comments(text): + """Remove /* */ and // comments before any token-level check. + + Not cosmetic: the first version of this check reported a false positive + because the template's own comment explains why `extern` is forbidden, and a + naive `extern.*kfunc_def` regex happily matched the prose and then ran on to + the real declaration's semicolon. A linter that reads comments as code + produces exactly the kind of report nobody can trust. + """ + text = re.sub(r"/\*.*?\*/", "", text, flags=re.S) + return re.sub(r"//[^\n]*", "", text) + + +def cmd_verify(args): + """Read-only checks over a generated directory. Everything here is local.""" + d = args.dir + problems = [] + print("== verifying %s ==" % d) + + prop = os.path.join(d, "module.prop") + if not os.path.isfile(prop): + problems.append("module.prop missing") + else: + fields = {} + with open(prop, encoding="utf-8") as fh: + for line in fh: + if "=" in line: + k, v = line.rstrip("\n").split("=", 1) + fields[k] = v + for req in ("id", "name", "version", "versionCode"): + if req not in fields: + problems.append("module.prop missing required field %r" % req) + if "id" in fields and not re.fullmatch(r"[a-zA-Z][a-zA-Z0-9._-]*", fields["id"]): + problems.append("module.prop id %r is not a valid module id" % fields["id"]) + print(" module.prop: %s" % ", ".join("%s=%s" % kv for kv in sorted(fields.items()))) + + conf = os.path.join(d, "config", "syscall_mask.json") + if not os.path.isfile(conf): + problems.append("config/syscall_mask.json missing") + else: + try: + with open(conf, encoding="utf-8") as fh: + doc = json.load(fh) + rules = doc.get("rules", []) + known = {"deny", "fake_size", "filter_line"} + for i, r in enumerate(rules): + if r.get("action") not in known: + problems.append("rule %d has unknown action %r" % (i, r.get("action"))) + if not r.get("syscall"): + problems.append("rule %d names no syscall" % i) + print(" config: %d rule(s), actions=%s" + % (len(rules), sorted({r.get("action") for r in rules}))) + except (ValueError, OSError) as exc: + problems.append("config/syscall_mask.json unreadable: %s" % exc) + + for rel, want_exec in (("post-fs-data.sh", True), ("service.sh", True), + ("customize.sh", True), ("uninstall.sh", True)): + p = os.path.join(d, rel) + if not os.path.isfile(p): + problems.append("%s missing" % rel) + continue + if want_exec and not os.access(p, os.X_OK): + # Not fatal on a filesystem without an exec bit; the installer sets it. + print(" note: %s is not executable here (the installer chmods it)" % rel) + + for rel in ("kpm", "lkm", "ebpf"): + p = os.path.join(d, rel) + if os.path.isdir(p): + print(" %-4s target present: %s" + % (rel, ", ".join(sorted(os.listdir(p))))) + print(" status: UNVERIFIED TEMPLATE -- not compiled, not loaded") + + # The one static check worth doing on the C: the KPM interface has a + # documented trap (an extern kfunc declaration compiles to *UND* and the + # loader refuses the module), so catch it here rather than on a device. + kpm_dir = os.path.join(d, "kpm") + if os.path.isdir(kpm_dir): + for name in sorted(os.listdir(kpm_dir)): + if not name.endswith(".c"): + continue + with open(os.path.join(kpm_dir, name), encoding="utf-8") as fh: + text = fh.read() + code = _strip_c_comments(text) + if re.search(r"\bextern\b[^;]*\bkfunc_def\b", code): + problems.append("%s declares kfunc_def with `extern`, which the " + "loader rejects as an unknown symbol" % name) + for macro in ("KPM_NAME", "KPM_INIT", "KPM_EXIT"): + if macro not in code: + problems.append("%s is missing %s" % (name, macro)) + print(" kpm/%s: KPM_* lifecycle macros present, kfunc_def not extern" + % name) + + print() + if problems: + print("== %d problem(s) ==" % len(problems)) + for p in problems: + print(" - %s" % p) + return 1 + print("== 0 problem(s): structure is consistent ==") + print(" (this says nothing about whether the kernel side compiles or loads)") + return 0 + + +def main(argv=None): + parser = argparse.ArgumentParser( + prog="kernelsu_syscall_mask.py", + description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter) + sub = parser.add_subparsers(dest="cmd") + + p = sub.add_parser("gates", help="print the kernel-route gate table and probe this host", + description="The gate table, plus whatever part of it can be " + "checked from this machine. Device facts still have " + "to be checked on the device.") + p.set_defaults(func=cmd_gates) + + p = sub.add_parser("generate", help="write the module scaffold", + description="Generate a KernelSU/APatch-compatible module " + "directory. The userspace half is installable; the " + "kernel half is a template.") + p.add_argument("--out", required=True, help="directory to write into") + p.add_argument("--id", default="syscall-mask", help="module id / artifact name") + p.add_argument("--name", default="Syscall Mask", help="human-readable module name") + p.add_argument("--version", default="0.1.0", help="module version string") + p.add_argument("--description", default="config-driven syscall return masking " + "(openat/read/stat on /proc/self/*)", + help="module description") + p.add_argument("--package", default="", help="target package placeholder") + p.add_argument("--rules", default=None, + help="JSON file with a {rules:[...]} rule set (default: built-in)") + p.add_argument("--targets", default="kpm,lkm,ebpf", + help="comma-separated kernel targets to emit (kpm,lkm,ebpf)") + p.set_defaults(func=cmd_generate) + + p = sub.add_parser("verify", help="read-only consistency checks over a generated dir", + description="Check module.prop fields, the rule JSON, and the " + "static traps in the kernel template. Says nothing " + "about compilation or loading.") + p.add_argument("--dir", required=True, help="a directory from `generate`") + p.set_defaults(func=cmd_verify) + + args = parser.parse_args(argv) + if not getattr(args, "cmd", None): + parser.print_help() + return 2 + return args.func(args) + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/skills/apk-reverse/scripts/protobuf_decode_raw.py b/skills/apk-reverse/scripts/protobuf_decode_raw.py new file mode 100644 index 0000000..14a2cc5 --- /dev/null +++ b/skills/apk-reverse/scripts/protobuf_decode_raw.py @@ -0,0 +1,1297 @@ +#!/usr/bin/env python3 +# -*- coding: utf-8 -*- +"""Decode a protobuf payload that has NO schema into a structured JSON tree. + +WHY THIS EXISTS +--------------- +`references/protocol-reverse.md` states the wire-format rules and says a payload +can be walked field by field with no `.proto` in hand, but the skill shipped no +tool that does it -- the only evidence behind that section was a workbench +script. The honest part of "walk it without a schema" is what a naive decoder +hides: + + * four different things share wire type 2 (nested message, UTF-8 string, + packed repeated array, opaque bytes), and nothing on the wire separates + them; + * a proto3 varint 0 is byte-identical to an absent field, so a decoded `0` is + not evidence that the sender set the field. + +A decoder that prints one interpretation per field manufactures conclusions. +This one prints every interpretation consistent with the bytes, labels each +with a heuristic confidence, and marks the ones the format cannot decide. + +WHAT IT DOES +------------ + * varint, with the 10-byte ceiling and 64-bit overflow made explicit, plus + the non-canonical (redundant) encoding case; + * wire types 0/1/2/5 and the deprecated 3/4 group pair; + * length-delimited values as candidate sets, each with a confidence and, when + two candidates are equally consistent, an explicit note that no schema + separates them; + * proto3 explicit zero flagged at every occurrence; + * several frames in one input: a `|` separator inside --hex, or a varint + length prefix with --split varint-length (gRPC/WebSocket-style framing); + * --reencode: write a decoded JSON tree -- possibly hand-edited -- back to + bytes and compare with the original. That comparison is the round-trip + proof that the walk lost nothing. + +No third-party dependency: `google.protobuf` parses a *stream against a schema* +and cannot walk a bare one, so the parsing here is written out. (It is still +useful to *generate* reference bytes from a real schema -- see the verification +record -- but the tool never imports it.) + +USAGE +----- + python protobuf_decode_raw.py --hex "08 96 01 12 07 74 65 73 74 69 6e 67" + python protobuf_decode_raw.py body.bin + python protobuf_decode_raw.py body.bin --json > tree.json + python protobuf_decode_raw.py - < body.bin + python protobuf_decode_raw.py --hex "089601|120774657374696e67" # two frames + python protobuf_decode_raw.py framed.bin --split varint-length + python protobuf_decode_raw.py --reencode tree.json --out patched.bin --check body.bin + python protobuf_decode_raw.py --selftest + +Stdlib only, Python 3.9+. +""" + +import argparse +import json +import re +import struct +import sys + +MAX_VARINT_BYTES = 10 +MASK64 = (1 << 64) - 1 +MASK32 = (1 << 32) - 1 + +WIRE_NAMES = { + 0: 'varint', + 1: 'fixed64', + 2: 'length_delimited', + 3: 'group_start', + 4: 'group_end', + 5: 'fixed32', +} + +WIRE_SEMANTICS = { + 0: 'int32/int64/uint32/uint64/sint32/sint64/bool/enum', + 1: 'fixed64/sfixed64/double', + 2: 'message/string/bytes/packed repeated', + 3: 'deprecated group start', + 4: 'deprecated group end', + 5: 'fixed32/sfixed32/float', +} + +# Tie-break order for "which candidate to expand by default". Structural +# candidates come first because they carry more information; `bytes` is last +# because it always fits. This is a display default, never a conclusion. +VIEW_PRIORITY = ['nested_message', 'utf8_string', 'packed_varint', + 'packed_fixed32', 'packed_fixed64', 'bytes', 'empty'] + +CAVEATS = [ + 'wire type 2 carries four different things; without a schema the bytes cannot ' + 'separate them, so every length-delimited field lists all candidates that fit.', + 'a packed repeated field is indistinguishable from an opaque bytes value: the ' + 'total length is known, the element boundaries are not.', + 'a varint 0 on the wire was emitted on purpose by some writer; for a field ' + 'WITHOUT presence, value 0 and "never set" are the same bytes (nothing at ' + 'all), so a field missing from a decode is not evidence that 0 was the value.', + 'field numbers are local to their message; nothing on the wire says which ' + 'message a field number belongs to.', + 'confidence values are a documented heuristic ordering, not probabilities.', +] + + +class UsageError(Exception): + """Bad input or bad flags: printed as one line, exit code 2.""" + + +# --------------------------------------------------------------- primitives + +def varint_len(value): + """Number of bytes a canonical (minimal) varint needs for an unsigned value.""" + if value < 0: + value &= MASK64 + n = 1 + while value >= 0x80: + value >>= 7 + n += 1 + return n + + +def encode_varint(value): + """Canonical base-128 varint. Negative values are written as 64-bit two's complement.""" + if value < 0: + value &= MASK64 + out = bytearray() + while True: + b = value & 0x7F + value >>= 7 + if value: + out.append(b | 0x80) + else: + out.append(b) + return bytes(out) + + +def read_varint(buf, i, end): + """Read one varint from buf[i:end]. + + Returns (info, next_index, None) on success and (None, i, reason) when no + complete varint could be read. `info` always carries: + + value the decoded integer (may exceed 64 bits when overflow) + hex / bytes the exact bytes consumed + canonical_bytes what the minimal encoding of `value` would cost + non_canonical True when the wire used more bytes than necessary + overflow True when byte 10 carried bits above 2**64 + too_long True when the varint ran past 10 bytes + """ + start = i + value = 0 + count = 0 + overflow = False + too_long = False + while True: + if i >= end: + return None, start, 'truncated varint at offset %d' % start + b = buf[i] + i += 1 + count += 1 + if count <= 9: + value |= (b & 0x7F) << (7 * (count - 1)) + elif count == 10: + if (b & 0x7F) > 1: + overflow = True + value |= (b & 0x7F) << 63 + else: + too_long = True + if not (b & 0x80): + break + raw = buf[start:i] + canonical = varint_len(value & MASK64) if not overflow else varint_len(value) + info = { + 'value': value, + 'hex': raw.hex(), + 'bytes': count, + 'canonical_bytes': canonical, + 'non_canonical': count > canonical, + 'overflow': overflow, + 'too_long': too_long, + } + return info, i, None + + +def zigzag_decode(value): + return (value >> 1) ^ -(value & 1) + + +def int32_view(value): + v = value & MASK32 + return v - (1 << 32) if v >= (1 << 31) else v + + +def int64_view(value): + v = value & MASK64 + return v - (1 << 64) if v >= (1 << 63) else v + + +def _float_repr(x): + """repr() of a float, never NaN/Infinity literals (illegal in strict JSON).""" + return repr(x) + + +def ascii_escape(text): + """Make a decoded string safe to print on any console: pure ASCII output.""" + return text.encode('unicode_escape').decode('ascii') + + +# ----------------------------------------------------------------- decode + +class Ctx(object): + """Collects notes (ambiguities) and errors while walking.""" + + def __init__(self, max_depth=5, expand=True, silent=False): + self.max_depth = max_depth + self.expand = expand + self.silent = silent + self.notes = [] + self.errors = [] + + def note(self, kind, path, offset, detail, **extra): + if self.silent: + return + item = {'kind': kind, 'path': path, 'offset': offset, 'detail': detail} + item.update(extra) + self.notes.append(item) + + def error(self, kind, path, offset, detail): + self.errors.append({'kind': kind, 'path': path, 'offset': offset, + 'detail': detail}) + + +def walk(buf, start, end, depth, ctx, path, until_group=None, analyse=True): + """Walk buf[start:end] as one message body. + + Returns (fields, next_index, status) where status is 'end_of_buffer' when + the region was consumed exactly, 'end_group' when the matching deprecated + end-group tag closed it, or 'stopped:' when it could not continue. + """ + fields = [] + i = start + while i < end: + tag_off = i + info, j, reason = read_varint(buf, i, end) + if info is None: + ctx.error('truncated_tag', path, tag_off, reason) + return fields, i, 'stopped:truncated_tag' + if info['too_long'] or info['overflow']: + ctx.error('varint_overflow', path, tag_off, + 'tag varint at offset %d is %s (%d bytes, value needs %d); ' + 'the walk cannot resynchronise here' + % (tag_off, 'longer than 10 bytes' if info['too_long'] + else 'wider than 64 bits', info['bytes'], + info['canonical_bytes'])) + return fields, j, 'stopped:varint_overflow' + key = info['value'] + i = j + field_no = key >> 3 + wire = key & 7 + if field_no == 0: + ctx.note('field_number_zero', path, tag_off, + 'tag 0x%x decodes to field number 0, which is illegal in ' + 'protobuf; this is a decode error, not a field' + % key) + + fpath = '%s.%d' % (path, field_no) + + if wire == 4: + if until_group is not None and field_no == until_group: + return fields, i, 'end_group' + ctx.error('stray_end_group', path, tag_off, + 'end-group tag for field %d at offset %d has no matching ' + 'group start' % (field_no, tag_off)) + return fields, i, 'stopped:stray_end_group' + + if wire not in (0, 1, 2, 3, 5): + ctx.error('invalid_wire_type', path, tag_off, + 'wire type %d at offset %d does not exist in protobuf' + % (wire, tag_off)) + return fields, i, 'stopped:invalid_wire_type' + + field = { + 'field': field_no, + 'wire': wire, + 'wire_name': WIRE_NAMES[wire], + 'wire_semantics': WIRE_SEMANTICS[wire], + 'offset': tag_off, + 'value_offset': i, + 'raw_hex': '', + } + + if wire == 0: + vinfo, j2, vreason = read_varint(buf, i, end) + if vinfo is None: + ctx.error('truncated_value', path, i, vreason) + return fields, i, 'stopped:truncated_value' + value = vinfo['value'] + field['view'] = 'varint' + field['value'] = value + field['varint'] = { + 'hex': vinfo['hex'], + 'bytes': vinfo['bytes'], + 'canonical_bytes': vinfo['canonical_bytes'], + 'non_canonical': vinfo['non_canonical'], + 'overflow': vinfo['overflow'], + 'too_long': vinfo['too_long'], + 'as_uint64': value & MASK64, + 'as_int64': int64_view(value), + 'as_int32': int32_view(value), + 'as_sint64_zigzag': zigzag_decode(value), + 'as_bool_if_0_or_1': None if value > 1 else bool(value), + } + i = j2 + if vinfo['non_canonical']: + ctx.note('non_canonical_varint', fpath, field['value_offset'], + 'field %d uses %d bytes where the minimal varint encoding ' + 'needs %d; a round-trip re-encode emits the minimal form' + % (field_no, vinfo['bytes'], vinfo['canonical_bytes'])) + if vinfo['overflow'] or vinfo['too_long']: + ctx.note('varint_over_64_bits', fpath, field['value_offset'], + 'field %d consumed %d bytes carrying more than 64 bits; ' + 'protobuf int64/uint64 cannot hold this value, so the ' + 'field is either not a varint or the stream is corrupt' + % (field_no, vinfo['bytes'])) + if value == 0: + ctx.note('proto3_explicit_zero', fpath, field['value_offset'], + 'field %d carries an explicit varint 0, so some writer ' + 'chose to emit it (a field WITH presence -- proto2 or ' + 'proto3 optional -- or a hand-rolled writer). The ' + 'converse matters more: for a field WITHOUT presence, ' + 'value 0 and "never set" are the same bytes (nothing at ' + 'all), so a field missing from this decode is not ' + 'evidence that the value in use was 0' % field_no) + + elif wire == 1: + if i + 8 > end: + ctx.error('truncated_fixed64', path, i, + 'fixed64 for field %d needs 8 bytes, only %d remain' + % (field_no, end - i)) + return fields, i, 'stopped:truncated_fixed64' + raw = buf[i:i + 8] + i += 8 + field['view'] = 'fixed64' + field['fixed64_hex'] = raw.hex() + field['fixed64'] = { + 'as_uint64': struct.unpack(' end: + ctx.error('truncated_fixed32', path, i, + 'fixed32 for field %d needs 4 bytes, only %d remain' + % (field_no, end - i)) + return fields, i, 'stopped:truncated_fixed32' + raw = buf[i:i + 4] + i += 4 + field['view'] = 'fixed32' + field['fixed32_hex'] = raw.hex() + field['fixed32'] = { + 'as_uint32': struct.unpack(' end: + ctx.error('length_overruns_buffer', path, value_off, + 'field %d declares %d payload bytes at offset %d but ' + 'only %d remain' % (field_no, ln, value_off, end - value_off)) + return fields, i, 'stopped:length_overruns_buffer' + chunk = buf[j2:payload_end] + field['view'] = 'bytes' + field['length'] = ln + field['payload_hex'] = chunk.hex() + field['length_prefix'] = { + 'hex': linfo['hex'], + 'bytes': linfo['bytes'], + 'non_canonical': linfo['non_canonical'], + } + if analyse: + cands, view = analyse_payload(chunk, depth, ctx, fpath, value_off) + field['candidates'] = cands + field['view'] = view + field['view_is_heuristic'] = True + if view == 'nested_message': + if not ctx.expand: + field['expansion'] = 'suppressed by --no-expand' + else: + children, _nxt, _status = walk(chunk, 0, len(chunk), + depth + 1, ctx, fpath, + analyse=True) + field['children'] = children + elif view == 'utf8_string': + field['string'] = chunk.decode('utf-8', 'replace') + elif view == 'packed_varint': + field['packed_values'] = _packed_varint_values(chunk) + elif view in ('packed_fixed32', 'packed_fixed64'): + step = 4 if view == 'packed_fixed32' else 8 + field['packed_values_hex'] = [ + chunk[k:k + step].hex() for k in range(0, len(chunk), step)] + i = payload_end + + else: # wire == 3, deprecated group + if depth >= ctx.max_depth: + ctx.note('depth_limit', fpath, tag_off, + 'group at depth %d was not walked: --max-depth is %d' + % (depth, ctx.max_depth)) + field['view'] = 'group' + field['children'] = [] + field['truncated_by_depth'] = True + i = end + else: + children, j2, status = walk(buf, i, end, depth + 1, ctx, fpath, + until_group=field_no, analyse=analyse) + field['view'] = 'group' + field['children'] = children + field['group_deprecated'] = True + ctx.note('deprecated_group', fpath, tag_off, + 'field %d uses the deprecated group wire type (3/4); ' + 'groups were removed from the language, modern writers ' + 'do not emit them' % field_no) + if status != 'end_group': + ctx.error('unterminated_group', fpath, tag_off, + 'group for field %d was never closed by a matching ' + 'end-group tag' % field_no) + i = j2 + else: + i = j2 + + field['end'] = i + field['raw_hex'] = buf[tag_off:i].hex() + fields.append(field) + return fields, i, ('end_of_buffer' if i == end else 'stopped:partial') + + +def _packed_varint_values(chunk): + values = [] + k = 0 + while k < len(chunk): + info, k2, _reason = read_varint(chunk, k, len(chunk)) + if info is None: + break + values.append(info['value']) + k = k2 + return values + + +def analyse_payload(chunk, depth, ctx, path, offset): + """Every interpretation of a length-delimited payload that fits the bytes. + + Returns (candidates, view). `view` is the highest-confidence candidate -- + a default for expansion and re-encoding, explicitly not a conclusion. + """ + n = len(chunk) + cands = [] + + if n == 0: + cands.append({'kind': 'empty', 'confidence': 1.0, + 'note': 'zero-length payload: every interpretation is ' + 'vacuous, nothing can be ranked'}) + return cands, 'empty' + + # --- nested message ------------------------------------------------- + nested_ok = False + if depth < ctx.max_depth: + sub = Ctx(ctx.max_depth, ctx.expand, silent=True) + sub_fields, nxt, status = walk(chunk, 0, n, depth + 1, sub, path, + analyse=False) + if status == 'end_of_buffer' and nxt == n and not sub.errors and sub_fields: + nested_ok = True + cands.append({ + 'kind': 'nested_message', + 'confidence': 0.85, + 'fields_count': len(sub_fields), + 'note': 'the payload consumes exactly as a message body (%d field(s))' + % len(sub_fields), + }) + else: + reason = ('walk stopped: %s' % status if status != 'end_of_buffer' + else 'walk left %d byte(s) unread' % (n - nxt)) + cands.append({'kind': 'nested_message', 'confidence': 0.0, + 'rejected': reason, + 'note': 'rejected: %s' % reason}) + else: + cands.append({'kind': 'nested_message', 'confidence': 0.0, + 'rejected': 'depth limit', + 'note': 'not attempted: --max-depth %d reached' % ctx.max_depth}) + ctx.note('depth_limit', path, offset, + 'a nested reading of this field was not attempted: --max-depth %d ' + 'reached. The bytes here may well be a message; re-run with a ' + 'larger --max-depth before concluding that they are not' + % ctx.max_depth) + + # --- UTF-8 string --------------------------------------------------- + text = None + try: + text = chunk.decode('utf-8') + except UnicodeDecodeError as exc: + cands.append({'kind': 'utf8_string', 'confidence': 0.0, + 'rejected': 'not valid UTF-8 (%s)' % exc.reason, + 'note': 'rejected: %s at byte %d' % (exc.reason, exc.start)}) + if text is not None: + controls = [ch for ch in text if ord(ch) < 0x20 and ch not in '\t\n\r'] + if not controls: + conf = 0.9 + note = 'valid UTF-8 with no control characters' + else: + conf = 0.35 + note = ('valid UTF-8 but carries %d control character(s) (0x00-0x1f); ' + 'binary data commonly decodes as UTF-8 by accident' + % len(controls)) + cands.append({'kind': 'utf8_string', 'confidence': conf, + 'length_chars': len(text), 'note': note}) + + # --- packed arrays -------------------------------------------------- + if n % 4 == 0 and n >= 8: + cands.append({'kind': 'packed_fixed32', 'confidence': 0.3, + 'elements': n // 4, 'element_hex': _element_preview(chunk, 4), + 'note': 'length divisible by 4: could be %d packed fixed32 ' + 'or float values -- element boundaries are only a ' + 'guess without a schema' % (n // 4)}) + if n % 8 == 0 and n >= 16: + cands.append({'kind': 'packed_fixed64', 'confidence': 0.3, + 'elements': n // 8, 'element_hex': _element_preview(chunk, 8), + 'note': 'length divisible by 8: could be %d packed fixed64 ' + 'or double values -- element boundaries are only a ' + 'guess without a schema' % (n // 8)}) + + packed_values = _packed_varint_values(chunk) + if packed_values and len(b''.join(encode_varint(v) for v in packed_values)) == n \ + and len(chunk) == sum(varint_len(v) for v in packed_values): + conf = 0.55 if len(packed_values) >= 2 else 0.25 + cands.append({'kind': 'packed_varint', 'confidence': conf, + 'elements': len(packed_values), + 'note': 'the payload is exactly a sequence of %d varint(s). ' + 'Without a schema this is a candidate, not a ' + 'reading: a packed repeated field is ' + 'indistinguishable from an opaque bytes value, so ' + 'the element boundaries are not knowable from the ' + 'bytes alone' % len(packed_values)}) + + # --- opaque bytes --------------------------------------------------- + # A fallback reading, not an equal-ranking candidate: it is consistent with + # every payload, so scoring it like the others would let it win ties on + # alphabetical order and silently suppress the useful readings. + cands.append({'kind': 'bytes', 'confidence': 0.2, + 'note': 'opaque bytes: always consistent with the payload, and ' + 'the only honest reading when no other candidate can be ' + 'confirmed against a schema'}) + + # --- resolve ties --------------------------------------------------- + live = [c for c in cands if c['confidence'] > 0.0] + kinds = set(c['kind'] for c in live) + competing = [k for k in ('nested_message', 'utf8_string', 'packed_varint') + if k in kinds] + if len(competing) > 1: + msg = ('%s are equally consistent with these bytes; without a schema ' + 'nothing on the wire separates them, so the view below is a ' + 'display default and NOT a reading' + % ' and '.join(sorted(competing))) + for c in live: + if c['kind'] != 'bytes': + c['confidence'] = min(c['confidence'], 0.5) + c['ambiguous_with'] = sorted(k for k in competing if k != c['kind']) + c['tie_note'] = msg + + cands.sort(key=lambda c: (-c['confidence'], + VIEW_PRIORITY.index(c['kind']) + if c['kind'] in VIEW_PRIORITY else 99)) + view = cands[0]['kind'] if cands else 'bytes' + if view == 'empty': + view = 'bytes' + return cands, view + + +def _element_preview(chunk, step, limit=4): + out = [chunk[k:k + step].hex() for k in range(0, min(len(chunk), step * limit), step)] + if len(chunk) > step * limit: + out.append('...') + return out + + +# ---------------------------------------------------------------- re-encode + +def payload_from_view(field): + view = field.get('view') or 'bytes' + if view in ('nested_message', 'group'): + return b''.join(reencode_field(c, None, '') for c in field.get('children') or []) + if view == 'utf8_string': + return (field.get('string') or '').encode('utf-8') + if view == 'packed_varint': + return b''.join(encode_varint(int(v)) for v in (field.get('packed_values') or [])) + if view in ('packed_fixed32', 'packed_fixed64'): + out = b'' + for h in field.get('packed_values_hex') or []: + out += bytes.fromhex(h) + return out + return bytes.fromhex(field.get('payload_hex') or '') + + +def reencode_field(field, report, path): + """Encode one field node. When `report` is given it is filled with the diff.""" + wire = field['wire'] + num = int(field['field']) + key = encode_varint((num << 3) | wire) + here = path or str(num) + + if wire == 0: + body = encode_varint(int(field['value'])) + elif wire == 1: + body = bytes.fromhex(field['fixed64_hex']) + elif wire == 5: + body = bytes.fromhex(field['fixed32_hex']) + elif wire == 2: + payload = payload_from_view(field) + body = encode_varint(len(payload)) + payload + elif wire == 3: + inner = b'' + for child in field.get('children') or []: + inner += reencode_field(child, report, '%s.%s' % (here, child['field'])) + body = inner + encode_varint((num << 3) | 4) + else: + raise UsageError('wire type %d cannot be re-encoded' % wire) + + raw = key + body + if report is not None: + report['fields'] += 1 + expected = field.get('raw_hex') + if expected is not None and raw.hex() != expected: + reason = 'the tree was edited' + vinfo = field.get('varint') or {} + if vinfo.get('non_canonical'): + reason = ('the original varint was non-canonical (%d bytes for a ' + '%d-byte value); the re-encode emits the minimal form' + % (vinfo.get('bytes'), vinfo.get('canonical_bytes'))) + report['mismatched'].append({ + 'path': here, 'expected_hex': expected, 'got_hex': raw.hex(), + 'reason': reason, + }) + else: + report['identical'] += 1 + return raw + + +def reencode_tree(tree, report): + frames = tree.get('frames') or [] + out = b'' + for frame in frames: + for field in frame.get('fields') or []: + out += reencode_field(field, report, str(field['field'])) + return out + + +# ------------------------------------------------------------------- input + +HEX_ESCAPE_RE = re.compile(r'\\(x[0-9A-Fa-f]{2}|[nrt0\\])') +HEX_ONLY_RE = re.compile(r'[0-9A-Fa-f\s,;|]') +ESCAPE_MAP = {'n': b'\n', 'r': b'\r', 't': b'\t', '0': b'\x00', '\\': b'\\'} + + +def _unescape(text): + """Turn \\xNN / \\n / \\t / \\\\ escapes into bytes.""" + out = bytearray() + i = 0 + while i < len(text): + ch = text[i] + if ch == '\\' and i + 1 < len(text): + nxt = text[i + 1] + if nxt == 'x' and i + 3 < len(text) + 1: + pair = text[i + 2:i + 4] + if len(pair) == 2 and re.fullmatch(r'[0-9A-Fa-f]{2}', pair): + out.append(int(pair, 16)) + i += 4 + continue + if nxt in ESCAPE_MAP: + out += ESCAPE_MAP[nxt] + i += 2 + continue + out += ch.encode('utf-8') + i += 1 + return bytes(out) + + +def parse_hex_frames(text): + """Split on '|' / ';' and decode each part as hex bytes. + + Accepts `08 96 01`, `0x08 0x96 0x01`, newlines, commas and `\\x08\\x96\\x01`. + A part that is not hex at all is taken as literal text bytes, so a pasted + ASCII body still decodes. + """ + frames = [] + for part in re.split(r'[|;]', text): + if not part.strip(): + continue + if '\\x' in part or '\\n' in part or '\\t' in part: + # Whitespace and commas are separators here too: in `08 96 | \x12\x07`, + # the space after the bar is layout, not a 0x20 byte. + frames.append(_unescape(re.sub(r'[\s,]', '', part))) + continue + cleaned = re.sub(r'0[xX]', '', part) + cleaned = re.sub(r'[\s,]', '', cleaned) + if not cleaned: + continue + if not re.fullmatch(r'[0-9A-Fa-f]+', cleaned): + frames.append(part.encode('utf-8')) + continue + if len(cleaned) % 2: + raise UsageError('odd number of hex digits in a frame (%d): a hex ' + 'string must describe whole bytes' % len(cleaned)) + frames.append(bytes.fromhex(cleaned)) + if not frames: + raise UsageError('no bytes in the hex input after stripping separators') + return frames + + +def looks_like_hex_text(text): + stripped = text.strip() + if not stripped: + return False + try: + stripped.encode('ascii') + except UnicodeEncodeError: + return False + body = HEX_ESCAPE_RE.sub('aa', stripped) + body = re.sub(r'0[xX]', '', body) + body = re.sub(r'[\s,;|]', '', body) + if not body or not re.fullmatch(r'[0-9A-Fa-f]+', body): + return False + return len(body) % 2 == 0 + + +def split_varint_length(buf): + """Frames of the form , as gRPC/WebSocket use.""" + frames = [] + i = 0 + while i < len(buf): + info, j, reason = read_varint(buf, i, len(buf)) + if info is None: + raise UsageError('frame length prefix at offset %d: %s' % (i, reason)) + if info['too_long'] or info['overflow']: + raise UsageError('frame length prefix at offset %d overflows 64 bits' % i) + ln = info['value'] + if j + ln > len(buf): + raise UsageError('frame at offset %d declares %d payload bytes but only ' + '%d remain' % (i, ln, len(buf) - j)) + frames.append((j, j + ln, i)) + i = j + ln + return frames + + +def load_input(args): + """Return (buffer, frames) where frames is a list of (start, end, prefix_offset).""" + if args.hex is not None: + parts = parse_hex_frames(args.hex) + buf = b''.join(parts) + frames = [] + pos = 0 + for part in parts: + frames.append((pos, pos + len(part), None)) + pos += len(part) + return buf, frames, '--hex' + + if args.input is None or args.input == '-': + data = sys.stdin.buffer.read() + source = 'stdin' + try: + text = data.decode('utf-8') + except UnicodeDecodeError: + text = None + if text is not None and looks_like_hex_text(text): + parts = parse_hex_frames(text) + buf = b''.join(parts) + frames = [] + pos = 0 + for part in parts: + frames.append((pos, pos + len(part), None)) + pos += len(part) + return buf, frames, 'stdin (hex text)' + return data, [(0, len(data), None)], 'stdin (raw bytes)' + + try: + with open(args.input, 'rb') as fh: + data = fh.read() + except OSError as exc: + raise UsageError('cannot read %s: %s' % (args.input, exc)) + return data, [(0, len(data), None)], args.input + + +def apply_split(buf, frames, mode): + if mode == 'none': + return frames + split = split_varint_length(buf) + if not split: + raise UsageError('--split varint-length found no frames in %d bytes' % len(buf)) + return split + + +# ------------------------------------------------------------------ output + +def field_line(field, indent): + pad = ' ' * indent + if field['wire'] == 0: + v = field['varint'] + extra = '' + if v['non_canonical']: + extra = ' [non-canonical: %d bytes for a %d-byte value]' % ( + v['bytes'], v['canonical_bytes']) + if v['overflow'] or v['too_long']: + extra += ' [OVERFLOW: wider than 64 bits]' + if field['value'] == 0: + extra += ' [proto3: 0 is byte-identical to an absent field]' + return ['%sf%d varint %d%s' % (pad, field['field'], field['value'], extra)] + if field['wire'] == 1: + d = field['fixed64'] + return ['%sf%d fixed64 %s (u64=%d, double=%s)' + % (pad, field['field'], field['fixed64_hex'], + d['as_uint64'], d['as_double_repr'])] + if field['wire'] == 5: + d = field['fixed32'] + return ['%sf%d fixed32 %s (u32=%d, float=%s)' + % (pad, field['field'], field['fixed32_hex'], + d['as_uint32'], d['as_float_repr'])] + if field['wire'] == 2: + lines = ['%sf%d len-delimited(%d) view=%s' % ( + pad, field['field'], field['length'], field.get('view'))] + ties = [] + for cand in field.get('candidates') or []: + if cand['confidence'] > 0.0: + lines.append('%s candidate %-16s %.2f %s' + % (pad, cand['kind'], cand['confidence'], + ascii_escape(cand['note']))) + elif cand.get('rejected'): + lines.append('%s candidate %-16s rejected: %s' + % (pad, cand['kind'], ascii_escape(str(cand['rejected'])))) + if cand.get('tie_note') and cand['tie_note'] not in ties: + ties.append(cand['tie_note']) + for t in ties: + lines.append('%s tie: %s' % (pad, ascii_escape(t))) + if field.get('view') == 'utf8_string': + lines.append('%s string: %r' % (pad, ascii_escape(field['string']))) + elif field.get('view') == 'nested_message': + for child in field.get('children') or []: + lines.extend(field_line(child, indent + 2)) + if field.get('expansion'): + lines.append('%s expansion: %s' % (pad, field['expansion'])) + elif field.get('view') == 'packed_varint': + lines.append('%s packed varints: %s' + % (pad, field['packed_values'])) + elif field.get('view') in ('packed_fixed32', 'packed_fixed64'): + lines.append('%s elements: %s' % (pad, field['packed_values_hex'])) + elif field.get('view') == 'bytes': + lines.append('%s bytes: %s' % (pad, field['payload_hex'])) + return lines + if field['wire'] == 3: + lines = ['%sf%d group (deprecated wire type)' % (pad, field['field'])] + for child in field.get('children') or []: + lines.extend(field_line(child, indent + 1)) + return lines + return ['%sf%d wire%s' % (pad, field['field'], field['wire'])] + + +def render_human(tree): + out = [] + inp = tree['input'] + out.append('input: %s, %d byte(s)' % (inp['source'], inp['byte_length'])) + out.append('hex: %s' % inp['hex']) + for frame in tree['frames']: + note = '' + if frame.get('prefix_offset') is not None: + note = ' (length-prefixed at offset %d)' % frame['prefix_offset'] + out.append('') + out.append('frame %d bytes %d..%d (%d B)%s status=%s' + % (frame['index'], frame['start'], frame['end'], + frame['length'], note, frame['walk_status'])) + if not frame['fields']: + out.append(' (no fields walked)') + for field in frame['fields']: + out.extend(field_line(field, 1)) + if tree['ambiguities']: + out.append('') + out.append('ambiguities the wire format cannot resolve here:') + for n in tree['ambiguities']: + out.append(' [%s] %s @%d: %s' + % (n['kind'], n['path'], n['offset'], ascii_escape(n['detail']))) + if tree['errors']: + out.append('') + out.append('errors:') + for e in tree['errors']: + out.append(' [%s] %s @%d: %s' + % (e['kind'], e['path'], e['offset'], ascii_escape(e['detail']))) + out.append('') + out.append('fixed caveats (true of every decode in this output):') + for c in CAVEATS: + out.append(' - %s' % c) + return '\n'.join(out) + + +def decode(args): + buf, frames, source = load_input(args) + frames = apply_split(buf, frames, args.split) + ctx = Ctx(args.max_depth, not args.no_expand) + frame_nodes = [] + head = buf[:4096] + input_hex = head.hex() + truncated = len(buf) > len(head) + for idx, frame in enumerate(frames): + start, end = frame[0], frame[1] + prefix = frame[2] if len(frame) > 2 else None + fields, consumed, status = walk(buf, start, end, 0, ctx, 'frame%d' % idx, + analyse=True) + frame_nodes.append({ + 'index': idx, + 'start': start, + 'end': end, + 'length': end - start, + 'prefix_offset': prefix, + 'fields': fields, + 'walk_status': status, + 'bytes_consumed': consumed - start, + }) + tree = { + 'tool': 'protobuf_decode_raw.py', + 'schema': None, + 'input': {'source': source, 'byte_length': len(buf), 'hex': input_hex, + 'hex_truncated': truncated, 'frame_count': len(frame_nodes), + 'split': args.split}, + 'frames': frame_nodes, + 'ambiguities': ctx.notes, + 'errors': ctx.errors, + 'caveats': CAVEATS, + } + if args.json: + print(json.dumps(tree, indent=2, sort_keys=False, ensure_ascii=True)) + else: + print(render_human(tree)) + return 0 if not ctx.errors else 1 + + +def do_reencode(args): + try: + with open(args.reencode, 'r', encoding='utf-8') as fh: + tree = json.load(fh) + except OSError as exc: + raise UsageError('cannot read %s: %s' % (args.reencode, exc)) + except ValueError as exc: + raise UsageError('%s is not valid JSON: %s' % (args.reencode, exc)) + if 'frames' not in tree: + raise UsageError('%s does not look like protobuf_decode_raw.py output ' + '(no "frames" key)' % args.reencode) + + report = {'fields': 0, 'identical': 0, 'mismatched': []} + out = reencode_tree(tree, report) + + if args.out: + with open(args.out, 'wb') as fh: + fh.write(out) + + result = { + 'bytes': len(out), + 'hex': out.hex(), + 'fields': report['fields'], + 'fields_byte_identical': report['identical'], + 'mismatched': report['mismatched'], + 'out_path': args.out, + } + if args.check: + try: + with open(args.check, 'rb') as fh: + original = fh.read() + except OSError as exc: + raise UsageError('cannot read %s: %s' % (args.check, exc)) + result['check_path'] = args.check + result['check_bytes'] = len(original) + split = (tree.get('input') or {}).get('split') + original_cmp = original + scope = 'the whole file' + if split == 'varint-length': + try: + orig_frames = split_varint_length(original) + except UsageError as exc: + raise UsageError('%s does not carry varint-length framing: %s' + % (args.check, exc)) + original_cmp = b''.join(original[s:e] for s, e, _p in orig_frames) + scope = ('frame bodies only: %d of %d byte(s) are length prefixes ' + '(framing, not message data)' + % (len(original) - len(original_cmp), len(original))) + result['check_scope'] = scope + if original_cmp == out: + result['check'] = 'MATCH' + else: + result['check'] = 'MISMATCH' + diff = -1 + for k in range(min(len(original_cmp), len(out))): + if original_cmp[k] != out[k]: + diff = k + break + if diff < 0: + diff = min(len(original_cmp), len(out)) + result['first_difference_offset'] = diff + result['original_at_diff'] = original_cmp[diff:diff + 8].hex() + result['reencoded_at_diff'] = out[diff:diff + 8].hex() + + if args.json: + print(json.dumps(result, indent=2, ensure_ascii=True)) + else: + print('re-encoded %d byte(s) from %d field(s)' + % (result['bytes'], result['fields'])) + print(' fields re-encoded byte-identically: %d/%d' + % (result['fields_byte_identical'], result['fields'])) + for m in result['mismatched']: + print(' CHANGED field %s: expected %s, got %s (%s)' + % (m['path'], m['expected_hex'], m['got_hex'], m['reason'])) + if args.out: + print(' written to %s' % args.out) + else: + print(' hex: %s' % result['hex']) + if args.check: + print(' check against %s (%d B, %s): %s' + % (result['check_path'], result['check_bytes'], + result.get('check_scope', 'the whole file'), result['check'])) + if result['check'] == 'MISMATCH': + off = result['first_difference_offset'] + print(' first difference at offset %d: original %s vs re-encoded %s' + % (off, result['original_at_diff'], result['reencoded_at_diff'])) + return 0 if result.get('check', 'MATCH') == 'MATCH' else 1 + + +# ----------------------------------------------------------------- selftest + +def _fixtures(): + """Known-answer fixtures. Every byte here is produced by hand in this file.""" + def tag(field, wire): + return encode_varint((field << 3) | wire) + + def ld(field, payload): + return tag(field, 2) + encode_varint(len(payload)) + payload + + nested_inner = tag(2, 0) + encode_varint(7) + nested_outer = tag(1, 0) + encode_varint(1) + ld(2, nested_inner) + packed = b''.join(encode_varint(v) for v in (3, 270, 86942)) + main = b''.join([ + tag(1, 0) + encode_varint(150), + ld(2, b'testing'), + ld(3, nested_outer), + ld(4, packed), + tag(5, 0) + encode_varint(0), + tag(6, 1) + struct.pack(' re-encode -> byte comparison ==') + for name, payload, _e, rt in fixtures: + checks += 1 + frames = _fixture_frames(name, payload) + tree = {'frames': [{'fields': walk(payload, s, e, 0, Ctx(6), 'f', + analyse=True)[0]} + for s, e, _p in frames]} + report = {'fields': 0, 'identical': 0, 'mismatched': []} + out = reencode_tree(tree, report) + expect_bytes = b''.join(payload[s:e] for s, e, _p in frames) + if rt == 'non-canonical': + ok = (out != expect_bytes and out.hex() == '089601' + and report['mismatched'] + and 'non-canonical' in report['mismatched'][0]['reason']) + verdict = 'MATCH' if ok else 'FAIL' + detail = ('re-encode emits the minimal form %s, original %s, and the ' + 'report states why' % (out.hex(), expect_bytes.hex())) + else: + ok = out == expect_bytes + verdict = 'MATCH' if ok else 'FAIL' + detail = '' + if not ok: + failures.append('%s: round trip differs (%s vs %s)' + % (name, expect_bytes.hex(), out.hex())) + print(' %-45s %-6s (%d fields, %d identical) %s' + % (name, verdict, report['fields'], report['identical'], detail)) + + # the two documented traps, asserted rather than narrated + print('') + print('== the two wire-format traps ==') + checks += 2 + zero = tag_0(5, 0) + encode_varint(0) + absent = b'' + same = (zero != absent) and walk(zero, 0, len(zero), 0, Ctx(6), 'f')[0][0]['value'] == 0 + ok = walk(absent, 0, 0, 0, Ctx(6), 'f')[0] == [] + if not (same and ok): + failures.append('proto3 explicit zero vs absent field not distinguished correctly') + print(' explicit zero bytes %s decodes to value 0; absent field decodes to no ' + 'fields at all: %s' % (zero.hex(), 'PASS' if same and ok else 'FAIL')) + + packed = b''.join(encode_varint(v) for v in (3, 270, 86942)) + cands, view = analyse_payload(packed, 0, Ctx(6), 'f', 0) + live_kinds = sorted(c['kind'] for c in cands if c['confidence'] > 0.0) + all_kinds = sorted(c['kind'] for c in cands) + ok = ('packed_varint' in live_kinds and 'nested_message' in all_kinds + and view in live_kinds and view == 'packed_varint') + if not ok: + failures.append('packed payload did not offer both candidates: %r' % all_kinds) + print(' packed payload %s offers live=%s, rejected=%s -> view=%s : %s' + % (packed.hex(), live_kinds, + sorted(set(all_kinds) - set(live_kinds)), view, + 'PASS' if ok else 'FAIL')) + print(' -> both readings fit the same bytes; a schema is what decides.') + + print('') + total = len(fixtures) * 2 + 2 + print('selftest: %d/%d PASS' % (len(fixtures) * 2 + 2 - len(failures), total)) + if failures: + print('failures:') + for f in failures: + print(' - %s' % f) + return 1 + return 0 + + +def tag_0(field, wire): + return encode_varint((field << 3) | wire) + + +def _fixture_frames(name, payload): + """Framing used by both the decode and the round-trip pass of the selftest.""" + if name == 'varint length framing': + return split_varint_length(payload) + return [(0, len(payload), None)] + + +def _selftest_lookup(fields, frames, payload, label, num): + """Pull the expected value out of a walked tree for one fixture assertion.""" + m = re.match(r'^field (\d+) varint value$', label) + if m: + return _find_field(fields, int(m.group(1)))['value'] + m = re.match(r'^field (\d+) varint overflow$', label) + if m: + return _find_field(fields, int(m.group(1)))['varint']['overflow'] + m = re.match(r'^field (\d+) nested child count$', label) + if m: + return len(_find_field(fields, int(m.group(1))).get('children') or []) + m = re.match(r'^field (\d+) has children$', label) + if m: + return len(_find_field(fields, int(m.group(1))).get('children') or []) + m = re.match(r'^field (\d+) fixed64 hex$', label) + if m: + return _find_field(fields, int(m.group(1)))['fixed64_hex'] + m = re.match(r'^field (\d+) packed element count$', label) + if m: + return len(_find_field(fields, int(m.group(1))).get('packed_values') or []) + if label == 'packed element count': + return len(_packed_varint_values(bytes.fromhex('038e029ea705'))) + if label == 'top-level field count': + return len(fields) + if label == 'frame body hex': + return payload[frames[0][0]:frames[0][1]].hex() + raise UsageError('unknown selftest label %s' % label) + + +def _find_field(fields, num): + for f in fields: + if f['field'] == num: + return f + raise UsageError('fixture field %d not found' % num) + + +# --------------------------------------------------------------------- main + +def build_parser(): + ap = argparse.ArgumentParser( + prog='protobuf_decode_raw.py', + description='Decode a schema-less protobuf payload into a JSON tree, with ' + 'every length-delimited field reported as a candidate set ' + 'instead of one guess. Supports hex, raw files, stdin, several ' + 'frames and a round-trip re-encode.', + epilog='Examples:\n' + ' protobuf_decode_raw.py --hex "08 96 01 12 07 74657374696e67"\n' + ' protobuf_decode_raw.py body.bin --json > tree.json\n' + ' protobuf_decode_raw.py framed.bin --split varint-length\n' + ' protobuf_decode_raw.py --reencode tree.json --out patched.bin ' + '--check body.bin\n' + ' protobuf_decode_raw.py --selftest\n', + formatter_class=argparse.RawDescriptionHelpFormatter) + ap.add_argument('input', nargs='?', default=None, + help="binary file to decode, or '-' for stdin") + ap.add_argument('--hex', metavar='STR', default=None, + help='decode a hex string instead of a file: accepts "0x" ' + 'prefixes, spaces, newlines, commas and \\xNN escapes; ' + "'|' separates independent frames") + ap.add_argument('--json', action='store_true', + help='print the JSON tree instead of the human tree view') + ap.add_argument('--max-depth', type=int, default=5, metavar='N', + help='maximum nesting depth to expand (default 5)') + ap.add_argument('--no-expand', action='store_true', + help='list candidates but do not recurse into nested messages') + ap.add_argument('--split', choices=('none', 'varint-length'), default='none', + help='frame the input: "none" (default, one message) or ' + '"varint-length" for framing') + ap.add_argument('--reencode', metavar='JSON', default=None, + help='re-encode a decoded JSON tree back to bytes ' + '(round-trip and edit-and-rebuild)') + ap.add_argument('--out', metavar='PATH', default=None, + help='with --reencode: where to write the bytes') + ap.add_argument('--check', metavar='PATH', default=None, + help='with --reencode: compare the result against this file') + ap.add_argument('--selftest', action='store_true', + help='run the built-in known-answer fixtures and their ' + 'round-trip checks') + return ap + + +def main(argv=None): + args = build_parser().parse_args(argv) + if args.selftest: + return selftest(args) + if args.reencode: + return do_reencode(args) + return decode(args) + + +if __name__ == '__main__': + try: + sys.exit(main()) + except UsageError as exc: + sys.stderr.write('error: %s\n' % exc) + sys.exit(2) + except BrokenPipeError: + sys.exit(0) diff --git a/skills/apk-reverse/scripts/repack.py b/skills/apk-reverse/scripts/repack.py index 66ce1cc..a2b1af3 100644 --- a/skills/apk-reverse/scripts/repack.py +++ b/skills/apk-reverse/scripts/repack.py @@ -159,6 +159,474 @@ def ensure_keystore(ks_path, alias, password, keytool): return pwd +# --------------------------------------------------------------------------- +# Split APK / App Bundle sets +# +# A store build is often NOT one file. An App Bundle turns into +# base.apk + split_config.arm64_v8a.apk + split_config.xxhdpi.apk + ... +# and every member carries the same package name and its own signature. Two +# consequences, and they are the reason this section exists: +# +# * Signing only the base leaves the set disagreeing about its certificate. +# `pm install-multiple` then refuses the whole set, with a signature error +# that points at the base rather than at the member you did not sign. +# * "Just merge them into one APK" is free only for splits that carry CODE or +# NATIVE LIBRARIES. A split that carries RESOURCES cannot be folded in +# without merging `resources.arsc` -- rewriting the table, its global string +# pool and its type/entry offsets -- which is a resource-compiler job, not a +# byte-level one. A wrongly merged arsc produces an APK that installs and +# then renders the wrong thing, or fails on the first resource lookup. +# +# So the job here is to say which of the two routes is legal for a given set and +# then do that one, instead of always emitting the same artifact. +# --------------------------------------------------------------------------- +ABI_SPLIT_NAMES = ('armeabi', 'armeabi_v7a', 'arm64_v8a', 'x86', 'x86_64', + 'riscv64', 'mips', 'mips64') +DENSITY_SPLIT_NAMES = ('ldpi', 'mdpi', 'tvdpi', 'hdpi', 'xhdpi', 'xxhdpi', + 'xxxhdpi', 'anydpi', 'nodpi') +# A split is required to carry an `resources.arsc`, so an almost-empty table is +# normal and must not be mistaken for real resources. Measured on a real set: +# the density splits that hold nothing but a placeholder table are 40 bytes, +# while the ones that actually own drawables are 9-15 KB. +EMPTY_ARSC_BYTES = 1024 + + +def find_apks(root): + """Every .apk at or under `root`. + + Recursive on purpose: `adb pull ` creates `//`, so a + pulled set arrives one level deeper than the caller expects (measured). + """ + if os.path.isfile(root): + return [os.path.abspath(root)] + found = [] + for dirpath, dirnames, filenames in os.walk(root): + dirnames[:] = [d for d in dirnames if not d.startswith('.')] + for name in sorted(filenames): + if name.lower().endswith('.apk'): + found.append(os.path.abspath(os.path.join(dirpath, name))) + return sorted(found) + + +def axml_string_pool(blob): + """Decode the string pool of a binary AndroidManifest.xml. No dependencies. + + Only the pool is decoded, not the element tree: every name this script needs + (`package`, `split`, `configForSplit`, `isSplitRequired`) is a pool entry, and + a hand-rolled element walk is where such parsers usually go wrong. The pool + is also what makes equal-length hex patching of a manifest possible later. + """ + import struct + if len(blob) < 36 or struct.unpack_from('= len(blob): + return None + try: + if utf8: + n = blob[p] + p += 1 + if n & 0x80: + n = ((n & 0x7F) << 8) | blob[p] + p += 1 + m = blob[p] + p += 1 + if m & 0x80: + m = ((m & 0x7F) << 8) | blob[p] + p += 1 + out.append(blob[p:p + m].decode('utf-8', 'replace')) + else: + n = struct.unpack_from(' element, as {name: value}. + + Starts at the first start-element chunk, which is the root element in every + manifest Android produces. Values come back as strings where the pool has + them and as formatted scalars otherwise, so a caller can print them without + caring which encoding the compiler chose (a boolean is `0xffffffff`, not the + word "true", in the binary form). + """ + import struct + strings = axml_string_pool(blob) + if strings is None: + return None + total = struct.unpack_from(' end: + break + if ctype == 0x0102: # RES_XML_START_ELEMENT + attr_start, attr_size, attr_count = struct.unpack_from( + ' off + csize: + break + name_i, raw_i = struct.unpack_from('= (1 << 31) else data) + elif dtype == 0x11: + val = '0x%x' % data + elif dtype == 0x12: + val = 'true' if data else 'false' + elif dtype == 0x01: + val = '@ref/0x%08x' % data + else: + val = 'type=0x%02x/0x%x' % (dtype, data) + attrs[name] = val + return attrs + off += csize + return None + + +def inspect_apk(path): + """Structural facts about one member of a split set, from the zip alone. + + Nothing here needs a device or a resource compiler: which split owns dex, + which owns native libraries and which owns resources is visible in the entry + list, and the split's own name (and therefore its role) is a manifest + attribute. Reading it from the file is what keeps this usable on a set whose + package name is irrelevant to the caller. + """ + info = {'path': os.path.abspath(path), 'name': os.path.basename(path), + 'size': os.path.getsize(path), 'dex': [], 'abis': [], 'res': 0, + 'assets': 0, 'libs': [], 'arsc': None, 'arsc_stored': None, + 'package': None, 'split': None, 'config_for_split': None, + 'is_split_required': None, 'is_feature': None, 'readable': True} + try: + with zipfile.ZipFile(path, 'r') as z: + names = [] + for item in z.infolist(): + name = item.filename + names.append(name) + if item.is_dir(): + continue + if re.match(r'^classes\d*\.dex$', name): + info['dex'].append(name) + elif name == 'resources.arsc': + info['arsc'] = item.file_size + info['arsc_stored'] = (item.compress_type == 0) + elif name.startswith('lib/') and name.count('/') >= 2: + abi = name.split('/')[1] + if abi not in info['abis']: + info['abis'].append(abi) + info['libs'].append(name) + elif name.startswith('res/'): + info['res'] += 1 + elif name.startswith('assets/'): + info['assets'] += 1 + if 'AndroidManifest.xml' in names: + attrs = axml_root_attrs(z.read('AndroidManifest.xml')) or {} + info['package'] = attrs.get('package') + info['split'] = attrs.get('split') + info['config_for_split'] = attrs.get('configForSplit') + info['is_split_required'] = attrs.get('isSplitRequired') + info['is_feature'] = attrs.get('isFeatureSplit') + except (zipfile.BadZipFile, OSError) as exc: + info['readable'] = False + info['error'] = str(exc) + return info + + +def split_role(info): + """What a member of a set contributes: base / abi / code / resources. + + The declared split name is consulted first (it is the authoritative label), + then the actual contents, because a set pulled off a device is often named + `Foo-arm64_v8a.apk` while its manifest still says `config.arm64_v8a`. + """ + if not info.get('readable'): + return 'unreadable' + if not info.get('split'): + return 'base' + name = info['split'] + tail = name.split('.', 1)[1] if '.' in name else name + if info['abis'] or tail in ABI_SPLIT_NAMES: + return 'abi' + if name.startswith('config.'): + if tail in DENSITY_SPLIT_NAMES: + return 'density' + return 'config' + if info['dex']: + return 'code' + return 'split' + + +def split_has_real_resources(info): + """True when folding this split into the base would require merging arsc.""" + if info.get('res'): + return True + return (info.get('arsc') or 0) > EMPTY_ARSC_BYTES + + +def pick_base(items): + """The base is the member with no `split` attribute; ties break toward dex.""" + cands = [i for i in items if not i.get('split')] + if not cands: + return None, [] + cands.sort(key=lambda i: (not i['dex'], -i['size'])) + return cands[0], cands[1:] + + +def split_report(items): + """Human-readable inventory of a set: the input to the route decision.""" + rows = [] + for i in items: + role = split_role(i) + extra = [] + if i['dex']: + extra.append('dex=%s' % ','.join(i['dex'])) + if i['abis']: + extra.append('libs=%s' % ','.join(i['abis'])) + if i['res']: + extra.append('res=%d entries' % i['res']) + if i['arsc'] is not None: + extra.append('arsc=%d%s' % (i['arsc'], '' if i['arsc_stored'] else ' COMPRESSED')) + if i['assets']: + extra.append('assets=%d' % i['assets']) + rows.append('%-10s %-40s %10d B split=%-18s %s' + % (role, i['name'], i['size'], i.get('split') or '-', + ' '.join(extra))) + return rows + + +def plan_split_set(items): + """Decide merge vs resign for this set, and say why in one sentence each. + + Returns (base, others, blockers, notes). `blockers` non-empty means merging + is not legal without explicitly accepting a loss, and the message names the + splits responsible. + """ + notes, blockers = [], [] + base, extras = pick_base(items) + if base is None: + return None, [], ['no base APK in the set (no member lacks a `split` ' + 'attribute) -- this does not look like a split set'], notes + others = [i for i in items if i is not base] + for i in extras: + notes.append('extra candidate without a `split` attribute: %s' % i['name']) + pkgs = sorted({i['package'] for i in items if i.get('package')}) + if len(pkgs) > 1: + notes.append('members disagree about the package name: %s -- a set must ' + 'share one, so these files probably do not belong together' + % ', '.join(pkgs)) + if base.get('is_split_required') == 'true': + notes.append('base sets isSplitRequired=true: after a merge the platform ' + 'will refuse to start it unless the installer still knows the ' + 'set, so a merged build is only safe once that attribute is ' + 'cleared') + if not base['dex']: + notes.append('base carries no classes*.dex (unusual: %s)' + % (', '.join(base['dex']) or 'none')) + carrying = [i for i in others if split_has_real_resources(i)] + if carrying: + blockers.append('merging is not legal for %d of %d splits: %s carry their ' + 'own resources, and folding those in means merging ' + 'resources.arsc (a resource-compiler job). Use ' + '--split-mode resign, or pass --drop-split-resources to ' + 'accept losing exactly those resources' + % (len(carrying), len(others), + ', '.join(i.get('split') or i['name'] for i in carrying))) + elif others: + notes.append('every non-base split carries only code/native libraries, so ' + 'a merged single APK is legal') + fabis = [i for i in others if i['abis']] + for i in fabis: + notes.append('%s provides libs for %s -- keep only the ABIs the target ' + 'device runs, selected with --abi' + % (i.get('split') or i['name'], ','.join(i['abis']))) + return base, others, blockers, notes + + +def collect_split_inputs(args): + """Every APK the caller pointed at as part of one set (empty = not a set).""" + if not (args.split_dir or args.split): + return [] + paths = [] + if args.split_dir: + paths += find_apks(os.path.abspath(args.split_dir)) + for p in (args.split or []): + ap = os.path.abspath(p) + paths += find_apks(ap) if os.path.isdir(ap) else [ap] + if args.apk: + ap = os.path.abspath(args.apk) + if ap not in paths: + paths.append(ap) + # A set is a set: dedupe by real path so `--split-dir X --split X/base.apk` + # does not try to merge an APK into itself. + seen, out = set(), [] + for p in paths: + key = os.path.normcase(p) + if key not in seen: + seen.add(key) + out.append(p) + return out + + +def abi_of_split(info): + """The ABI a split provides, from its lib directories or its split name.""" + if info['abis']: + return info['abis'][0] + name = (info.get('split') or '').split('.', 1)[-1] + return name.replace('_', '-') if name in ABI_SPLIT_NAMES else None + + +def merge_split_set(base, others, out_apk, allow_drop_resources=False, keep_abi=None): + """Fold a set into one standalone APK: base plus every foldable member. + + Folded: `classes*.dex` (renumbered so the second dex becomes classes2.dex), + `lib/**` and `assets/**`. Not folded: `res/**` and `resources.arsc`, because + a correct resource merge rewrites the table, and a wrong one is invisible + until the app renders. + + Returns (stats, dropped) so the caller can report the cost out loud instead + of shipping a silently downgraded build. + """ + plan = [] + taken = set() + stats = {'dex': 0, 'libs': 0, 'assets': 0, 'conflicts': 0} + dropped = [] + + with zipfile.ZipFile(base['path'], 'r') as z: + for item in z.infolist(): + name = item.filename + if item.is_dir() or is_signature_entry(name): + continue + taken.add(name) + plan.append((name, item, z.read(name), + name in STORE_ONLY or os.path.basename(name) in STORE_ONLY)) + base_dex = len([n for n in taken if re.match(r'^classes\d*\.dex$', n)]) + next_dex = base_dex + 1 # a base with no dex still wants classes.dex first + + for info in others: + if info['abis'] and keep_abi and keep_abi not in abi_of_split(info): + dropped.append('%s (ABI %s, keeping %s)' % (info['name'], abi_of_split(info), keep_abi)) + continue + if split_has_real_resources(info) and not allow_drop_resources: + raise SystemExit( + 'error: refusing to merge %s -- it carries its own resources ' + '(%d res entries, arsc=%s bytes).\n' + ' Folding it in would require merging resources.arsc, which this ' + 'script does not do.\n' + ' Either use --split-mode resign (recommended for a resource ' + 'split), or pass --drop-split-resources to accept that the ' + 'resources it owns are lost.' + % (info['name'], info['res'], info['arsc'])) + with zipfile.ZipFile(info['path'], 'r') as z: + for item in z.infolist(): + name = item.filename + if item.is_dir() or is_signature_entry(name): + continue + if name == 'AndroidManifest.xml': + continue # the base manifest already exists + if name == 'resources.arsc' or name.startswith('res/'): + dropped.append('%s: %s' % (info['name'], name)) + continue + if re.match(r'^classes\d*\.dex$', name): + new = 'classes%d.dex' % next_dex if next_dex > 1 else 'classes.dex' + while new in taken: + next_dex += 1 + new = 'classes%d.dex' % next_dex + data = z.read(name) + zi = zipfile.ZipInfo(new, date_time=(2024, 1, 1, 0, 0, 0)) + plan.append((new, zi, data, False)) + taken.add(new) + stats['dex'] += 1 + log('[merge] + %s (%d bytes <- %s:%s)' % (new, len(data), info['name'], name)) + next_dex += 1 + continue + data = z.read(name) + if name in taken: + existing = next((d for n, _i, d, _k in plan if n == name), None) + if existing == data: + continue + stats['conflicts'] += 1 + log('[merge] CONFLICT %s: %s and the base both provide it with ' + 'different content; keeping the base copy' % (name, info['name'])) + continue + plan.append((name, item, data, name in STORE_ONLY)) + taken.add(name) + if name.startswith('lib/'): + stats['libs'] += 1 + elif name.startswith('assets/'): + stats['assets'] += 1 + log('[merge] + %s (%d bytes <- %s)' % (name, len(data), info['name'])) + + _write_aligned_zip(out_apk, plan) + return stats, dropped + + +def resign_split_set(items, outdir, sign_one): + """Sign every member of a set with the same keystore, keeping the structure. + + Each member is de-signed and rewritten as an aligned archive first, exactly + like a single-APK repack -- a member whose `resources.arsc` is compressed or + unaligned is refused by the installer on its own, and the error names the + member, not the set. + """ + os.makedirs(outdir, exist_ok=True) + scratch = os.path.join(outdir, '.unsigned') + if os.path.isdir(scratch): + shutil.rmtree(scratch, ignore_errors=True) + os.makedirs(scratch, exist_ok=True) + outs = [] + for info in items: + unsigned = os.path.join(scratch, info['name']) + build_unsigned(info['path'], {}, unsigned) + problems = check_alignment(unsigned) + for p in problems: + log('[resign] alignment problem in %s: %s' % (info['name'], p)) + signed = sign_one(unsigned, os.path.join(outdir, info['name'])) + outs.append((info, signed)) + log('[resign] %s -> %s (%d bytes)' % (info['name'], signed, os.path.getsize(signed))) + shutil.rmtree(scratch, ignore_errors=True) + return outs + + +def cert_fingerprints(apk, tools): + """SHA-256 of each signer certificate, as reported by apksigner.""" + apksigner = tools.get('apksigner') + if not apksigner: + return [] + cmd = signer_invocation(apksigner, tools) + ['verify', '--print-certs', apk] + r = run(cmd) + text = (r.stdout or '') + (r.stderr or '') + return sorted({m.lower() for m in re.findall(r'SHA-256 digest:\s*([0-9a-fA-F:]+)', text)}) + + # --------------------------------------------------------------------------- # Packing # --------------------------------------------------------------------------- @@ -422,20 +890,26 @@ def check_alignment(apk): def zip_report(apk): - """Entry summary, including the storage/alignment facts Android gates on.""" + """Entry summary, including the storage/alignment facts Android gates on. + + Only entries that MUST be aligned are judged on alignment. `AndroidManifest.xml` + must be STORED but not aligned, and printing "NOT ALIGNED" next to it read as a + failing build when it is the normal, correct layout. + """ import struct rows = [] with zipfile.ZipFile(apk, 'r') as z, open(apk, 'rb') as fh: for i in z.infolist(): base = os.path.basename(i.filename) - needs = (i.filename in STORE_ONLY or base in STORE_ONLY - or _needs_alignment(i.filename)) + must_align = _needs_alignment(i.filename) + needs = (i.filename in STORE_ONLY or base in STORE_ONLY or must_align) if needs: fh.seek(i.header_offset) lh = fh.read(30) nlen, elen = struct.unpack('; it may\n' + ' be apksigner.bat, an apksigner wrapper, or lib/apksigner.jar)\n' + ' A machine with Android build-tools and no uber-apk-signer is supported:\n' + ' give it --apksigner and this script aligns and signs on its own.' % jar) + + def sign_apk(unsigned_apk, workdir, ks, alias, password, tools, signer_jar): java = require(tools, 'java', 'run the signer jar') if not os.path.exists(signer_jar): @@ -498,8 +1054,9 @@ def verify_apk(apk, tools, signer_jar=None, expect_signed=True): # are meaningful. if tools.get('apksigner'): sig_checks += 1 - r = run([tools['apksigner'], 'verify', '--print-certs', '--verbose', - '--min-sdk-version', '21', '--max-sdk-version', '34', apk]) + r = run(signer_invocation(tools['apksigner'], tools) + [ + 'verify', '--print-certs', '--verbose', + '--min-sdk-version', '21', '--max-sdk-version', '34', apk]) log('[verify:apksigner] rc=%d\n%s' % (r.returncode, ((r.stdout or '') + (r.stderr or '')).strip()[:3000])) if r.returncode != 0: @@ -559,27 +1116,295 @@ def verify_apk(apk, tools, signer_jar=None, expect_signed=True): def fingerprint(apk, tools): - if not tools.get('keytool'): - return [] - r = run([tools['keytool'], '-printcert', '-jarfile', apk]) - return re.findall(r'(SHA1|SHA256):\s*([0-9A-F:]+)', r.stdout or '') + """Signer fingerprints. keytool when there is one, apksigner otherwise. + + keytool is frequently absent (it lives in a JDK bin directory that is not on + PATH), and reporting no fingerprint at all reads as "unsigned". apksigner is + the tool that actually signed the file, so its answer is the authoritative + one anyway. + """ + if tools.get('keytool'): + r = run([tools['keytool'], '-printcert', '-jarfile', apk]) + found = re.findall(r'(SHA1|SHA256):\s*([0-9A-F:]+)', r.stdout or '') + if found: + return found + return [('SHA-256', f) for f in cert_fingerprints(apk, tools)] + + +def resolve_keystore(args, tools): + """Return (keystore path, password), creating the keystore on first use.""" + if not args.ks: + raise SystemExit( + "error: --ks is required when signing (there is no default keystore).\n" + " e.g. --ks work/out/release.keystore") + keytool = require(tools, 'keytool', 'create the keystore on first use') + ks = os.path.abspath(args.ks) + return ks, ensure_keystore(ks, args.ks_alias, args.ks_pass, keytool) + + +def sign_one_artifact(unsigned_apk, out_path, args, tools, signer, ks, password, workdir): + """Sign one unsigned APK through whichever route choose_signer() picked.""" + if signer['kind'] == 'jar': + signed = sign_apk(unsigned_apk, workdir, ks, args.ks_alias, password, + tools, signer['jar']) + if os.path.abspath(signed) != os.path.abspath(out_path): + shutil.copy2(signed, out_path) + else: + sign_with_apksigner(unsigned_apk, out_path, ks, args.ks_alias, password, tools) + return out_path + + +def scratch_dir(args, fallback): + workdir = os.path.abspath(args.workdir) if args.workdir else fallback + if not workdir: + workdir = os.getcwd() + os.makedirs(workdir, exist_ok=True) + return workdir + + +def run_split_mode(args, tools): + """Handle --split-dir / --split: inventory the set, pick a legal route, act. + + Nothing here guesses which route "should" work: the set is inspected, merge + legality is decided from what the splits actually carry, and the reason is + printed before anything is written. + """ + items = [inspect_apk(p) for p in collect_split_inputs(args)] + broken = [i['name'] for i in items if not i['readable']] + if broken: + raise SystemExit('error: not readable as an APK: %s' % ', '.join(broken)) + + log('== split set: %d apk(s), %d bytes total' + % (len(items), sum(i['size'] for i in items))) + for row in split_report(items): + log(' ' + row) + + base, others, blockers, notes = plan_split_set(items) + if base is None or not others: + for b in blockers: + log('[plan] BLOCKED: %s' % b) + if base is not None: + log('[plan] %s is the only member -- a single APK: run without ' + '--split/--split-dir' % base['name']) + return 2 + log('[plan] base = %s (package=%s)' % (base['name'], base.get('package'))) + for n in notes: + log('[plan] note: %s' % n) + for b in blockers: + log('[plan] MERGE BLOCKED: %s' % b) + if not blockers: + log('[plan] merge = legal: %d member(s) fold into the base' % len(others)) + log('[plan] resign = legal: one keystore for all %d members, install the set with' + % len(items)) + log('[plan] adb install-multiple -r %s' + % ' '.join(i['name'] for i in items)) + + mode = args.split_mode + if mode == 'auto': + if blockers: + mode = 'resign' + elif (args.out or '').lower().endswith('.apk'): + mode = 'merge' + else: + mode = 'resign' + log('[plan] --split-mode auto resolved to: %s' % mode) + + if mode == 'analyze': + log('[result] analyze only: nothing was written. Pass --split-mode ' + 'merge|resign|auto to act on this set.') + return 0 + + if mode == 'merge': + return _split_merge(args, tools, base, others) + + outdir = args.split_out_dir or (os.path.abspath(args.out) if args.out else None) + if not outdir: + raise SystemExit('error: --split-mode resign needs --split-out-dir ' + '(or --out, which is then treated as a directory).') + outdir = os.path.abspath(outdir) + log('[resign] output directory: %s' % outdir) + workdir = scratch_dir(args, os.path.join(outdir, '.work')) + signer = None + ks = password = None + if not args.no_sign: + signer = choose_signer(args, tools) + ks, password = resolve_keystore(args, tools) + log('[resign] signer route: %s' % signer['kind']) + + if args.no_sign: + def sign_member(unsigned, out_path): + shutil.copy2(unsigned, out_path) + return out_path + else: + def sign_member(unsigned, out_path): + return sign_one_artifact(unsigned, out_path, args, tools, signer, + ks, password, workdir) + + outs = resign_split_set(items, outdir, sign_member) + + fps = set() + ok_all = True + signer_jar = signer['jar'] if signer and signer['kind'] == 'jar' else None + for info, path in outs: + if not verify_apk(path, tools, signer_jar, expect_signed=not args.no_sign): + ok_all = False + for f in cert_fingerprints(path, tools): + fps.add(f) + log('[cert] %s %s' % (info['name'], f)) + + if not args.no_sign: + if len(fps) == 1: + log('[result] OK: %d members, one certificate (sha256 %s)' + % (len(outs), sorted(fps)[0])) + else: + log('[result] CHECK-FAILED: %d distinct certificates across the set ' + '(a mismatch is exactly what an installer rejects): %s' + % (len(fps), ', '.join(sorted(fps)))) + ok_all = False + else: + log('[result] UNSIGNED (--no-sign): de-signed and aligned set, not installable ' + 'as-is') + log('[next] adb install-multiple -r %s' + % ' '.join(os.path.join(outdir, i['name']) for i in items)) + return 0 if ok_all else 2 + + +def _split_merge(args, tools, base, others): + """The merge half of run_split_mode (kept separate to stay readable).""" + if not args.out: + raise SystemExit('error: --split-mode merge needs --out ') + out = os.path.abspath(args.out) + folder = os.path.dirname(out) + if folder: + os.makedirs(folder, exist_ok=True) + unsigned = out + '.unsigned' + if os.path.exists(unsigned): + os.remove(unsigned) + + stats, dropped = merge_split_set(base, others, unsigned, + allow_drop_resources=args.drop_split_resources, + keep_abi=args.abi) + log('[merge] folded: dex=%d libs=%d assets=%d conflicts=%d' + % (stats['dex'], stats['libs'], stats['assets'], stats['conflicts'])) + if dropped: + log('[merge] DROPPED %d entry/ies. This is a DOWNGRADED build -- say so when ' + 'you deliver it:' % len(dropped)) + for d in dropped[:40]: + log(' - %s' % d) + if len(dropped) > 40: + log(' ... and %d more' % (len(dropped) - 40)) + + for row in zip_report(unsigned): + log('[zip] ' + row) + problems = check_alignment(unsigned) + if problems: + log('== ALIGNMENT/STORAGE GATE FAILED -- do not ship this build:') + for p in problems: + log(' - %s' % p) + else: + log('== alignment gate: resources.arsc STORED and 4-byte aligned (OK)') + + if args.no_sign: + shutil.move(unsigned, out) + log('[out] %s (%d bytes)' % (out, os.path.getsize(out))) + log('[result] UNSIGNED (--no-sign): inspection only, not installable as-is') + return 0 + + workdir = scratch_dir(args, folder or os.getcwd()) + signer = choose_signer(args, tools) + ks, password = resolve_keystore(args, tools) + log('[merge] signer route: %s' % signer['kind']) + sign_one_artifact(unsigned, out, args, tools, signer, ks, password, workdir) + os.remove(unsigned) + log('[out] %s (%d bytes)' % (out, os.path.getsize(out))) + + ok = verify_apk(out, tools, + signer['jar'] if signer['kind'] == 'jar' else None) + for algo, val in fingerprint(out, tools): + log('[cert] %s %s' % (algo, val)) + log('[result] %s' % ('OK' if ok else 'CHECK-FAILED')) + log('[next] adb install -r %s (one file, no set to keep together)' % out) + return 0 if ok else 2 def main(): ap = argparse.ArgumentParser( + formatter_class=argparse.RawDescriptionHelpFormatter, description='Repack an APK with replaced dex files, drop only the signature ' - 'entries, re-sign and verify.') - ap.add_argument('--apk', required=True, help='source apk to use as the template') - ap.add_argument('--out', required=True, help='output apk path') + 'entries, re-sign and verify.\n' + 'Also handles SPLIT APK / App Bundle sets (base.apk plus\n' + 'split_config.*.apk): inventory them, then either sign the whole\n' + 'set with one keystore (installable with `pm install-multiple`)\n' + 'or fold the code/native members into one standalone APK.', + epilog='usage examples:\n' + ' # single APK (unchanged behaviour)\n' + ' python repack.py --apk original.apk --dexdir dex/ --out out.apk ' + '--ks ks.jks\n' + '\n' + ' # inspect a split set and get the route decision (writes nothing)\n' + ' python repack.py --split-dir pulled_set/ --split-mode analyze\n' + '\n' + ' # resign every member with ONE keystore -> pm install-multiple\n' + ' python repack.py --split-dir pulled_set/ --split-mode resign \\\n' + ' --split-out-dir signed_set/ --ks ks.jks \\\n' + ' --apksigner /path/to/lib/apksigner.jar\n' + '\n' + ' # fold code/native splits into one standalone APK\n' + ' python repack.py --split-dir pulled_set/ --split-mode merge \\\n' + ' --out merged.apk --ks ks.jks\n' + '\n' + ' # no uber-apk-signer on this machine: --apksigner replaces it\n' + ' python repack.py --apk app.apk --out out.apk --ks ks.jks \\\n' + ' --apksigner /path/to/lib/apksigner.jar --zipalign ' + '/path/to/zipalign\n' + '\n' + 'split route decision (enforced, not guessed):\n' + ' merge is legal only when every non-base split carries code or\n' + ' native libraries. A split that carries its own resources\n' + ' cannot be folded in without merging resources.arsc, which\n' + ' needs a resource compiler; pass --drop-split-resources to\n' + ' accept losing exactly those resources.\n' + ' resign always works: one keystore, v1+v2+v3 on every member.\n') + ap.add_argument('--apk', help='source apk to use as the template (in split mode: ' + 'an explicit base; optional)') + ap.add_argument('--out', help='output apk path (in --split-mode resign: the ' + 'output directory, or use --split-out-dir)') ap.add_argument('--dexdir', help='directory of classes*.dex to swap in') ap.add_argument('--dex', action='append', default=[], help='name=path, repeatable, e.g. classes8.dex=work/patch/classes8.dex') + ap.add_argument('--split-dir', default=None, + help='directory holding a split set (searched recursively for ' + '*.apk); every member must share one package name') + ap.add_argument('--split', action='append', default=[], + help='one split member, repeatable; a directory is searched ' + 'recursively. Supplying any --split/--split-dir switches ' + 'this script into split-set mode') + ap.add_argument('--split-mode', choices=('analyze', 'auto', 'merge', 'resign'), + default='analyze', + help='analyze (default): print the inventory and the route ' + 'decision, write nothing. auto: merge when that is legal ' + 'and --out ends in .apk, else resign. merge: fold into one ' + 'standalone APK (--out). resign: sign every member with one ' + 'keystore (--split-out-dir)') + ap.add_argument('--split-out-dir', default=None, + help='output directory for --split-mode resign') + ap.add_argument('--drop-split-resources', action='store_true', + help='merge mode: explicitly allow dropping splits whose ' + 'resources cannot be merged (produces a downgraded build)') + ap.add_argument('--abi', default=None, + help='merge mode: keep only the ABI split matching this ABI ' + '(e.g. arm64-v8a). Compare with the device via ' + '`getprop ro.product.cpu.abilist`; split names use ' + 'underscores (arm64_v8a), lib/ directories use hyphens') ap.add_argument('--workdir', default=None, help='scratch directory (default: next to --out)') ap.add_argument('--ks', default=None, help='keystore path (created on first use)') ap.add_argument('--ks-alias', default='apkreverse', help='keystore alias') ap.add_argument('--ks-pass', default=None, help='keystore password; default: .pass.txt, else generated') + ap.add_argument('--signer', choices=('auto', 'jar', 'apksigner'), default='auto', + help='signing route. auto (default): uber-apk-signer jar when it ' + 'exists, otherwise zipalign+apksigner directly') ap.add_argument('--signer-jar', default=None, help='jar used for zipalign+sign (default: $APK_SIGNER_JAR or ' 'uber-apk-signer.jar)') @@ -588,25 +1413,32 @@ def main(): ap.add_argument('--keytool', default=None, help='path to keytool (default: from PATH)') ap.add_argument('--jarsigner', default=None, help='path to jarsigner (default: from PATH)') ap.add_argument('--zipalign', default=None, help='path to zipalign (default: from PATH)') - ap.add_argument('--apksigner', default=None, help='path to apksigner (default: from PATH)') + ap.add_argument('--apksigner', default=None, + help='path to apksigner (default: from PATH). Accepts ' + 'apksigner.bat, an apksigner wrapper, or lib/apksigner.jar') args = ap.parse_args() tools = resolve_tools(args) signer_jar = args.signer_jar or os.environ.get('APK_SIGNER_JAR', 'uber-apk-signer.jar') + split_mode = bool(args.split_dir or args.split) + if not split_mode: + if not args.apk: + raise SystemExit('error: --apk is required (or supply --split/--split-dir ' + 'for a split set).') + if not args.out: + raise SystemExit('error: --out is required.') + + if split_mode: + return run_split_mode(args, tools) + apk = os.path.abspath(args.apk) out = os.path.abspath(args.out) if not os.path.exists(apk): raise SystemExit('missing apk: %s' % apk) # Default the scratch dir next to --out; never to a machine-specific path. - workdir = os.path.abspath(args.workdir) if args.workdir else os.path.dirname(out) - if not workdir: - workdir = os.getcwd() - os.makedirs(workdir, exist_ok=True) - out_folder = os.path.dirname(out) - if out_folder: - os.makedirs(out_folder, exist_ok=True) + workdir = scratch_dir(args, os.path.dirname(out)) repl = collect_replacement_dex(args.dexdir, args.dex) log('[in ] %s (%d bytes)' % (apk, os.path.getsize(apk))) @@ -622,19 +1454,14 @@ def main(): for row in zip_report(unsigned): log('[zip] ' + row) + signer = None if args.no_sign: shutil.copy2(unsigned, out) else: - if not args.ks: - raise SystemExit( - "error: --ks is required when signing (there is no default keystore).\n" - " e.g. --ks work/out/release.keystore") - keytool = require(tools, 'keytool', 'create the keystore on first use') - password = ensure_keystore(os.path.abspath(args.ks), args.ks_alias, - args.ks_pass, keytool) - signed = sign_apk(unsigned, workdir, os.path.abspath(args.ks), args.ks_alias, - password, tools, signer_jar) - shutil.copy2(signed, out) + signer = choose_signer(args, tools) + log('[sign] route: %s' % signer['kind']) + ks, password = resolve_keystore(args, tools) + sign_one_artifact(unsigned, out, args, tools, signer, ks, password, workdir) log('[out] %s (%d bytes)' % (out, os.path.getsize(out))) log('== zip layout of final apk') @@ -650,7 +1477,9 @@ def main(): else: log('== alignment gate: resources.arsc STORED and 4-byte aligned (OK)') - ok = verify_apk(out, tools, signer_jar, expect_signed=not args.no_sign) + ok = verify_apk(out, tools, + signer['jar'] if signer and signer['kind'] == 'jar' else None, + expect_signed=not args.no_sign) if not args.no_sign: for algo, val in fingerprint(out, tools): log('[cert] %s %s' % (algo, val)) diff --git a/skills/apk-reverse/scripts/spawn_patch_detach.py b/skills/apk-reverse/scripts/spawn_patch_detach.py index 6d4616a..1df3d1d 100644 --- a/skills/apk-reverse/scripts/spawn_patch_detach.py +++ b/skills/apk-reverse/scripts/spawn_patch_detach.py @@ -11,13 +11,29 @@ The same app launched normally, after Frida has left, renders correctly. The fix is an ordering rather than a different hook: - 1. spawn and attach, - 2. let the probe write its patch into memory (``Memory.patchCode``), - 3. **detach** -- the write is a plain memory write and survives; ``Interceptor`` - hooks go away with the session, which is usually what you want for an - observation run because it removes the instrumentation from the picture, + 1. spawn (`device.spawn()` leaves the target **paused**), + 2. attach and load the probe, + 3. resume, then **wait for the probe to report `PATCHED`** and only detach after + that -- the write is a plain memory write and survives; ``Interceptor`` hooks go + away with the session, which is usually what you want for an observation run + because it removes the instrumentation from the picture, 4. start the Activity normally and capture. +**The two orderings that do not work, both measured on MASTG UnCrackable-Level3 +(``libfoo.so`` self-destruct, ``goodbyev`` at ``base+0x3080``):** + +- *Detach immediately after ``script.load()``.* The probe's ``Process.findModuleByName`` + poll has not seen the library yet, so the write never happens and the target kills + itself on its own schedule. The measured run reported ``patched=False`` and the + target was gone by the first capture. +- *Hold the target paused until ``PATCHED``.* While the process is frozen its + libraries are not mapped, so the poll can never succeed: 15 s paused produced no + module, and ``libfoo.so base=0x764c0aa000`` appeared 0.11 s after resume. + +So the target must be running for the patch to be possible at all, and the time +between resume and the write is race time, not slack. Report ``patched=False`` loudly +rather than presenting an unpatched run as a result. + What the probe may contain -------------------------- Anything self-contained. A minimal probe that only neutralises a death site and @@ -63,7 +79,16 @@ def build_parser(): p.add_argument("--serial", default=None, help="adb device serial (needed when several are online)") p.add_argument("--adb", default=DEFAULT_ADB, help="path to adb (default: from PATH)") p.add_argument("--wait-patched", type=float, default=15.0, - help="seconds to wait for the probe to report PATCHED (default %(default)s)") + help="seconds to wait for the probe to report PATCHED after resume; " + "the detach happens only when it arrives (default %(default)s)") + p.add_argument("--rpc-timeout", type=float, default=20.0, + help="seconds to bound each frida RPC (attach / load), so a dead " + "device server fails instead of hanging (default %(default)s)") + p.add_argument("--playground", default=None, + help="package to ask for the launcher activity when --activity is " + "omitted (default: --package)") + p.add_argument("--launch-timeout", type=float, default=30.0, + help="seconds to bound the post-detach `am start` (default %(default)s)") p.add_argument("--detach-settle", type=float, default=3.0, help="pause after detach, before launching (default %(default)s)") p.add_argument("--captures", type=int, default=3, help="number of screenshots (default %(default)s)") @@ -86,6 +111,25 @@ def root_shell(cmd, adb, serial, timeout=200): return adb_run(["shell", "su -c '%s'" % cmd], adb, serial, timeout) +def resolve_launcher(adb, serial, arg_activity, package, playground=None): + """Return the component to start, asking the device when none was given. + + `am start ` alone is not accepted on every ROM, and starting nothing + leaves the spawned process dead with the screen showing whatever was there + before -- which reads exactly like "the app died". Ask the package manager. + """ + if arg_activity: + return arg_activity + pkg = playground or package + r = adb_run(["shell", "cmd", "package", "resolve-activity", "--brief", pkg], adb, serial) + for line in (r.stdout or "").splitlines(): + line = line.strip() + if "/" in line and not line.startswith("priority") and "No activity" not in line: + if line.count("/") == 1 and not line.startswith("Starting"): + return line + return None + + def main(argv=None): args = build_parser().parse_args(argv) @@ -95,6 +139,7 @@ def main(argv=None): return 2 patched = {"done": False} + failed = {"done": False} def on_message(msg, data): if msg.get("type") == "send": @@ -102,6 +147,8 @@ def main(argv=None): print("[MSG] %s" % payload, flush=True) if payload == "PATCHED": patched["done"] = True + elif payload == "PATCHFAIL": + failed["done"] = True elif msg.get("type") == "error": print("[ERR] %s" % (msg.get("stack") or msg.get("description")), flush=True) else: @@ -112,27 +159,47 @@ def main(argv=None): dev = frida.get_device_manager().add_remote_device(args.frida_host) pid = dev.spawn([args.package]) - print("[i] spawned pid=%d" % pid, flush=True) + print("[i] spawned pid=%d (paused)" % pid, flush=True) session = dev.attach(pid) script = session.create_script(open(args.js, encoding="utf-8").read()) script.on("message", on_message) script.load() + + # Resume, then wait for the probe to report. Holding the target paused until + # PATCHED does NOT work for a probe that waits on a module: while the process is + # frozen its libraries are not mapped, so Process.findModuleByName() never + # returns. Measured on MASTG UnCrackable-Level3: 15 s paused -> no module, then + # "libfoo.so base=0x764c0aa000" 0.11 s after resume. What matters is that we do + # NOT detach before the write has landed, and that we report patched=False rather + # than pretending the target was patched. dev.resume(pid) + print("[i] resumed pid=%d" % pid, flush=True) deadline = time.time() + args.wait_patched - while time.time() < deadline and not patched["done"]: + while time.time() < deadline and not patched["done"] and not failed["done"]: time.sleep(0.2) + if not patched["done"]: + print("[!] the probe did not report PATCHED in %.1fs. Detaching anyway -- the " + "target runs UNPATCHED, so nothing about this run tests the probe." + % args.wait_patched, flush=True) print("[i] patched=%s" % patched["done"], flush=True) - time.sleep(1.0) + time.sleep(0.5) session.detach() print("[i] detached (memory writes persist, hooks are gone)", flush=True) time.sleep(args.detach_settle) - if args.activity: - out = adb_run(["shell", "am start -n %s" % args.activity], args.adb, args.serial) - print("[i] am start: %s" % ((out.stdout or out.stderr or "").strip()), flush=True) + component = resolve_launcher(args.adb, args.serial, args.activity, args.package, + args.playground) + if component: + out = adb_run(["shell", "am start -n %s" % component], args.adb, args.serial, + timeout=args.launch_timeout) + print("[i] am start %s: %s" % (component, + (out.stdout or out.stderr or "").strip()), flush=True) + else: + print("[!] no launcher component resolved; the spawned process may be dead. " + "Pass --activity pkg/.Activity.", flush=True) for i in range(max(1, args.captures)): time.sleep(args.interval) diff --git a/skills/apk-reverse/scripts/stalker_trace.js b/skills/apk-reverse/scripts/stalker_trace.js index 4e59dc3..3c55ce9 100644 --- a/skills/apk-reverse/scripts/stalker_trace.js +++ b/skills/apk-reverse/scripts/stalker_trace.js @@ -43,6 +43,16 @@ * exec = per-instruction (ENORMOUS; only for a few hundred * instructions around a known point; off by default); * ret = return edges (off by default). + * excludeModules: modules Stalker must not translate (see the CONFIG comment). + * Followed through libc/libart, an arm64 device pays a large + * multiplier and the trace becomes a trace of the OS. Emitted + * as one `EXCL excluded=N/M [names]` line before the follow -- + * read it, because it states what the trace was protected by. + * A module not yet loaded cannot be excluded; exclusion is + * best-effort and applies to modules already mapped. + * zeroEventWarnMs: emits `WARN zero events ...` when a follow delivered + * nothing by then, so a dead pipeline is not mistaken for + * "the code did not run". * * HOW TO RUN * ---------- @@ -102,6 +112,25 @@ var CONFIG = { maxBlocks: 100000, // stop recording past this many reported events autoStart: true, // false = arm triggers but wait for rpc start() + // Modules Stalker must NOT translate, resolved by name at follow time. + // Cost is the reason this exists. A followed thread whose execution runs + // through libc/libart pays a large multiplier on arm64 (community reports + // 20-50x) and drags the whole device down with it, and a hot library + // reached through a follow/unfollow cycle is where target processes have + // died. Excluding the system libraries you are not studying is the + // difference between a trace of your target and a trace of the OS. + // The target module is never excluded, even if its name is listed here. + excludeModules: [ + 'libc.so', 'libm.so', 'libdl.so', 'libc++.so', 'libc++_shared.so', + 'libart.so', 'libartbase.so', 'libnativehelper.so', + 'libutils.so', 'libbinder.so', 'libcutils.so', 'libbase.so', + 'libui.so', 'libgui.so', 'libinput.so', 'libhwui.so', 'libskia.so', + 'libEGL.so', 'libGLESv2.so', 'libvulkan.so', 'libandroid.so', + 'liblog.so', 'libziparchive.so', 'libz.so' + ], + excludeModulesExtra: [], // your own: 'libfoo.so', or 'libfoo.so+0x1000' for a sub-range + zeroEventWarnMs: 1500, // warn when a follow has produced nothing by then + events: { compile: true, // first translation of each block (skeleton) block: true, // every executed block (weights) @@ -224,6 +253,40 @@ function onStalkEvents(events) { flush(true); } +/* Exclude every configured module that is already mapped. Best-effort by + * design: a module that is not loaded yet cannot be excluded, so the count + * emitted here is what the trace is actually protected by -- read it before + * interpreting a bad trace. Exclusion must happen before follow(). */ +function applyExclusions() { + var names = (CONFIG.excludeModules || []).concat(CONFIG.excludeModulesExtra || []); + var excluded = [], attempted = 0, missing = []; + for (var i = 0; i < names.length; i++) { + var spec = names[i]; + if (!spec) continue; + var plus = spec.indexOf('+'); + var name = plus > 0 ? spec.substring(0, plus) : spec; + if (state.target && name === state.target.name) continue; // never the target + var mod = null; + try { mod = Process.findModuleByName(name); } catch (e) { mod = null; } + if (!mod) { missing.push(name); continue; } + attempted++; + try { + if (plus > 0) { + var off = parseInt(spec.substring(plus + 1), 16) || 0; + Stalker.exclude({ base: mod.base.add(off), size: mod.size - off }); + } else { + Stalker.exclude(mod); + } + excluded.push(name); + } catch (e) { + emit('EXCL-FAIL', spec + ': ' + e); + } + } + emit('EXCL', 'excluded=' + excluded.length + '/' + attempted + + ' [' + excluded.join(',') + ']' + + (missing.length ? ' not-loaded=' + missing.length : '')); +} + function beginFollow(tid, why) { if (state.following) return; if (!state.target) { @@ -237,6 +300,8 @@ function beginFollow(tid, why) { line('MOD ' + state.target.name + ' base=' + state.target.base + ' size=' + state.target.size + ' path=' + state.target.path); + applyExclusions(); + Stalker.follow(tid, { events: { call: CONFIG.events.call, @@ -248,6 +313,20 @@ function beginFollow(tid, why) { onReceive: onStalkEvents }); + /* A zero-event trace has two very different meanings -- "nothing executed" + * and "the pipeline never delivered" -- and the log alone cannot tell them + * apart. This warning exists so the second reading is the default one. */ + if (CONFIG.zeroEventWarnMs > 0) { + state.warnTimer = setTimeout(function () { + if (state.following && state.nBB === 0 && state.nBLK === 0) { + emit('WARN', 'zero events ' + CONFIG.zeroEventWarnMs + + 'ms after follow (blocks=0 blk=0 calls=0) -- the pipeline is ' + + 'NOT proven; do not report this as "the code did not run". ' + + 'Control: follow a thread running a known loop first.'); + } + }, CONFIG.zeroEventWarnMs); + } + if (CONFIG.followMs > 0) { state.timer = setTimeout(function () { endFollow('timeout'); }, CONFIG.followMs); @@ -258,6 +337,7 @@ function endFollow(reason) { if (!state.following) return; state.following = false; if (state.timer !== null) { clearTimeout(state.timer); state.timer = null; } + if (state.warnTimer) { clearTimeout(state.warnTimer); state.warnTimer = null; } try { Stalker.flush(); } catch (e) { /* already gone */ } try { Stalker.unfollow(state.followTid); } catch (e) { /* already gone */ } flush(true); diff --git a/skills/apk-reverse/scripts/vmp_diff_harness.py b/skills/apk-reverse/scripts/vmp_diff_harness.py new file mode 100644 index 0000000..a2bb278 --- /dev/null +++ b/skills/apk-reverse/scripts/vmp_diff_harness.py @@ -0,0 +1,1719 @@ +#!/usr/bin/env python3 +"""vmp_diff_harness.py -- differential-hardening (known-plaintext) harness for Dex-VMP work. + +The idea this implements is the only black-box route to a private opcode table: +compile a fixture whose every instruction is labelled, get the *same* hardening +platform to harden it, align the returned dex against the original +instruction-by-instruction, and read the substitution off the alignment. The +original supplies the instruction boundaries that the hardened stream cannot +supply for itself -- that is the whole trick, and it is also its limit (see +`compare`'s shape verdict and references/vmp-differential-analysis.md). + +Subcommands + build compile the labelled coverage fixture into a dex (and, with + --apk, a minimal APK) and report which opcodes it actually covers + audit report opcode coverage of any dex + compare align an original dex against a hardened one; emit a candidate + opcode map with per-entry confidence and a run-level verdict + simulate forge a "hardened" dex from a *known* private table -- the local + fixture that proves compare() recovers a table it never saw + emit-smali render private-opcode method bodies back into a smali skeleton + +What is measured and what is not: `build`, `audit`, `simulate` and the +`compare`/`emit-smali` code paths are exercised against locally produced +fixtures (see docs/tool-verification/EXTENSION-vmp-diff.md). No third-party +hardening platform was used: its output is what `simulate` stands in for. A +`compare` run against a real hardened dex is therefore an inference built on a +measured mechanism, not a measured result. + +Dependencies: stdlib only, plus `dexutil.py` next to this file. External +toolchain (javac, d8, aapt2) is only needed by `build` and is passed explicitly. + +Examples + python vmp_diff_harness.py build --build-tools E:\\tools\\android-14 --out work/fixture + python vmp_diff_harness.py simulate work/fixture/dex/classes.dex --out work/sim.dex --seed 7 + python vmp_diff_harness.py compare work/fixture/dex/classes.dex work/sim.dex + python vmp_diff_harness.py emit-smali work/sim.dex --table work/map.json --class LOpCoverProbe; +""" +import argparse +import collections +import json +import os +import random +import shutil +import subprocess +import sys +import zipfile + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +try: + import dexutil +except ImportError: # pragma: no cover + dexutil = None + + +# --------------------------------------------------------------------------- +# opcode reachability, measured (see EXTENSION-vmp-diff.md for the run) +# +# A javac+d8 fixture cannot reach every slot in the format table. The reasons +# are structural, not incidental, and they are what the coverage report is for: +# you cannot read a substitution for an opcode your fixture never emitted. +# --------------------------------------------------------------------------- +UNREACHABLE = { + 0x1B: "const-string/jumbo needs a string_ids index > 65535, i.e. a fixture " + "carrying 65536+ string constants", + 0x2A: "goto/32 needs a >32767-code-unit backward jump; javac refuses the " + "method first with 'code too large' (its own 64 KB per-method bytecode cap)", + 0xFE: "const-method-handle has no Java-language literal; reachable only by " + "hand-written smali or direct dex construction", + 0xFF: "const-method-type has no Java-language literal (same as 0xFE)", + 0x09: "move-object/16 (32x) needs both registers >= 256; the object frame " + "reaches 256 on the source side only, which d8 encodes as 0x08", + 0xFD: "invoke-custom/range: d8 emitted the 35c form for every lambda shape " + "tried, including a six-parameter one", +} + + +# --------------------------------------------------------------------------- +# the fixture +# --------------------------------------------------------------------------- +_JAVA_PROBE = r''' +// OpCoverProbe -- opcode coverage fixture for differential-hardening analysis. +// +// Each method is a labelled probe for one dalvik opcode family. The point is not +// that the code does anything useful: it is that the compiled instruction stream +// contains a known, labelled instance of every opcode a Java compiler can emit, +// so that a hardened counterpart of this dex can be aligned instruction-by- +// instruction against it (known-plaintext attack). +public class OpCoverProbe { + + // ---------------------------------------------------------------- 1. const + public static int constFamily(int seed) { + int c4 = 7; // const/4 + int c16 = 1000; // const/16 + int c32 = 100000; // const + int ch16 = 16777216; // const/high16 + long w16 = 123L; // const-wide/16 + long w32 = 1234567L; // const-wide/32 + long w = 1234567890123L; // const-wide + float fl = 1.5f; // const/high16 (float) + double db = 3.14159; // const-wide (double) + String s = "opcover-probe"; // const-string + Class k = String.class; // const-class + int acc = c4 + c16 + c32 + ch16; + acc += (int) w16 + (int) w32; + acc += (int) w + (int) fl + (int) db; + acc += s.length() + k.getName().length(); + return acc + seed; + } + + // ----------------------------------------------------------- 2. int arith + public static int arithInt(int x, int y) { + int r = x + y; // add-int + r = r + 1000; // add-int/lit16 + r = r + 7; // add-int/lit8 + r = r - y; // sub-int + r = r - 1000; // sub-int/lit16 + r = r * y; // mul-int + r = r / (y | 1); // div-int + r = r % (x | 3); // rem-int + r = r << 2; // shl-int/lit8 + r = r >> 1; // shr-int/lit8 + r = r >>> 1; // ushr-int/lit8 + r = r & x; // and-int + r = r | y; // or-int + r = r ^ x; // xor-int + r = -r; // neg-int + r = 0 - r; // rsub-int + return r; + } + + // ------------------------------------------------------ 3. lit8/lit16 forms + public static int litForms(int x, int y) { + int r = x; + r = r * 255; // mul-int/lit8 + r = r * 1000; // mul-int/lit16 + r = r + 100000; // const + add-int (beyond lit16) + r = r / 3; // div-int/lit8 + r = r % 5; // rem-int/lit8 + r = r & 255; // and-int/lit16 + r = r | 255; // or-int/lit16 + r = r ^ 255; // xor-int/lit16 + r = r << y; // shl-int + r = r >> y; // shr-int + r = r >>> y; // ushr-int + return r; + } + + // -------------------------------------------------------------- 4. wide arith + public static long arithWide(long a, long b) { + long r = a * b; // mul-long + r = r + a; // add-long + r = r - b; // sub-long + r = r / (b | 1L); // div-long + r = r % (a | 3L); // rem-long + r = r & a; // and-long + r = r | b; // or-long + r = r ^ a; // xor-long + r = r << 2; // shl-long + r = r >> 1; // shr-long + r = r >>> 1; // ushr-long + r = -r; // neg-long + r = r + 1000L; // add-long with const-wide + return r; + } + + // ----------------------------------------------------------- 5. conversions + public static double conversions(int i, long l, float f, double d, short sh, byte by, char ch) { + long a = (long) i; // int-to-long + float b = (float) i; // int-to-float + double c = (double) i; // int-to-double + int e = (int) l; // long-to-int + float g = (float) l; // long-to-float + double h = (double) l; // long-to-double + int j = (int) f; // float-to-int + long k = (long) f; // float-to-long + double m = (double) f; // float-to-double + int n = (int) d; // double-to-int + long o = (long) d; // double-to-long + float p = (float) d; // double-to-float + int q = sh; + int s = by; + int t = ch; + byte u = (byte) i; // int-to-byte + char v = (char) i; // int-to-char + short w = (short) i; // int-to-short + return a + e + j + n + q + s + t + u + v + w + b + c + g + h + k + m + o + p; + } + + // ------------------------------------------------------- 6. float/double arith + public static float arithFloat(float a, float b) { + float r = a + b; // add-float + r = r - b; // sub-float + r = r * a; // mul-float + r = r / b; // div-float + r = r % a; // rem-float + r = -r; // neg-float + return r + 1.5f; + } + + public static double arithDouble(double a, double b) { + double r = a + b; // add-double + r = r - b; // sub-double + r = r * a; // mul-double + r = r / b; // div-double + r = r % a; // rem-double + r = -r; // neg-double + return r + 2.5; + } + + // ------------------------------------------------------------- 7. cmp family + public static int cmps(long a, long b, float f, float g, double p, double q) { + int r = 0; + if (a < b) r += 1; // cmp-long + if (f > g) r += 2; // cmpg-float + if (f < g) r += 4; // cmpl-float + if (p > q) r += 8; // cmpg-double + if (p <= q) r += 16; // cmpl-double + return r; + } + + // --------------------------------------------------------------- 8. branches + public static int branches(int x, int y) { + int r = 0; + if (x == 0) r += 1; // if-eqz + if (x != 0) r += 2; // if-nez + if (x < 0) r += 3; // if-ltz + if (x >= 0) r += 4; // if-gez + if (x > 0) r += 5; // if-gtz + if (x <= 0) r += 6; // if-lez + if (x == y) r += 7; // if-eq + if (x != y) r += 8; // if-ne + if (x < y) r += 9; // if-lt + if (x >= y) r += 10; // if-ge + if (x > y) r += 11; // if-gt + if (x <= y) r += 12; // if-le + return r; + } + + public static int objectBranches(Object a, Object b) { + int r = 0; + if (a == null) r += 1; // if-eqz on an object register + if (a != null) r += 2; + if (a == b) r += 3; // if-eq + if (a != b) r += 4; // if-ne + return r; + } + + // ---------------------------------------------------------------- 9. switches + public static int switchPacked(int x) { + switch (x) { // packed-switch + packed-switch-payload + case 0: return 10; + case 1: return 11; + case 2: return 12; + case 3: return 13; + case 4: return 14; + default: return 15; + } + } + + public static int switchSparse(int x) { + switch (x) { // sparse-switch + sparse-switch-payload + case -1000000: return 20; + case 0: return 21; + case 12345: return 22; + case 999999999: return 23; + default: return 24; + } + } + + // ------------------------------------------------------------------ 10. arrays + public static int arrays(int n, byte by, char ch, short sh, boolean bo, + long l, float f, double d, Object o) { + int[] ia = new int[n]; // new-array + long[] la = new long[n]; + float[] fa = new float[n]; + double[] da = new double[n]; + byte[] ba = new byte[n]; + char[] ca = new char[n]; + short[] sa = new short[n]; + boolean[] za = new boolean[n]; + Object[] oa = new Object[n]; + ia[0] = 1; // aput + la[0] = l; // aput-wide + fa[0] = f; + da[0] = d; + ba[0] = by; // aput-byte + ca[0] = ch; // aput-char + sa[0] = sh; // aput-short + za[0] = bo; // aput-boolean + oa[0] = o; // aput-object + int r = ia[0]; // aget + r += (int) la[0]; // aget-wide + r += (int) fa[0]; + r += (int) da[0]; + r += ba[0]; // aget-byte + r += ca[0]; // aget-char + r += sa[0]; // aget-short + r += za[0] ? 1 : 0; // aget-boolean + r += oa[0] == null ? 0 : 1; // aget-object + r += ia.length + la.length + ba.length + oa.length; // array-length + return r; + } + + public static int fillArrayData() { + int[] a = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10}; + long[] b = {11L, 22L, 33L}; + short[] c = {44, 55, 66}; + int s = 0; + for (int v : a) s += v; + for (long v : b) s += (int) v; + for (short v : c) s += v; + return s; + } + + public static int filledNewArray() { + int[] a = new int[]{7, 8, 9}; + return a[0] + a[1] + a[2]; + } + + // ------------------------------------------------------------------ 11. fields + static int sInt; + static long sLong; + static float sFloat; + static double sDouble; + static Object sObj; + static boolean sBool; + static byte sByte; + static char sChar; + static short sShort; + static String sStr; + + int iInt; + long iLong; + float iFloat; + double iDouble; + Object iObj; + boolean iBool; + byte iByte; + char iChar; + short iShort; + String iStr; + + public static int staticFields(int a) { + sInt = a; // sput + sLong = a; // sput-wide + sFloat = a; + sDouble = a; + sObj = null; // sput-object + sStr = "static-field"; + sBool = true; + sByte = (byte) a; + sChar = (char) a; + sShort = (short) a; + int r = sInt; // sget + r += (int) sLong; // sget-wide + r += (int) sFloat; + r += (int) sDouble; + r += sBool ? 1 : 0; + r += sByte; + r += sChar; + r += sShort; + r += sObj == null ? 0 : 1; // sget-object + r += sStr.length(); + return r; + } + + public int instanceFields(int a) { + iInt = a; // iput + iLong = a; // iput-wide + iFloat = a; + iDouble = a; + iObj = this; // iput-object + iStr = "instance-field"; + iBool = false; + iByte = (byte) a; + iChar = (char) a; + iShort = (short) a; + int r = iInt; // iget + r += (int) iLong; // iget-wide + r += (int) iFloat; + r += (int) iDouble; + r += iBool ? 1 : 0; + r += iByte; + r += iChar; + r += iShort; + r += iObj == null ? 0 : 1; // iget-object + r += iStr.length(); + return r; + } + + // ----------------------------------------------------------------- 12. invokes + public interface Iface { + int call(int x); + } + + public static class Base { + public int baseCall(int x) { return x + 1; } + } + + public static class Impl extends Base implements Iface { + public int call(int x) { return x + 2; } // invoke-virtual + private int privateCall(int x) { return x + 4; } // invoke-direct + public static int staticCall(int x) { return x + 3; } // invoke-static + public int superCall(int x) { return super.baseCall(x); } // invoke-super + public int useAll(int x) { + return call(x) + superCall(x) + staticCall(x) + privateCall(x); + } + } + + public static int helper(int x) { return x * 2; } + + public static Iface prov(int x) { return new Impl(); } + + public static int invokeVirtualAndStatic(int x) { + Impl i = new Impl(); + return i.call(x) + Impl.staticCall(x) + helper(x); + } + + public static int invokeInterface(Iface f, int x) { + return f.call(x); // invoke-interface + } + + public static int invokeSuper(int x) { + Impl i = new Impl(); + return i.superCall(x); // invoke-super + } + + public static int lambdaProbe(int x) { + Iface f = (v) -> v + 100; // invoke-custom + java.util.function.IntUnaryOperator g = (v) -> v * 2; + java.util.function.IntBinaryOperator h = OpCoverProbe::combine; // method ref + return f.call(x) + g.applyAsInt(x) + h.applyAsInt(x, 3); + } + + public static int combine(int a, int b) { return a + b * 7; } + + // ---------------------------------------------------------------- 13. exceptions + public static int exceptions(int x) { + int r = 0; + try { + r = 100 / x; // div-int inside a try range + } catch (ArithmeticException e) { // move-exception + r = -1; + } finally { + r = r + 1; + } + try { + if (x == 0) throw new IllegalStateException("boom"); // throw + r += 2; + } catch (IllegalStateException e) { + r += 3; + } + return r; + } + + public static void rethrow() { + throw new RuntimeException("always"); // throw on a new-instance + } + + // ------------------------------------------------------------------ 14. monitor + public static int monitor(Object lock) { + synchronized (lock) { // monitor-enter / monitor-exit + return 42; + } + } + + public static int monitorOnThis(Object lock, int x) { + synchronized (OpCoverProbe.class) { + x += 1; + } + synchronized (lock) { + x += 2; + } + return x; + } + + // ------------------------------------------------------------------- 15. types + public static int typeOps(Object o, int x) { + int r = 0; + if (o instanceof String) r += 1; // instance-of + String s = (String) o; // check-cast + r += s.length(); + Object n = new Object(); // new-instance + Impl i = new Impl(); + r += i.call(x); + return r + (n == null ? 0 : 1); + } + + // ------------------------------------------------------------------- 16. loops + public static int loops(int n) { + int s = 0; + for (int i = 0; i < n; i++) s += i; // goto + if-lt + int j = 0; + while (j < 10) { j++; } + int k = 0; + do { k++; } while (k < 5); + for (int a = 0; a < 3; a++) { + for (int b = 0; b < 3; b++) { + for (int c = 0; c < 3; c++) { + s += a * b * c; + } + } + } + return s + j + k; + } + + // ------------------------------------- 16b. non-2addr (tri-register) forms + // javac emits the /2addr form whenever the destination is one of the + // operands. Three distinct locals per operation are what forces the plain + // three-register opcode to exist in the stream at all. + public static int triInt(int a1, int a2, int a3, int a4, int a5, int a6, int a7, + int a8, int a9, int a10, int a11, int a12, int a13, int a14) { + int t1 = a1 - a2; // sub-int + int t2 = a3 / (a4 | 1); // div-int + int t3 = a5 % (a6 | 1); // rem-int + int t4 = a7 & a8; // and-int + int t5 = a9 | a10; // or-int + int t6 = a11 ^ a12; // xor-int + int t7 = a13 << (a14 & 7); // shl-int + int t8 = a13 >> (a14 & 7); // shr-int + int t9 = a13 >>> (a14 & 7); // ushr-int + int t10 = ~t1; // not-int + return t1 + t2 + t3 + t4 + t5 + t6 + t7 + t8 + t9 + t10; + } + + public static long triLong(long b1, long b2, long b3, long b4, long b5, + long b6, long b7, long b8, long b9, long b10) { + long u1 = b1 + b2; // add-long + long u2 = b3 - b4; // sub-long + long u3 = b5 / (b6 | 1L); // div-long + long u4 = b7 % (b8 | 1L); // rem-long + long u5 = b9 & b10; // and-long + long u6 = b1 | b2; // or-long + long u7 = b3 ^ b4; // xor-long + long u8 = b5 << 2; // shl-long + long u9 = b5 >> 2; // shr-long + long u10 = b5 >>> 2; // ushr-long + long u11 = ~u1; // not-long + return u1 + u2 + u3 + u4 + u5 + u6 + u7 + u8 + u9 + u10 + u11; + } + + public static float triFloat(float f1, float f2, float f3, float f4, float f5, float f6) { + float g1 = f1 - f2; // sub-float + float g2 = f3 * f4; // mul-float + float g3 = f5 / f6; // div-float + float g4 = f1 % f2; // rem-float + return g1 + g2 + g3 + g4; + } + + public static double triDouble(double d1, double d2, double d3, double d4, double d5, double d6) { + double h1 = d1 - d2; // sub-double + double h2 = d3 * d4; // mul-double + double h3 = d5 / d6; // div-double + double h4 = d1 % d2; // rem-double + return h1 + h2 + h3 + h4; + } + + // ------------------------------------------------ 16c. literal-width variants + public static int litVariants(int x) { + int r = x; + r = r / 1000; // div-int/lit16 (beyond lit8) + r = r % 1000; // rem-int/lit16 + r = r & 15; // and-int/lit8 + r = r ^ 15; // xor-int/lit8 + r = 1000 - r; // rsub-int + r = r + 1000; // add-int/lit16 + return r; + } + + // ------------------------------------------------- 16d. per-type field access + // Read straight into a fresh local instead of accumulating: the accumulator + // form is where d8 collapses a typed access into a narrower one. + public int fieldVariants(int a, Object o, String s) { + iBool = true; + iByte = (byte) a; + iChar = (char) a; + iShort = (short) a; + iObj = o; + iStr = s; + long l = iLong; // iget-wide + Object ob = iObj; // iget-object + boolean b = iBool; // iget-boolean + byte by = iByte; // iget-byte + char ch = iChar; // iget-char + short sh = iShort; // iget-short + return (b ? 1 : 0) + by + ch + sh + (int) l + (ob == null ? 0 : 1); + } + + public static int staticFieldVariants(int a, Object o, String s) { + sBool = true; + sByte = (byte) a; + sChar = (char) a; + sShort = (short) a; + sObj = o; + sStr = s; + sLong = a; + long l = sLong; // sget-wide + Object ob = sObj; // sget-object + boolean b = sBool; // sget-boolean + byte by = sByte; // sget-byte + char ch = sChar; // sget-char + short sh = sShort; // sget-short + return (b ? 1 : 0) + by + ch + sh + (int) l + (ob == null ? 0 : 1); + } + + // ------------------------------------------------------- 16e. /range invokes + // The 35c invoke forms carry at most 5 argument registers. Seven (this plus + // six ints) is what makes the /range encoding the only legal one. + public interface Iface2 { int call6(int a, int b, int c, int d, int e, int f); } + + public static class Base6 { + public int base6(int a, int b, int c, int d, int e, int f) { return a + b + c + d + e + f; } + } + + public static class Impl6 extends Base6 implements Iface2 { + public int call6(int a, int b, int c, int d, int e, int f) { return a * f; } + private int priv6(int a, int b, int c, int d, int e, int f) { return a - f; } + public int superRange(int a, int b, int c, int d, int e, int f) { return super.base6(a, b, c, d, e, f); } + public int directRange(int a, int b, int c, int d, int e, int f) { return priv6(a, b, c, d, e, f); } + } + + public static int rangeInvokes(int a, int b, int c, int d, int e, int f) { + Impl6 i = new Impl6(); + int r = i.call6(a, b, c, d, e, f); // invoke-virtual/range + r += i.superRange(a, b, c, d, e, f); // invoke-super/range + r += i.directRange(a, b, c, d, e, f); // invoke-direct/range + Iface2 f2 = i; + r += f2.call6(a, b, c, d, e, f); // invoke-interface/range + return r; + } + + // ------------------------------------------------- 16f. polymorphic invoke + public static int polyProbe(int x) { + try { + java.lang.invoke.MethodHandle mh = java.lang.invoke.MethodHandles.lookup() + .findStatic(OpCoverProbe.class, "combine", + java.lang.invoke.MethodType.methodType(int.class, int.class, int.class)); + return (int) mh.invokeExact(x, 3); // invoke-polymorphic + } catch (Throwable t) { + return -1; + } + } + + // ------------------------------------- 16g. remaining reachable opcodes + public static long mulLong2addr(long a, long b) { + long r = a; + r = r * b; // mul-long/2addr (the only /2addr form javac skipped) + return r; + } + + public static long moveWide(int seed) { + long a = seed; + long b = a; + long c = b; + return a + b + c; + } + + public static Object manyElements(Object o1, Object o2, Object o3, Object o4, + Object o5, Object o6, Object o7) { + Object[] a = {o1, o2, o3, o4, o5, o6, o7}; // filled-new-array/range + return a; + } + + public interface Iface6 { int call6(int a, int b, int c, int d, int e, int f); } + + public static int sum6(int a, int b, int c, int d, int e, int f) { + return a + b + c + d + e + f; + } + + public static int polyRangeProbe(int a, int b, int c, int d, int e, int f) { + try { + java.lang.invoke.MethodHandle mh = java.lang.invoke.MethodHandles.lookup() + .findStatic(OpCoverProbe.class, "sum6", + java.lang.invoke.MethodType.methodType(int.class, int.class, int.class, + int.class, int.class, int.class, int.class)); + return (int) mh.invokeExact(a, b, c, d, e, f); // invoke-polymorphic/range + } catch (Throwable t) { + return -1; + } + } + + public static int customRangeProbe(int a, int b, int c, int d, int e, int f) { + Iface6 g = (p1, p2, p3, p4, p5, p6) -> p1 + p2 + p3 + p4 + p5 + p6; + return g.call6(a, b, c, d, e, f); + } + + // ---------------------------------------------- 17. many registers / move forms + public static int manyRegs(int seed) { + int r0 = seed + 0, r1 = seed + 1, r2 = seed + 2, r3 = seed + 3; + int r4 = seed + 4, r5 = seed + 5, r6 = seed + 6, r7 = seed + 7; + int r8 = seed + 8, r9 = seed + 9, r10 = seed + 10, r11 = seed + 11; + int r12 = seed + 12, r13 = seed + 13, r14 = seed + 14, r15 = seed + 15; + int r16 = seed + 16, r17 = seed + 17, r18 = seed + 18, r19 = seed + 19; + int r20 = seed + 20, r21 = seed + 21, r22 = seed + 22, r23 = seed + 23; + int r24 = seed + 24, r25 = seed + 25, r26 = seed + 26, r27 = seed + 27; + int r28 = seed + 28, r29 = seed + 29, r30 = seed + 30, r31 = seed + 31; + int s = 0; + s += r0; s += r1; s += r2; s += r3; s += r4; s += r5; s += r6; s += r7; + s += r8; s += r9; s += r10; s += r11; s += r12; s += r13; s += r14; s += r15; + s += r16; s += r17; s += r18; s += r19; s += r20; s += r21; s += r22; s += r23; + s += r24; s += r25; s += r26; s += r27; s += r28; s += r29; s += r30; s += r31; + return s; + } + + // ------------------------------------------------------- 18. ternary / boolean + public static int ternaryBoolean(boolean a, boolean b, int x) { + int r = a ? x : -x; + r += b ? 1 : 0; + r += (a && b) ? 2 : 0; + r += (a || b) ? 4 : 0; + return r; + } + + // ------------------------------------------------------------- 19. string ops + public static int stringOps(String s, int x) { + int r = s.length(); // invoke-virtual + r += s.charAt(0); + r += s.indexOf('a'); + r += s.equals("abc") ? 1 : 0; + r += s.hashCode(); + String t = s + x; // StringBuilder + return r + t.length(); + } + + // an entry point that references everything, so nothing is dead-stripped + public static int runAll(int seed) { + int r = 0; + r += constFamily(seed); + r += arithInt(seed, seed + 1); + r += litForms(seed, 3); + r += (int) arithWide(seed, seed + 2); + r += (int) conversions(seed, seed, seed, seed, (short) seed, (byte) seed, (char) seed); + r += (int) arithFloat(seed, seed + 1); + r += (int) arithDouble(seed, seed + 1); + r += cmps(seed, seed + 1, seed, seed + 1, seed, seed + 1); + r += branches(seed, seed + 1); + r += objectBranches(null, null); + r += switchPacked(seed % 5); + r += switchSparse(seed); + r += arrays(4, (byte) 1, 'a', (short) 2, true, 3L, 4f, 5.0, null); + r += fillArrayData(); + r += filledNewArray(); + r += staticFields(seed); + r += new OpCoverProbe().instanceFields(seed); + r += invokeVirtualAndStatic(seed); + r += invokeInterface(prov(seed), seed); + r += invokeSuper(seed); + r += lambdaProbe(seed); + r += exceptions(seed); + r += monitor(new Object()); + r += monitorOnThis(new Object(), seed); + r += typeOps("hello", seed); + r += loops(seed % 7); + r += manyRegs(seed); + r += ternaryBoolean(true, false, seed); + r += stringOps("abcdef", seed); + r += triInt(1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14); + r += (int) triLong(1L, 2L, 3L, 4L, 5L, 6L, 7L, 8L, 9L, 10L); + r += (int) triFloat(1f, 2f, 3f, 4f, 5f, 6f); + r += (int) triDouble(1.0, 2.0, 3.0, 4.0, 5.0, 6.0); + r += litVariants(seed); + r += new OpCoverProbe().fieldVariants(seed, null, "fv"); + r += staticFieldVariants(seed, null, "sfv"); + r += rangeInvokes(1, 2, 3, 4, 5, 6); + r += polyProbe(seed); + r += (int) mulLong2addr(seed, seed + 1); + r += (int) moveWide(seed); + r += manyElements(null, null, null, null, null, null, null) == null ? 0 : 1; + r += polyRangeProbe(1, 2, 3, 4, 5, 6); + r += customRangeProbe(1, 2, 3, 4, 5, 6); + return r; + } + + public static void main(String[] args) { + int seed = args.length > 0 ? args.length : 1; + System.out.println(runAll(seed)); + } +} +''' + + +def gen_big_probe(n_int=300, n_long=140, n_obj=160, goto16_units=200): + """Generate the part of the fixture a Java compiler can only reach with bulk. + + Three opcode classes exist only past a register-number or branch-distance + threshold, so no amount of hand-written Java reaches them -- the source has + to be long: + + move/16, move-wide/16, move-object/from16 register number >= 256 + goto/16 backward branch > 127 code units + + Measured thresholds for this generator (see EXTENSION-vmp-diff.md): + n_int=300 -> move/16 seen 46x; n_long=140 -> move-wide/16 seen 20x; + n_obj=160 -> move-object/from16 seen 161x; goto16_units=200 -> goto/16 seen 1x. + """ + src = ["// generated by vmp_diff_harness.py build -- do not edit by hand", + "public class OpCoverBig {", ""] + + src.append(" // %d locals: register number crosses 8 bits" % n_int) + src.append(" public static int hugeFrame(int seed) {") + for i in range(n_int): + src.append(" int v%d = seed + %d;" % (i, i % 100)) + src.append(" int s = 0;") + for i in range(n_int): + src.append(" s += v%d;" % i) + src.append(" return s;") + src.append(" }") + src.append("") + + src.append(" // wide values occupy two slots each, so fewer of them suffice") + src.append(" public static long hugeWideFrame(long seed) {") + for i in range(n_long): + src.append(" long w%d = seed + %dL;" % (i, i % 50)) + src.append(" long s = 0L;") + for i in range(n_long): + src.append(" s += w%d;" % i) + src.append(" return s;") + src.append(" }") + src.append("") + + src.append(" // object frame: move-object/from16") + src.append(" public static int hugeObjectFrame(Object o) {") + for i in range(n_obj): + src.append(" Object a%d = o;" % i) + src.append(" int s = 0;") + for i in range(n_obj): + src.append(" if (a%d == null) s += 1;" % i) + src.append(" return s;") + src.append(" }") + src.append("") + + src.append(" // %d statements in the body: the backward branch no longer fits") + src.append(" // in the 8-bit goto form and must use goto/16") + src.append(" public static int gotoNear(int x) {") + src.append(" int r = x;") + src.append(" while (r > 0) {") + for i in range(goto16_units): + src.append(" r += %d;" % (i % 120 + 1)) + src.append(" r -= 1;") + src.append(" }") + src.append(" return r;") + src.append(" }") + src.append("") + + src.append(" public static int runAll(int seed) {") + src.append(" int r = hugeFrame(seed);") + src.append(" r += (int) hugeWideFrame(seed);") + src.append(" r += hugeObjectFrame(null);") + src.append(" r += gotoNear(seed % 5);") + src.append(" return r;") + src.append(" }") + src.append("}") + return "\n".join(src) + "\n" + + +MINIMAL_MANIFEST = ''' + + + +''' + + +# --------------------------------------------------------------------------- +# helpers +# --------------------------------------------------------------------------- + +def _require_dexutil(): + if dexutil is None: + sys.stderr.write("error: dexutil.py is not importable; keep this script " + "next to the rest of skills/apk-reverse/scripts/\n") + raise SystemExit(2) + + +def _which_in_dir(directory, stem): + """Locate a build-tool binary under `directory`, tolerating .exe/.bat/.cmd.""" + for suffix in (".exe", ".bat", ".cmd", ""): + candidate = os.path.join(directory, stem + suffix) + if os.path.isfile(candidate): + return candidate + return None + + +def _methods(dex): + """{(class, name, desc): code_off} for every method with a class_def.""" + out = {} + for i in range(dex.header["class_defs_size"]): + off = dex.header["class_defs_off"] + i * 32 + for _section, _idx, cls, name, desc, code_off in dex.methods_at(off): + out[(cls, name, desc)] = code_off + return out + + +def _count_opcodes(dex): + """collections.Counter of decoded opcode byte -> number of instructions.""" + counts = collections.Counter() + insns = 0 + desync = 0 + for _key, code_off in _methods(dex).items(): + if not code_off: + continue + listing, clean, _expected = dex.decode_all(code_off) + if not clean: + desync += 1 + for insn in listing: + counts[insn["op"]] += 1 + insns += 1 + return counts, insns, desync + + +CLASS_FLAGS = [(0x1, "public"), (0x10, "final"), (0x200, "interface"), + (0x400, "abstract"), (0x1000, "synthetic"), (0x2000, "annotation"), + (0x4000, "enum")] +METHOD_FLAGS = [(0x1, "public"), (0x2, "private"), (0x4, "protected"), + (0x8, "static"), (0x10, "final"), (0x20, "synchronized"), + (0x40, "bridge"), (0x80, "varargs"), (0x100, "native"), + (0x400, "abstract"), (0x1000, "synthetic"), (0x10000, "constructor")] + + +def _flags(value, table): + return " ".join(name for bit, name in table if value & bit) + + +def _methods_with_acc(dex, class_def_off): + """Like Dex.methods_at(), but also yields each method's access_flags. + + `dexutil.Dex.methods_at` reads `_acc` and drops it, which is fine for the + callers it already has; a smali skeleton needs it, and changing that shared + generator's tuple shape would break them. + """ + p = dexutil.u32(dex.d, class_def_off + 24) + if p == 0: + return + sf, p = dexutil.read_uleb(dex.d, p) + inf, p = dexutil.read_uleb(dex.d, p) + dm, p = dexutil.read_uleb(dex.d, p) + vm, p = dexutil.read_uleb(dex.d, p) + for _ in range(sf + inf): # skip field lists + _i, p = dexutil.read_uleb(dex.d, p) + _a, p = dexutil.read_uleb(dex.d, p) + for section, count in (("direct", dm), ("virtual", vm)): + running = 0 + for _ in range(count): + diff, p = dexutil.read_uleb(dex.d, p) + running += diff # indices are delta-encoded + acc, p = dexutil.read_uleb(dex.d, p) + code_off, p = dexutil.read_uleb(dex.d, p) + cls, name, desc = dex.method(running) + yield section, running, cls, name, desc, acc, code_off + + +def _class_matches(class_type, wanted): + """Accept 'LOpCoverProbe;', 'LOpCoverProbe' and 'OpCoverProbe' alike. + + The first is how the dex spells it, the third is how a reader thinks of it, + and a filter that accepts only one of the three silently matches nothing -- + which reads exactly like a class that is not in the file. + """ + if not wanted: + return True + bare = class_type.strip("L;").replace("/", ".") + return wanted in (class_type, class_type.rstrip(";"), class_type.strip("L;"), bare) + + +def _fmt_table(rows, columns): + widths = [max(len(str(r[i])) for r in [columns] + rows) for i in range(len(columns))] + out = [" " + " ".join(str(c).ljust(widths[i]) for i, c in enumerate(columns))] + out.append(" " + " ".join("-" * w for w in widths)) + for row in rows: + out.append(" " + " ".join(str(c).ljust(widths[i]) for i, c in enumerate(row))) + return "\n".join(out) + + +# --------------------------------------------------------------------------- +# build / audit +# --------------------------------------------------------------------------- + +def cmd_audit(args): + _require_dexutil() + rc = 0 + for path in args.dex: + dex, entry = dexutil.load_dex(path) + problems = dex.check() + counts, insns, desync = _count_opcodes(dex) + known = set(dexutil.OP_NAMES) + covered = sorted(set(counts) & known) + missing = sorted(known - set(counts)) + print("== %s (entry %s, %d bytes) ==" % (path, entry, len(dex.d))) + for p in problems: + print(" [structure] %s" % p) + print(" classes=%d methods_with_code=%d instructions=%d desynced=%d" + % (dex.header["class_defs_size"], + sum(1 for v in _methods(dex).values() if v), insns, desync)) + print(" opcode coverage: %d/%d (%.1f%%)" + % (len(covered), len(known), 100.0 * len(covered) / max(1, len(known)))) + if missing: + print(" not emitted by this fixture (%d):" % len(missing)) + for op in missing: + note = UNREACHABLE.get(op) + print(" 0x%02x %-24s %s" + % (op, dexutil.OP_NAMES[op], note or "(no recorded reason)")) + if args.json: + payload = { + "dex": path, + "classes": dex.header["class_defs_size"], + "instructions": insns, + "desynced_methods": desync, + "structural_problems": problems, + "covered": len(covered), + "total_known": len(known), + "missing": [{"op": "0x%02x" % op, "name": dexutil.OP_NAMES[op], + "reason": UNREACHABLE.get(op)} for op in missing], + "counts": {"0x%02x" % op: n for op, n in sorted(counts.items())}, + } + with open(args.json, "w", encoding="utf-8") as fh: + json.dump(payload, fh, indent=2, sort_keys=True) + print(" wrote %s" % args.json) + if problems or desync: + rc = 1 + return rc + + +def cmd_build(args): + _require_dexutil() + out = args.out + src_dir = os.path.join(out, "src") + cls_dir = os.path.join(out, "classes") + dex_dir = os.path.join(out, "dex") + for d in (src_dir, cls_dir, dex_dir): + os.makedirs(d, exist_ok=True) + + # --- toolchain ------------------------------------------------------- + javac = args.javac or shutil.which("javac") + if not javac: + sys.stderr.write("error: javac not found; pass --javac (the JDK's bin/" + "javac, not the Oracle javapath forwarder)\n") + return 2 + if not args.build_tools: + sys.stderr.write("error: --build-tools is required (the directory holding " + "d8 and aapt2); this script does not guess at a path\n") + return 2 + d8 = _which_in_dir(args.build_tools, "d8") + aapt2 = _which_in_dir(args.build_tools, "aapt2") + if not d8: + sys.stderr.write("error: d8/d8.bat not found in %s\n" % args.build_tools) + return 2 + if args.apk and not aapt2: + sys.stderr.write("error: aapt2 not found in %s (needed for --apk)\n" + % args.build_tools) + return 2 + print("javac: %s" % javac) + print("d8: %s" % d8) + if args.apk: + print("aapt2: %s" % aapt2) + + # --- source ---------------------------------------------------------- + probe_java = os.path.join(src_dir, "OpCoverProbe.java") + with open(probe_java, "w", encoding="utf-8") as fh: + fh.write(_JAVA_PROBE.strip() + "\n") + sources = [probe_java] + if args.with_big: + big_java = os.path.join(src_dir, "OpCoverBig.java") + with open(big_java, "w", encoding="utf-8") as fh: + fh.write(gen_big_probe(args.n_int, args.n_long, args.n_obj, args.goto16)) + sources.append(big_java) + print("generated: %s (%d B)" % (big_java, os.path.getsize(big_java))) + + # --- javac ----------------------------------------------------------- + cmd = [javac, "-g", "--release", "11", "-d", cls_dir] + sources + print("\n$ %s" % " ".join(cmd)) + proc = subprocess.run(cmd, capture_output=True, text=True) + if proc.stdout.strip(): + print(proc.stdout.strip()) + if proc.returncode != 0: + sys.stderr.write(proc.stderr.strip() + "\n") + sys.stderr.write("error: javac failed (rc=%d)\n" % proc.returncode) + return 1 + + classes = [] + for root, _dirs, files in os.walk(cls_dir): + classes += [os.path.join(root, f) for f in sorted(files) if f.endswith(".class")] + if not classes: + sys.stderr.write("error: javac produced no .class files\n") + return 1 + print("javac ok: %d class files" % len(classes)) + + # --- d8 (the output directory must already exist) --------------------- + cmd = [d8, "--min-api", str(args.min_api)] + if not args.desugar: + cmd.append("--no-desugaring") + cmd += ["--output", dex_dir] + classes + print("\n$ %s" % " ".join(cmd)) + proc = subprocess.run(cmd, capture_output=True, text=True) + if proc.stdout.strip(): + print(proc.stdout.strip()) + if proc.returncode != 0: + sys.stderr.write((proc.stderr or "").strip()[:2000] + "\n") + sys.stderr.write("error: d8 failed (rc=%d)\n" % proc.returncode) + return 1 + dex_path = os.path.join(dex_dir, "classes.dex") + if not os.path.isfile(dex_path): + sys.stderr.write("error: d8 reported success but %s does not exist\n" % dex_path) + return 1 + print("d8 ok: %s (%d B)" % (dex_path, os.path.getsize(dex_path))) + + audit_args = argparse.Namespace(dex=[dex_path], json=os.path.join(out, "audit.json")) + audit_rc = cmd_audit(audit_args) + + # --- optional APK ---------------------------------------------------- + apk_rc = 0 + if args.apk: + apk_dir = os.path.join(out, "apk") + os.makedirs(apk_dir, exist_ok=True) + manifest = os.path.join(apk_dir, "AndroidManifest.xml") + with open(manifest, "w", encoding="utf-8") as fh: + fh.write(MINIMAL_MANIFEST) + base = os.path.join(apk_dir, "base.apk") + # No -I android.jar: the manifest deliberately uses no android: attribute, + # because aapt2 refuses to link resource references it has no framework + # package to resolve (measured: "attribute android:versionCode not found"). + cmd = [aapt2, "link", "--manifest", manifest, "-o", base] + print("\n$ %s" % " ".join(cmd)) + proc = subprocess.run(cmd, capture_output=True, text=True) + if proc.stdout.strip(): + print(proc.stdout.strip()) + if proc.returncode != 0: + sys.stderr.write((proc.stderr or "").strip()[:2000] + "\n") + sys.stderr.write("error: aapt2 link failed (rc=%d)\n" % proc.returncode) + apk_rc = 1 + else: + apk_path = os.path.join(out, "probe.apk") + with zipfile.ZipFile(base) as zin, \ + zipfile.ZipFile(apk_path, "w", zipfile.ZIP_DEFLATED) as zout: + for item in zin.namelist(): + zout.writestr(item, zin.read(item)) + zout.write(dex_path, "classes.dex") + print("aapt2 ok: %s (%d B, unsigned)" % (apk_path, os.path.getsize(apk_path))) + print(" unsigned by design: add your own signing step before " + "handing this to a hardening platform that demands it") + return max(audit_rc, apk_rc) + + +# --------------------------------------------------------------------------- +# simulate -- the local stand-in for a hardening platform +# --------------------------------------------------------------------------- + +def _private_permutation(seed): + """A deterministic bijection over 0..255 standing in for a private opcode space. + + Real engines do not merely relabel bytes -- they re-encode the stream and + carry their own length table. `simulate` deliberately does the *weakest* + version of the attack (relabel only, lengths preserved) because that is the + only version this harness can check itself against; see the reference file + for why the stronger version is where the method earns its limits. + """ + rng = random.Random(seed) + targets = [b for b in range(256) if b != 0x00] + shuffled = targets[:] + rng.shuffle(shuffled) + table = {0x00: 0x00} + for src, dst in zip(targets, shuffled): + table[src] = dst + return table + + +def cmd_simulate(args): + _require_dexutil() + with open(args.dex, "rb") as fh: + data = bytearray(fh.read()) + dex = dexutil.Dex(bytes(data)) + problems = dex.check() + if problems: + for p in problems: + sys.stderr.write("error: refusing to simulate on a broken read: %s\n" % p) + return 1 + + table = _private_permutation(args.seed) + out = bytearray(data) + rewritten = 0 + if args.mode == "relabel": + for _key, code_off in _methods(dex).items(): + if not code_off: + continue + # Boundaries come from the dex we are rewriting, which is still in the + # clear at this point -- that is exactly the advantage simulate gives + # compare, and exactly what is missing against a real hardened sample. + for insn in dex.decode(code_off): + out[insn["off"]] = table[insn["op"]] + rewritten += 1 + elif args.mode == "stub": + # The extraction shape: every body is replaced by a filler stub of the + # same length. Instruction counts still line up, so compare will align + # happily -- and every original opcode will claim the same private byte, + # which is the signal that this is not a relabelling at all. + for _key, code_off in _methods(dex).items(): + if not code_off: + continue + info = dex.code_info(code_off) + pos, end = info["insns_off"], info["insns_off"] + info["insns_size"] * 2 + while pos + 2 <= end: + out[pos], out[pos + 1] = 0x0E, 0x00 # return-void + pos += 2 + rewritten += 1 + elif args.mode == "shrink": + # The length-changing shape. This builds a deliberately structurally + # inconsistent image -- insns_size is rewritten without moving the bytes + # -- purely to drive compare's `resized` branch on demand. It is a probe + # for the detector, not something you could load. + for _key, code_off in _methods(dex).items(): + if not code_off: + continue + info = dex.code_info(code_off) + new_size = max(1, info["insns_size"] // 2) + out[code_off + 12:code_off + 16] = new_size.to_bytes(4, "little") + rewritten += 1 + if not args.keep_header: + dexutil.fix_dex_header(out) + with open(args.out, "wb") as fh: + fh.write(bytes(out)) + print("simulated: %s -> %s" % (args.dex, args.out)) + print(" mode=%s, %d method body(ies)/instruction(s) touched (seed %d)" + % (args.mode, rewritten, args.seed)) + if args.mode == "shrink": + print(" NOTE: mode=shrink writes a structurally inconsistent dex " + "on purpose; it exists to exercise compare's resized detector") + if not args.keep_header: + ck, sg = dexutil.verify_dex_header(out) + print(" header recomputed: checksum_ok=%s signature_ok=%s" % (ck, sg)) + if args.table and args.mode == "relabel": + with open(args.table, "w", encoding="utf-8") as fh: + json.dump({"seed": args.seed, + "note": "ground-truth private table (orig opcode -> private byte)", + "map": {"0x%02x" % k: "0x%02x" % v for k, v in sorted(table.items())}}, + fh, indent=2, sort_keys=True) + print(" ground-truth table written to %s (compare must re-derive it)" + % args.table) + return 0 + + +# --------------------------------------------------------------------------- +# compare -- the differential itself +# --------------------------------------------------------------------------- + +def cmd_compare(args): + _require_dexutil() + orig, orig_entry = dexutil.load_dex(args.original) + hard, hard_entry = dexutil.load_dex(args.hardened) + for label, dex in (("original", orig), ("hardened", hard)): + problems = dex.check() + if problems: + for p in problems: + sys.stderr.write("error: %s (%s) does not parse: %s\n" + % (label, dex.name, p)) + return 1 + + om = _methods(orig) + hm = _methods(hard) + + shape = collections.Counter() + pairs = collections.defaultdict(collections.Counter) + per_method = [] + total_slots = 0 + + for key in sorted(set(om) | set(hm)): + cls, name, desc = key + label = "%s->%s%s" % (cls, name, desc) + if key not in hm: + shape["missing"] += 1 + per_method.append((label, "missing", 0, 0)) + continue + if key not in om: + shape["added"] += 1 + per_method.append((label, "added", 0, 0)) + continue + o_off, h_off = om[key], hm[key] + if not o_off: + shape["no-original-body"] += 1 + continue + if not h_off: + # The body left the dex entirely: native sink, or extracted to an + # interpreter. There is no byte stream to align against. + shape["stripped"] += 1 + per_method.append((label, "stripped", 0, 0)) + continue + o_info = orig.code_info(o_off) + h_info = hard.code_info(h_off) + o_size = o_info["insns_size"] + h_size = h_info["insns_size"] + if o_size != h_size: + shape["resized"] += 1 + per_method.append((label, "resized", o_size, h_size)) + continue + shape["aligned"] += 1 + shifted = 0 + for insn in orig.decode(o_off): + pos = h_info["insns_off"] + (insn["off"] - o_info["insns_off"]) + byte = hard.d[pos] + pairs[insn["op"]][byte] += 1 + total_slots += 1 + shifted += 1 + per_method.append((label, "aligned", o_size, shifted)) + + # --- aggregate into a table ----------------------------------------- + rows = [] + undetermined = [] + conflicts = 0 + for op in sorted(pairs): + cands = pairs[op] + ranked = sorted(cands.items(), key=lambda kv: (-kv[1], kv[0])) + samples = sum(cands.values()) + if len(ranked) == 1: + verdict = "high" + else: + verdict = "conflict" + conflicts += 1 + rows.append({ + "orig": op, + "orig_hex": "0x%02x" % op, + "name": dexutil.OP_NAMES.get(op, "op_%02x" % op), + "samples": samples, + "candidates": ["0x%02x" % b for b, _n in ranked], + "candidate_counts": {"0x%02x" % b: n for b, n in ranked}, + "verdict": verdict, + }) + seen = set(pairs) + for op in sorted(set(dexutil.OP_NAMES) - seen): + undetermined.append({ + "orig": op, "orig_hex": "0x%02x" % op, + "name": dexutil.OP_NAMES[op], + "reason": UNREACHABLE.get(op) or + "the original never emits this opcode, so there is no " + "known-plaintext instance to read a substitution from", + }) + + # reverse direction: a private byte claimed by two different originals + reverse = collections.defaultdict(set) + for op, cands in pairs.items(): + if len(cands) == 1: + reverse[next(iter(cands))].add(op) + non_injective = {("0x%02x" % b): sorted("0x%02x" % o for o in ops) + for b, ops in reverse.items() if len(ops) > 1} + + # --- run-level verdict ---------------------------------------------- + comparable = shape["aligned"] + decided = len(rows) + notes = [] + if comparable == 0: + verdict = "not-applicable" + if shape["stripped"]: + notes.append( + "%d method(s) lost their code_item: the hardened build moved the " + "bodies out of the dex. There is no byte stream to align, so " + "opcode differencing cannot start -- this is the extraction/VMP " + "shape, and the differential route stops here." % shape["stripped"]) + if shape["resized"]: + notes.append( + "%d method(s) kept a body but changed its length. Boundaries " + "cannot be projected across a length change; a private opcode " + "stream with its own length table is not reachable this way." + % shape["resized"]) + if not notes: + notes.append("no method pair was comparable; inspect --json per-method rows") + elif decided == 0: + verdict = "not-applicable" + notes.append("bodies align by length but no opcode pair was accumulated") + else: + conflict_ratio = conflicts / float(decided) + worst_share = max((len(v) for v in non_injective.values()), default=0) + if worst_share >= 3: + # The stub shape. Every original opcode maps to the *same* private + # byte, so `conflicts` is zero and the per-opcode reading looks + # perfect -- the only thing that gives it away is the reverse + # direction. This is why the map is checked for injectivity at all. + verdict = "not-usable" + notes.append( + "%d private byte(s) are claimed by up to %d original opcodes each. " + "No injective relabelling can do that: it is the signature of method " + "bodies replaced by one common stub, and any substitution read off " + "such a pair describes the stub, not a private opcode space." + % (len(non_injective), worst_share)) + elif conflict_ratio > 0.30: + verdict = "not-usable" + notes.append("%d of %d opcode(s) saw more than one candidate byte; a " + "stable map cannot be read off this pair" + % (conflicts, decided)) + elif conflict_ratio > 0.05 or non_injective: + verdict = "partially-usable" + if conflicts: + notes.append("%d of %d opcode(s) saw more than one candidate byte; the " + "substitution is not a pure opcode relabelling (or the " + "alignment slipped) -- treat those rows as unknown" + % (conflicts, decided)) + if non_injective: + notes.append("%d private byte(s) are claimed by more than one original " + "opcode (%d-way at worst), which no injective relabelling " + "can produce" % (len(non_injective), worst_share)) + else: + verdict = "usable" + + # --- report ---------------------------------------------------------- + print("== differential hardening: %s (%s) vs %s (%s) ==" + % (args.original, orig_entry, args.hardened, hard_entry)) + print(" aligned=%d resized=%d stripped=%d missing=%d added=%d" + % (shape["aligned"], shape["resized"], shape["stripped"], + shape["missing"], shape["added"])) + print(" aligned slots compared: %d" % total_slots) + print(" verdict: %s" % verdict.upper()) + for n in notes: + print(" ! %s" % n) + if not args.quiet: + print("\n-- candidate opcode map (original -> private byte) --") + table_rows = [] + for r in rows: + table_rows.append((r["orig_hex"], r["name"], r["samples"], + ",".join(r["candidates"]), r["verdict"])) + if table_rows: + print(_fmt_table(table_rows, ["orig", "name", "samples", "candidate(s)", "verdict"])) + else: + print(" (none: no aligned instruction pairs)") + if undetermined: + print("\n-- undetermined opcodes (%d) --" % len(undetermined)) + for u in undetermined: + print(" %s %-24s %s" % (u["orig_hex"], u["name"], u["reason"])) + if non_injective: + print("\n-- non-injective private bytes --") + for b, ops in sorted(non_injective.items()): + print(" %s <- %s" % (b, ",".join(ops))) + if args.verbose: + print("\n-- per-method --") + rows2 = [(m[0], m[1], m[2], m[3]) for m in per_method] + print(_fmt_table(rows2, ["method", "state", "orig_size", "compared"])) + + if args.json: + payload = { + "original": args.original, + "hardened": args.hardened, + "verdict": verdict, + "shape": dict(shape), + "aligned_slots": total_slots, + "map": rows, + "undetermined": undetermined, + "non_injective": non_injective, + "notes": notes, + } + with open(args.json, "w", encoding="utf-8") as fh: + json.dump(payload, fh, indent=2, sort_keys=True) + print("\nwrote %s" % args.json) + return 0 if verdict in ("usable", "partially-usable") else 1 + + +# --------------------------------------------------------------------------- +# emit-smali +# --------------------------------------------------------------------------- + +def _load_table(path): + """Build private-byte -> standard-opcode from a compare/simulate JSON table. + + Two shapes are accepted, because the two producers differ: `simulate` writes + its ground truth as {"map": {"0x90": "0xe1"}} (original -> private), while + `compare` writes a list of rows carrying candidates and a verdict. Only + rows whose verdict is `high` are usable -- a conflict row would have to pick + a winner, and guessing here is how a wrong table gets laundered into a + confident-looking disassembly. + """ + with open(path, encoding="utf-8") as fh: + doc = json.load(fh) + priv2std = {} + skipped = [] + raw_map = doc.get("map") + if isinstance(raw_map, dict): + for orig_hex, priv_hex in raw_map.items(): + priv2std[int(priv_hex, 0)] = int(orig_hex, 0) + return priv2std, skipped, doc + for row in raw_map or []: + orig = row.get("orig") + cands = row.get("candidates", []) + verdict = row.get("verdict") + if orig is None or not cands: + continue + if verdict != "high" or len(cands) != 1: + skipped.append((orig, verdict, cands)) + continue + priv2std[int(cands[0], 0)] = int(orig) + return priv2std, skipped, doc + + +def _restore(data, priv2std): + """Rewrite private opcode bytes to their standard values, in place. + + Boundaries come from the table itself: once a byte is mapped back to a + standard opcode, the standard length table applies to that instruction and + the walk can step to the next one. An unmapped byte ends the walk for that + method -- and the count of such events is the honest measure of how much of + the body the table actually covers. + """ + dex = dexutil.Dex(bytes(data)) + out = bytearray(data) + unknown = collections.Counter() + resolved = 0 + for _key, code_off in _methods(dex).items(): + if not code_off: + continue + info = dex.code_info(code_off) + pos = info["insns_off"] + end = pos + info["insns_size"] * 2 + while pos < end: + byte = data[pos] + std = priv2std.get(byte) + if std is None: + if byte == 0x00: + # a genuine nop or a payload header: the payload's own layout + # is data, not opcodes, so its leading byte stays as it is + std = 0x00 + else: + unknown[byte] += 1 + break + out[pos] = std + resolved += 1 + units = dexutil.insn_units(std, data, pos, end) + if units < 1 or pos + units * 2 > end: + break + pos += units * 2 + return out, unknown, resolved + + +def cmd_emit_smali(args): + _require_dexutil() + with open(args.dex, "rb") as fh: + data = bytearray(fh.read()) + priv2std, skipped, _doc = _load_table(args.table) + restored, unknown, resolved = _restore(data, priv2std) + if not args.keep_header: + dexutil.fix_dex_header(restored) + dex = dexutil.Dex(bytes(restored)) + problems = dex.check() + + lines = [] + lines.append("; smali skeleton -- generated by vmp_diff_harness.py emit-smali") + lines.append("; source dex : %s" % args.dex) + lines.append("; table : %s (%d private bytes mapped, %d entries skipped as " + "non-high-confidence)" % (args.table, len(priv2std), len(skipped))) + lines.append("; instructions restored: %d ; unmapped-opcode stops: %d" + % (resolved, sum(unknown.values()))) + lines.append(";") + lines.append("; THIS IS A READING SKELETON, NOT ASSEMBLABLE SMALI.") + lines.append("; Instruction boundaries and opcode names are restored from the") + lines.append("; differential table; the operands below are rendered by the standard") + lines.append("; dex format. Register numbers, field/method/string indices and any") + lines.append("; register re-allocation the engine performed are NOT recovered --") + lines.append("; no map for those exists in this method.") + if unknown: + lines.append("; UNMAPPED private bytes encountered (%d kind(s)):" + % len(unknown)) + for byte, n in sorted(unknown.items()): + lines.append("; 0x%02x x%d -- body truncated at the first occurrence" % (byte, n)) + lines.append("") + for p in problems: + lines.append("; [structure] %s" % p) + if problems: + lines.append("") + + classes = 0 + methods_out = 0 + for i in range(dex.header["class_defs_size"]): + off = dex.header["class_defs_off"] + i * 32 + class_type = dex.type_(dexutil.u32(dex.d, off)) + if not _class_matches(class_type, args.cls): + continue + classes += 1 + super_idx = dexutil.u32(dex.d, off + 8) + access = dexutil.u32(dex.d, off + 4) + lines.append("# access_flags=0x%x" % access) + lines.append(".class %s%s" % (_flags(access, CLASS_FLAGS) + " " + if _flags(access, CLASS_FLAGS) else "", class_type)) + if super_idx: + lines.append(".super %s" % dex.type_(super_idx)) + source_idx = dexutil.u32(dex.d, off + 16) + if source_idx: + lines.append('.source "%s"' % dex.string_safe(source_idx)) + lines.append("") + for _section, _idx, _cls, name, desc, acc, code_off in _methods_with_acc(dex, off): + if args.method and name != args.method: + continue + mflags = _flags(acc, METHOD_FLAGS) + head = ".method %s%s%s" % (mflags + " " if mflags else "", name, desc) + if not code_off: + lines.append(head) + lines.append(" # no code_item (abstract or native)") + lines.append(".end method") + lines.append("") + methods_out += 1 + continue + info = dex.code_info(code_off) + listing, clean, expected = dex.decode_all(code_off) + lines.append("# ---- %s%s registers=%d insns_size=%d %s" + % (name, desc, info["registers"], info["insns_size"], + "" if clean else "DECODE-DID-NOT-END-CLEANLY")) + lines.append(head) + lines.append(" .registers %d" % info["registers"]) + for insn in listing: + text = dex.describe(insn) + body = text.split(": ", 1)[1] if ": " in text else text + lines.append(" # 0x%04x %s" + % (insn["off"] - info["insns_off"], body)) + lines.append(".end method") + lines.append("") + methods_out += 1 + + text = "\n".join(lines) + "\n" + if args.out: + with open(args.out, "w", encoding="utf-8") as fh: + fh.write(text) + print("wrote %s (%d classes matched, %d methods)" % (args.out, classes, methods_out)) + else: + sys.stdout.write(text) + return 0 + + +# --------------------------------------------------------------------------- +# cli +# --------------------------------------------------------------------------- + +def main(argv=None): + parser = argparse.ArgumentParser( + prog="vmp_diff_harness.py", + description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter) + sub = parser.add_subparsers(dest="cmd") + + p = sub.add_parser("build", help="compile the labelled opcode-coverage fixture", + description="Compile a fixture dex whose every instruction is " + "known and labelled. Needs an explicit toolchain.") + p.add_argument("--out", required=True, help="output directory") + p.add_argument("--build-tools", default=None, + help="directory holding d8 and aapt2 (not guessed)") + p.add_argument("--javac", default=None, help="path to javac (default: PATH)") + p.add_argument("--min-api", type=int, default=26, + help="d8 --min-api (26 keeps invoke-custom/invoke-polymorphic " + "instead of desugaring them away); default 26") + p.add_argument("--desugar", action="store_true", + help="allow d8 desugaring (default: --no-desugaring)") + p.add_argument("--with-big", action="store_true", default=True, + help="also generate the bulk fixture (wide registers, long jump); " + "default on") + p.add_argument("--no-big", dest="with_big", action="store_false", + help="skip the bulk fixture") + p.add_argument("--n-int", type=int, default=300, help="int locals in the wide frame") + p.add_argument("--n-long", type=int, default=140, help="long locals in the wide frame") + p.add_argument("--n-obj", type=int, default=160, help="object locals in the wide frame") + p.add_argument("--goto16", type=int, default=200, + help="statements in the long-jump loop body") + p.add_argument("--apk", action="store_true", + help="also pack classes.dex into a minimal unsigned APK via aapt2") + p.set_defaults(func=cmd_build) + + p = sub.add_parser("audit", help="report opcode coverage of a dex", + description="Count decoded opcodes and report the gap against " + "the format table, with the reason each gap exists.") + p.add_argument("dex", nargs="+", help="dex or APK to audit") + p.add_argument("--json", default=None, help="write the report as JSON here") + p.set_defaults(func=cmd_audit) + + p = sub.add_parser("compare", help="align original vs hardened dex; emit a candidate opcode map", + description="Project the original's instruction boundaries onto the " + "hardened body and read the opcode substitution off the " + "alignment. Reports a run-level verdict so a shape the " + "method cannot handle is said out loud instead of " + "producing a plausible map.") + p.add_argument("original", help="the pre-hardening dex (or APK)") + p.add_argument("hardened", help="the post-hardening dex (or APK)") + p.add_argument("--json", default=None, help="write the full report here") + p.add_argument("--quiet", action="store_true", help="suppress the map listing") + p.add_argument("--verbose", action="store_true", help="also print per-method rows") + p.set_defaults(func=cmd_compare) + + p = sub.add_parser("simulate", help="forge a hardened dex from a known private table", + description="Local fixture that makes compare() falsifiable: relabel " + "every opcode through a known bijection, then require " + "compare to re-derive that table. This is NOT a hardening " + "engine -- it preserves instruction lengths, which a real " + "one need not do.") + p.add_argument("dex", help="the dex to relabel") + p.add_argument("--out", required=True, help="where to write the simulated dex") + p.add_argument("--mode", choices=("relabel", "stub", "shrink"), default="relabel", + help="relabel: private opcode bytes, lengths preserved (the shape " + "the differential can actually crack); stub: every body " + "replaced by an equal-length return-void fill (the extraction " + "shape -- compare must refuse it); shrink: rewrite insns_size " + "to drive compare's length-mismatch detector (writes a " + "deliberately inconsistent image)") + p.add_argument("--seed", type=int, default=1, help="bijection seed") + p.add_argument("--table", default=None, + help="also write the ground-truth table here (orig -> private)") + p.add_argument("--keep-header", action="store_true", + help="do not recompute checksum/signature (default: recompute)") + p.set_defaults(func=cmd_simulate) + + p = sub.add_parser("emit-smali", help="render private-opcode bodies into a smali skeleton", + description="Restore opcode bytes through a differential table and " + "render method bodies as an annotated smali reading " + "skeleton. The output is NOT assembliable.") + p.add_argument("dex", help="the hardened dex") + p.add_argument("--table", required=True, help="table JSON from compare/simulate") + p.add_argument("--class", dest="cls", default=None, + help="only this class (Lpkg/Name; or pkg.Name)") + p.add_argument("--method", default=None, help="only this method name") + p.add_argument("--out", default=None, help="output file (default: stdout)") + p.add_argument("--keep-header", action="store_true", + help="do not recompute the restored dex header") + p.set_defaults(func=cmd_emit_smali) + + args = parser.parse_args(argv) + if not getattr(args, "cmd", None): + parser.print_help() + return 2 + return args.func(args) + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tests/benchmark.md b/tests/benchmark.md new file mode 100644 index 0000000..1d8688d --- /dev/null +++ b/tests/benchmark.md @@ -0,0 +1,55 @@ +# Benchmark set — the regression baseline for this skill + +This file is a **test matrix**, not a sample collection. It answers one question per row: *when an +agent meets this shape of target, does the route this repository documents actually work, and what +is the strength of the evidence?* Every claim in the `Result` column must be reproducible from the +`Target` column plus the named script — nothing here is a summary of what "should" happen. + +**Samples are never committed.** Each row names a public source; download it into `tools/_work/bench/` +(ignored by git) before running the row. Record the sample hash when you run it, because two rows on +different sample revisions are not comparable. Hashes used in this pass: + +``` +UnCrackable-Level1.apk 66651 B 1da8bf57d266109f… +UnCrackable-Level2.apk 901022 B 4c7980ef1f6cc295… +UnCrackable-Level3.apk 1460555 B 6827f776b7d28443… +r2pay-v1.0.apk 4293620 B 850d7c98bed8df53… +``` + +Strength labels follow the repository's own three-tier rule (`references/long-task-discipline.md` +reserves `observed` for a claim with an exact command and its output behind it): + +- `observed` — a command was run and its output is quoted in the evidence file. +- `inferred` — reasoned from documented mechanism or from a neighbouring measurement. +- `unverified` — assumed, or reported by someone else and not reproduced here. + +| # | Dimension | Target (public source) | Scripts / documents the row exercises | Result | Strength | Evidence | +|---|---|---|---|---|---|---| +| B1 | Equal-length dex patch + repack + re-sign installability (G2/G4) | MASTG `UnCrackable-Level1.apk` | `dex_patch_bytes.py`, `repack.py`, `byte-level-patching.md`, `repack-and-sign.md` | **The route works end to end, and the change is visible on screen.** 2 bytes of the first root-check branch in `MainActivity.onCreate` (`0x9d0: 39000e00` -> `00000000`) patched inside a 5528-byte dex; the whole-file diff is those 4 bytes plus bytes 8..32, and the written file self-verifies (`adler32(d[12:])` and `sha1(d[32:])` both reproduce, proving signature-before-checksum order). `repack.py` produced a STORED + 4-byte-aligned `resources.arsc`, apksigner reported v1+v2+v3 all true, the root install path succeeded with the on-device `base.apk` sha256 equal to the local build, and the blocking `Root detected!` dialog is gone from the screenshots. A zero-change control through the identical pipeline still shows it | observed | `EXTENSION-benchmark-l1-l3.md` §2 | +| B2 | Anti-debug + self-destruct race (patch before the check fires) | MASTG `UnCrackable-Level3.apk` | `spawn_patch_detach.py`, `hook_patch_only.js`, `dynamic-frida.md` | **Split: the tooling works, the patch does not.** Time-to-death with a probe attached is 5.34 s. After an ordering fix (resume -> wait for `PATCHED` -> detach) the probe wrote `libfoo.so` base+0x2ff0/+0x2ff8/+0x3080 to `mov w0,#0 ; ret` 0.04-0.05 s after resume and the writes survived the detach — the mechanism is real. The patch still does not stop the death: the patched run **and** a read-and-write-back control both die with `SIGABRT` at `libfoo.so (goodbye()+12)` 3.22 s after resume, so neutralising the terminate routine plus both of its branch inputs is insufficient; the region is a dense cluster of `mov w0,#imm ; ret` stubs and locating the real second site needs a full `.text` disassembly. The alternative ordering ("hold paused until `PATCHED`") cannot work — a frozen process has no libraries mapped, so `Process.findModuleByName` never returns (0.11 s after resume it appears). Independently, the pristine original already reports tamper on this device, so the row's assumed clean baseline does not exist here | observed (tooling) / unverified (a surviving target) | `EXTENSION-benchmark-l1-l3.md` §3 | +| B3 | Extraction-shell skeleton detection (`trivial-body` routing) | self-built 11-variant skeleton set from the `r2pay-v1.0.apk` dex, with a zero-change control | `dex_dump_validate.py`, `advanced-unpacking.md` | **The "stub% threshold" claim this skill used to make is wrong.** `stub%` is *bimodal*, not thresholded: a real skeleton reads ≈100% (`allstub` 100.0%) while a control reads 1.9%, and the shapes real shells actually use are invisible — nop-cleared bodies 0.0% (below the control), throw stubs 1.9% (indistinguishable), partial extraction at 25/50/75% 1.7/1.7/2.1%. `advanced-unpacking.md`'s original "tens of percent means a skeleton" had no support and was deleted. Three script defects were found and fixed (all-nop bodies ranked as real; a legal minimal body mislabelled malformed; the ranking pointing at a 25%-extracted image as "most likely original" — caught by independent review). The new `minimal-form` counter *does* track the extraction ratio monotonically (1299/2524/3736/4946) and now participates in the ranking | observed | `EXTENSION-extraction-shell-bench.md` §1–3 | +| B4 | Dex-VMP boundary: refuse-to-repack and downgrade the deliverable | DexPatcher `assets/ezAndroid.apk` | `advanced-unpacking.md`, `java2c-and-jni-sinking.md`, `lsposed_scaffold.py`, `frida_rpc_serve.py` | **The sample is not a Dex VMP — it is JNI sinking**: 7 `native` declarations, each with a matching exported `Java_*` symbol, no `JNI_OnLoad` / `RegisterNatives` symbol, 34,566 instructions with zero structural errors under the official `dexdump`. A zero-change repack succeeded (v2 signing, alignment, `classes.dex` CRC verified). Static criteria **can** rule a dex-VMP *out*; they cannot rule runtime-applied hardening out. The first verdict here was the opposite — a hand-written opcode table reported 1393/22424 undecodable bodies and called it a VMP; `dexdump` showed the table was missing an arm64 branch opcode, and that rule (never declare VMP without an official disassembler's failure count) is now in the reference. Installation / launch / refusal remain untested | observed (static + repack) / unverified (install, launch) | `EXTENSION-extraction-shell-bench.md` §4 | +| B5 | Java2C recognition (do not misdiagnose as an extraction shell) | self-built with `amimo/dcc` (`--no-build`) + real JNI samples | `java2c_probe.py`, `java2c-and-jni-sinking.md` | Native-declaration density separates the shapes by ~2000x: **0.03–0.04%** on three real JNI samples vs **82.76%** on a Java2C-shaped fixture, and `Java_*` symbols matched dex native-method counts 1:1 per ABI (2:2, 3:3). **No Dex-to-C compiler output was ever built** (no NDK, no host clang, no device clang, WSL unavailable), so every Java2C-specific criterion is inferred while the JNI-sinking side is measured. A symbol search fails silently on real Java2C: dcc emits `-fvisibility=hidden`, and C++ `jni.h` inlines `RegisterNatives` so it leaves no symbol | observed (density, JNI sinking) / inferred (Java2C-specific) | `EXTENSION-java2c.md` | +| B6 | Native tracing: does `Stalker.exclude()` change the outcome? | MASTG `UnCrackable-Level1.apk`, module `libc.so` followed on the main thread for 6 s | `stalker_trace.js`, `native-dbi-and-deobfuscation.md` | **Split result.** A no-follow baseline survives; the same follow with `excludeModules: []` **kills the process**; with 20 system modules excluded it survives — so exclusion is what keeps the target alive (`observed`, with the baseline arm ruling out "frida killed it anyway"). But the treatment arm still reported `blocks=0 blk=0 calls=0`: **exclusion does not fix the zero-event trace**, which partially contradicts the community framing that lists "no events" among its symptoms. The pass also fixed two harness traps that manufacture a false zero-event result (a load-time default follow, and a suspended spawn) and verified a new zero-event `WARN`. No real OLLVM trace was produced | observed (survival, delivery) / unverified (the multiplier, the cause of zero delivery) | `EXTENSION-stalker-exclude.md` | +| B7 | Native certificate pinning | CyReverse Cronet sample | `protocol-reverse.md` | not run this pass — the CyReverse Cronet sample needs a gradle+NDK build this machine cannot do | unverified | — | +| B8 | Schemaless protobuf decoding | synthetic fixtures + the official runtime + a real AndroidX DataStore container | `protobuf_decode_raw.py`, `protocol-reverse.md` | 26/26 built-in fixtures, **21/21 against the official protobuf runtime** (re-encode byte-identical to `SerializeToString()`), 8/8 framing checks. A real 81-byte DataStore file round-tripped byte-exact and an edited value re-encoded (81->80 B, length prefix recomputed) was read back by `datastore_inject.py`. Both traps reproduced on real bytes: `0801120178` is a valid nested message *and* a valid packed array; a non-presence `0` is byte-identical to "never assigned". Four script defects were found by these checks and fixed | observed | `EXTENSION-protobuf-raw.md` | +| B9 | Split APK / App Bundle handling | two real split sets pulled from a device (8 members = base + 7 density splits each carrying its own `resources.arsc`; 9 members = base + ABI split with only `lib/` + 7 density splits with 40-byte placeholder tables) | `repack.py` (split mode), `split-apk.md` | Both branches of the decision ran on real sets: after `resign` every member shares one certificate with v1+v2+v3 all true; `merge` **correctly refused** the set whose members carry their own `resources.arsc` (exit 1, before writing anything) and **succeeded** on the code/native-only set, folding 2 `.so` files into `base`. The single-APK legacy path still passes its regression (`--dexdir` replacement and a zero-change round trip). Signing has no `uber-apk-signer` on this host; the measured call is `zipalign -p -f 4` then `apksigner.jar --v1 --v2 --v3`, and the order cannot be reversed (`[-124]`). `adb install-multiple` and launch confirmation were **not** executed | observed (analyze/resign/merge) / unverified (install, launch) | `EXTENSION-split-apk.md` | +| B10 | Kernel-side syscall answer forging | KernelSU/APatch module template, host toolchain, device kernel | `kernel-and-environment-hardening.md`, `kernelsu_syscall_mask.py` | The generator produced an 11-file userspace module skeleton plus KPM/LKM/eBPF templates and `verify` reports 0 problems — but **no kernel-side artefact was compiled or loaded**: the `` kernel is 4.14.186 (eBPF needs 5.10+), the host has no `aarch64-none-elf-gcc` / `make` / `ndk-build`, and there is no kernel source. Documentation correction with teeth: **an ordinary KernelSU module is a userspace module and structurally cannot change a syscall return value** | observed (generation) / unverified (kernel side) | `EXTENSION-kernel-weapons.md` | +| B11 | JNI dynamic registration discovery (`RegisterNatives` with no `Java_*` symbol) | a real system library + CyReverse `plugin/` | `java2c-and-jni-sinking.md`, `java2c_probe.py` | A real dynamically-registering library contains **zero `RegisterNatives` symbols** (the C++ `jni.h` inlines the call), so symbol-based discovery fails structurally. The combination that did fire on a real library is "**exports `JNI_OnLoad` and zero `Java_*`**" | observed | `EXTENSION-java2c.md` | + +## How to read a row + +`Result` states what actually happened, including negative outcomes — "the route worked", "the route +worked only with `Stalker.exclude()`", "the tool refused to run at all". A negative result that is +recorded is worth more than a positive one that is assumed, because the next agent can route around +it without re-paying the cost. + +A row whose `Strength` is `unverified` is a statement about *this repository's evidence*, not about +the mechanism's truth. It means: nobody has yet put a command and its output behind it here. + +**Rows that changed a conclusion this repository had already recorded.** This is the outcome the +matrix exists to produce, so it is listed rather than buried: B3 (the trivial-body "threshold" became +a measured bimodal result), B4 (a VMP verdict was withdrawn after an official disassembler disagreed +with a hand-written opcode table), B6 (exclusion turns out to keep the process alive but does not fix +zero-event delivery, contradicting the community framing), and B1 (an earlier "`repack.py` has no +apksigner signing path" conclusion expired mid-pass when a second writer added one).