diff --git a/README.md b/README.md index a336f48..bd911d0 100644 --- a/README.md +++ b/README.md @@ -458,6 +458,7 @@ Linux; where a snippet is POSIX-only it is labelled. Nothing here assumes a Unix | Python 3.9+ | all scripts | | **`droidasc`** (ASC) (**optional but strongly recommended — install this first**) | whole-APK cross-reference index: `findrefs` / `listclass` / `getclass` / `getmanifest`. One `pip install droidasc`, no JVM, no SDK, no index build. Turns "which of N thousand classes mentions this string" into a sub-second query, and it is the only route to a class whose name R8 mangled. **This is the tool an agent should reach for before any full decompile** — see `skills/apk-reverse/references/toolchain.md` §droidasc (ASC) — ask an APK "who references this?", in one query | | `ddc` (optional but strongly recommended) | single-binary dex→Java decompiler with query subcommands (`info`, `findrefs`, `strings --with-locations`, per-class decompile). No JVM. **Reads** what ASC **locates**; also reports package identity reliably — see `skills/apk-reverse/references/toolchain.md` §ddc — dex-to-Java with query subcommands (worth adopting) | +| **`dexsql`** (optional but strongly recommended) | single-binary **SQL over the whole dex**, plus the APK manifest and `resources.arsc`, plus an MCP server. Its class-definition set matched `rasc`/`droidasc`/`ddc` exactly on four archives, and it answers what the other two cannot express: callers of a callee, field accesses, numeric constants, debug lines, annotations, hiddenapi flags. Reach for it when the question is a **join or an aggregate** — see `skills/apk-reverse/references/dexsql-sql-query-layer.md` | | `baksmali` / `smali` + `dexlib2` jars | disassembly, assembly, surgical patching | | JDK (`javac`, `java`) | building/running the dexlib2 patcher; also provides `keytool`/`jarsigner` | | Android SDK build-tools (`aapt`, `zipalign`, `apksigner`) | manifest info, alignment, signing. **`apksigner` is the signer to use** — `jarsigner` rewrites the archive and breaks the alignment Android R+ requires | diff --git a/README.zh-CN.md b/README.zh-CN.md index 2ee9d0c..aeb8fd1 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -375,6 +375,7 @@ export APKREV_TOOLS=: # POSIX | Python 3.9+ | 所有脚本 | | **`droidasc`**(ASC)(**可选,但强烈建议——先装这个**) | 整 APK 交叉引用索引:`findrefs` / `listclass` / `getclass` / `getmanifest`。一句 `pip install droidasc`,无 JVM、无 SDK、无需建索引。把「这几千个类里哪一个提到过这个字符串」变成亚秒级查询,也是找到名字被 R8 混淆过的类的唯一路线。**这是 agent 在做任何完整反编译之前应当伸手去拿的工具** —— 见 `skills/apk-reverse/references/toolchain.md` §droidasc (ASC) —— 一次查询就能问一个 APK「谁引用了这个?」 | | `ddc`(可选,但强烈建议) | 单文件 dex→Java 反编译器,带查询子命令(`info`、`findrefs`、`strings --with-locations`、按类反编译)。无 JVM。**读** ASC 所**定位**的东西;也能可靠地报告包身份 —— 见 `skills/apk-reverse/references/toolchain.md` §ddc —— 带查询子命令的 dex 转 Java(值得采用) | +| **`dexsql`**(可选,但强烈建议) | 单文件,**用 SQL 查询整个 dex**,外加 APK manifest 与 `resources.arsc`,并内置 MCP server。它的类定义集合在四个归档上与 `rasc`/`droidasc`/`ddc` 完全一致,而且能回答另两个表达不了的问题:某被调方法的调用者、字段访问、数值常量、调试行表、注解、hiddenapi 标志。当问题是**连接或聚合**时用它 —— 见 `skills/apk-reverse/references/dexsql-sql-query-layer.md` | | `baksmali` / `smali` + `dexlib2` jars | 反汇编、汇编、外科式打补丁 | | JDK(`javac`、`java`) | 构建/运行 dexlib2 补丁器;同时提供 `keytool`/`jarsigner` | | Android SDK build-tools(`aapt`、`zipalign`、`apksigner`) | manifest 信息、对齐、签名。**`apksigner` 才是该用的签名器** —— `jarsigner` 会重写归档并破坏 Android R+ 所要求的对齐 | diff --git a/docs/tool-verification/EXTENSION-dexsql.md b/docs/tool-verification/EXTENSION-dexsql.md new file mode 100644 index 0000000..f94b9a9 --- /dev/null +++ b/docs/tool-verification/EXTENSION-dexsql.md @@ -0,0 +1,149 @@ +# Extension verification — dexsql, the SQL query layer (parity, decode, cache, MCP) + +Covers the pass that adopted `dexsql` as a third indexer beside `rasc`/`droidasc`/`ddc` and +registered it as the `dex_query_sql` capability. Strength labels as this directory's README defines +them. + +Reference environment: Windows 10.0.26100, PowerShell 7, Python 3.14.0, no device attached. The tool +is a release download (`dexsql-windows-amd64.exe`, v0.1.0, 21,365,760 B, sha256 +`f354ebcf715de9a66ef8ebe08b2b36807ab58e25cd5a1c0812e497fc1fcfc448`) — no build step, no Python, no +JVM, no Android SDK. + +**Headline: the class-definition parity check passes on all four archives — identical to `rasc`, +`droidasc` and `ddc`, zero differences either way — and the string cross-reference agrees site for +site. Three claims came back *smaller* than they could have been sold: the persistent cache is not a +speed switch, `decode` emits smali rather than Java, and the SQL match semantics are not `findrefs`'s. +The MCP surface was exercised to a tool listing, not end to end from a client.** + +--- + +## 1. Class-definition parity — observed + +The check that decides whether a new indexer may be trusted for location questions at all, and the +same comparison `rasc_build.py --verify` performs between `rasc` and `droidasc`. Each tool prints +class identity in its own shape, so each output is parsed per format and normalised to a DEX +descriptor before comparing — a single regex over all four outputs measures the regex, not the tools +(this pass's own first defect: an unescaped `$` in a descriptor pattern silently dropped classes whose +names contain `$` or non-ASCII characters, and reported `droidasc` as 38 classes where it returns 647). + +| Archive | dexsql | rasc | droidasc | ddc | verdict | +|---|---|---|---|---|---| +| 1.4 MB MASTG challenge | 1,396 | 1,396 | 1,396 | 1,396 | identical | +| 4.3 MB hardened | 647 | 647 | 647 | 647 | identical | +| 38.7 MB hardened | 4 | 4 | 4 | 4 | identical | +| 34.8 MB app | 30,768 | 30,768 | 30,768 | 30,768 | identical | + +Zero differences in either direction on every row. The two small counts are the hardened targets, not +the tools: the business dex is encrypted inside a shell and is in no DEX any of the four can read. + +Full class listing, best of 3, stdout redirected to a file — the same shape for all four tools: + +| Archive | dexsql | rasc | droidasc | ddc | +|---|---|---|---|---| +| 1.4 MB / 1,396 classes | 0.035 s | 0.028 s | 0.149 s | 0.109 s | +| 4.3 MB / 647 classes | 0.031 s | 0.025 s | 0.139 s | 0.099 s | +| 38.7 MB / 4 classes | 0.040 s | 0.021 s | 0.156 s | 0.092 s | +| 34.8 MB / 30,768 classes | 0.136 s | 0.055 s | 0.200 s | 0.156 s | + +The `rasc` and `droidasc` columns reproduce this repository's earlier figures for the same 34.8 MB +archive (0.048 s / 0.190 s), which is what makes the new column comparable rather than merely new. +`dexsql` is 1.5× faster than the Python indexer and 2.5× slower than the Rust one, so **it does not +replace `rasc` for a plain class list** — it answers questions the other two cannot express. + +## 2. Cross-reference parity, and the semantics that differ — observed + +``` +$ dexsql query --input <1.4 MB challenge> --sql \ + "SELECT DISTINCT method_class, method_name FROM method_strings WHERE value LIKE '%http%'" +Landroid/support/v4/content/res/TypedArrayUtils; | hasAttribute +Landroid/support/v4/text/util/LinkifyCompat; | addLinks +Landroid/support/v4/util/PatternsCompat; | +``` + +`rasc findrefs … string http` and `droidasc findrefs … string http` return exactly those three sites, +and `ddc findrefs` agrees. **The matching semantics differ, and the difference is a trap worth +recording:** `findrefs` matches a *substring*, while SQL needs `LIKE '%…%'`. `WHERE value = 'http'` +returns **zero rows** and is not a broken query — no method in that archive holds a literal that *is* +`http`. A first pass read that empty result as a tool defect; the control was the same query with +`LIKE`, which reproduced ASC's answer exactly. + +## 3. `decode` against `ddc disasm` — observed + +``` +$ dexsql decode --input --method 'Lsg/vantagepoint/uncrackable3/MainActivity;->verifyLibs()V' +.method private verifyLibs()V + .registers 14 + const-string v0, "UnCrackable3" + new-instance v1, Ljava/util/HashMap; + ... +``` + +The instruction sequence matches `ddc disasm sg.vantagepoint.uncrackable3.MainActivity.verify` +line for line on that method. `decode` emits **smali, not Java** — that is a feature for the +bytecode-level decision `dex-patching.md` requires, and a limitation when the question is Java-level, +where `ddc getclass` still owns it. A missing class exits 1 with `Error: not_found: selected file, +method, or class not found`. + +## 4. The persistent cache is not a speed switch — observed + +``` +$ dexsql prepare-cache --input <34.8 MB app> --output cache # 0.196 s, writes cache.dxc2 34,481,119 B +$ dexsql query --input --full --sql "SELECT count(*) FROM method_calls" # 0.121 s +$ dexsql query-cache --cache-file cache/cache.dxc2 --sql "SELECT count(*) FROM method_calls" # 0.098 s +$ dexsql query --input --full --sql "" # 0.470 s +$ dexsql query-cache --cache-file cache/cache.dxc2 --sql "" # 0.458 s +``` + +**Under 20%, not an order of magnitude**, because parse-and-build is already 0.12 s on that archive. +On the 38.7 MB sample the cache is a net *loss* (0.071 s from cache against 0.042 s direct). The +honest claim is "hold one dataset across many queries in a long session", not "queries get fast" — +and the cache file is roughly the size of the source archive. + +## 5. Error surface and read-only enforcement — observed + +Every failure exits 1 with a named cause, and writes are refused: + +| Input | Result | +|---|---| +| missing class | `Error: not_found: selected file, method, or class not found` | +| missing file, or a non-dex file | `Error: load_failed at load_inputs: failed to prepare native input cache` | +| unknown column | `Error: invalid_query at execute_sql: unknown column `nope`` | +| unknown relation | `Error: invalid_query at execute_sql: unknown relation `nope`` | +| `DELETE FROM classes` | `Error: SQL statement 1 failed` — the statement is rejected | + +## 6. The MCP surface, exercised to its boundary — observed (listing), unverified (client) + +`dexsql mcp` speaks JSON-RPC over stdio; an `initialize` + `tools/list` handshake returns 13 tools: +`open_dataset`, `list_datasets`, `close_dataset`, `list_tables`, `describe_table`, `query`, `decode`, +`get_manifest`, `get_manifest_components`, `list_resource_files`, `read_resource`, `query_resources`, +plus an embedded English documentation catalog on `resources/list`. Every list-shaped result carries +`next_offset`, and the resource tools read `resources.arsc` and archive entries without extracting +them to disk. Quotas are explicit (`--max-datasets`, `--max-input-bytes`, `--max-dex-bytes`, +`--max-output-bytes`, `--timeout-ms`), and the server's own instructions say a running timeout closes +the affected dataset. **No client session was driven against it in this pass** — the tool list and the +schemas are what is measured. + +## 7. What this pass deliberately did not do + +- No Linux or macOS asset was downloaded or run: the release publishes `linux-amd64`, `linux-arm64` + and `darwin-arm64` binaries and none was exercised. +- No MCP client session: the handshake and tool inventory were read, the tools were not called. +- No patching through `dexsql`: it has no write path, and nothing here claims one. +- No claim about a packed target beyond the parity row: on both hardened archives it returns the same + class count as the other three tools, and the payload is not in any readable DEX. + +## 8. Facts this pass pinned for the next one + +- **A new indexer earns trust by the class-set comparison, not by its feature list.** The parity check + is cheap (four tools, four archives, seconds) and it is the only evidence that separates "answers + location questions correctly" from "parses most of the file". Do it before adopting the next one. +- **Parse each tool's own output format.** The first pass compared four differently-shaped outputs + with one regex and produced two false verdicts — one an unescaped `$`, one a dotted-vs-descriptor + mismatch. A comparison harness is itself a tool that needs a control. +- **`doctor.py` is the registration surface.** Adding `tool:dexsql` plus a `dex_query_sql` capability + makes G2 report the tool instead of guessing, and `INSTALL_HINTS` is where a release download or a + from-source build declares the location this kit's own install step produces, so a machine that + followed the instructions is not reported BLOCKED. +- **The three indexers now have a stated division of labour:** `rasc` locates fastest, `droidasc` is + the Python cross-check, `dexsql` answers joins and aggregates (and carries the resource model). + Writing that down is what keeps the next pass from re-deriving it. diff --git a/docs/tool-verification/README.md b/docs/tool-verification/README.md index 1fef332..9f3cfd3 100644 --- a/docs/tool-verification/README.md +++ b/docs/tool-verification/README.md @@ -87,6 +87,7 @@ written down. | `EXTENSION-desensitization.md` | The leak scanner's rules and exemptions, its four `RESULT=` states and exit codes, the escaping false positives, and **the pass that found its own evidence file leaking 26 strong hits** — the fixture's literal match values, quoted to prove the rules fire | | `EXTENSION-dart-aot-formats.md` | Two string-table claims tested against a real dual-ABI Flutter app: the arm64 packed scheme confirmed against literal tag bytes, and the assumed armv7 `len*2`/UTF-16 form **refuted** — the 32-bit record is `[header u32][byte-count u32le][UTF-8]`, and the extractor's zero is a format mismatch rather than an empty table | | `EXTENSION-detection-pipeline.md` | Naming the check that fires: three arms on a public MASTG target (control alive, probe arm dead in ≈300 ms via `strstr("frida")`, with the `TracerPid=0` versus 4-frida-mappings asymmetry), `svc_scan.py`'s two decoders agreeing on an identical 214-site set, the layered Java-to-svc descent, and the ptrace-free dump boundary on targets whose dex is deflated inside the APK | +| `EXTENSION-dexsql.md` | The SQL query layer adopted as a **third indexer** beside rasc/droidasc/ddc: identical class-definition sets on four archives (zero differences either way), the string xref agreement and the `=` versus `LIKE` match-semantics trap, `decode` against `ddc disasm` line for line, and the persistent cache measured as **not** a speed switch | ## Adding a run diff --git a/skills/apk-reverse/SKILL.md b/skills/apk-reverse/SKILL.md index 6108e32..2e52eef 100644 --- a/skills/apk-reverse/SKILL.md +++ b/skills/apk-reverse/SKILL.md @@ -81,7 +81,7 @@ invisible from the inside, because the weak route still produces output. | Obligation | Do this | Cost of ignoring it | |---|---|---| -| **Orient with an indexer, not an export** | Build the ability to *ask the artifact questions* before reading code: `droidasc findrefs` / `ddc findrefs` (string/type/method → every reference site, sub-second). A decompile reads a class you have **already located** — it is not how you locate it. `jadx` is a readable viewer of last resort, never the source of truth and never the entry point of a recon | hours of `grep` per question asked, over an export you paid for first | +| **Orient with an indexer, not an export** | Build the ability to *ask the artifact questions* before reading code: `droidasc findrefs` / `ddc findrefs` (string/type/method → every reference site, sub-second), and `dexsql query` when the question is a **join or an aggregate** — callers of a callee, methods holding a numeric constant, the debug line table — rather than a class list. A decompile reads a class you have **already located** — it is not how you locate it. `jadx` is a readable viewer of last resort, never the source of truth and never the entry point of a recon | hours of `grep` per question asked, over an export you paid for first | | **Install the missing tool; it is part of the task** | A missing arm64 decompiler is the next step, not a constraint to route around. Ask a human only when installation is genuinely impossible | hours re-derived by hand for output a decompiler gives in minutes | | **Name the gap before spending against it** | State the capability the blocker requires and whether this machine has it, **out loud**. This is a G2 item | designing around a tool that installs in ten minutes | | **Reach for a shipped script before writing one** | Parsing, hashing, alignment and hot-plug probes are already written; a bespoke script in place of `scripts/dex_find_insn.py` is how offsets get **guessed instead of computed** | a wrong offset that decodes cleanly and behaves wrongly | diff --git a/skills/apk-reverse/evidence/capability-matrix.json b/skills/apk-reverse/evidence/capability-matrix.json index 86442eb..c644608 100644 --- a/skills/apk-reverse/evidence/capability-matrix.json +++ b/skills/apk-reverse/evidence/capability-matrix.json @@ -415,6 +415,23 @@ "evidence": ["references/server-api.md", "references/handoff-boundaries.md"], "evidence_not_shipped": ["docs/tool-verification/README.md (repository root, not shipped)"], "limits": "Determining who owns a gate is covered; breaking an authorization the server performs is not. There is no client-side patch that reaches it, and the honest deliverable is the residual: which authority decides, and why the artifact cannot change it." + }, + { + "capability": "dex-sql-query-layer-dexsql", + "group": "dex", + "status": "ok", + "route_measured": true, + "strength": "observed", + "benchmark_rows": [], + "evidence": [ + "references/dexsql-sql-query-layer.md", + "references/toolchain.md", + "scripts/capabilities.py" + ], + "evidence_not_shipped": [ + "docs/tool-verification/EXTENSION-dexsql.md (repository root, not shipped)" + ], + "limits": "Measured on dexsql v0.1.0, Windows amd64 release asset only; no other version or platform was exercised. Class-definition sets were identical to rasc, droidasc and ddc on four archives (1,396 / 647 / 4 / 30,768 classes, zero differences either way), and a string cross-reference query reproduced ASC's hits site for site on one archive. Three boundaries are load-bearing: `decode` emits smali rather than Java, SQL `=` is not `findrefs` (ASC matches a substring, so `LIKE` is the equivalent), and the persistent cache is not a speed switch (under 20% on a 34.8 MB archive, a net loss on a small one). The MCP server was exercised to a tool listing and a resource-model inventory, not end to end from a client." } ] } diff --git a/skills/apk-reverse/evidence/tested-tool-versions.json b/skills/apk-reverse/evidence/tested-tool-versions.json index 6ae33ac..1b19059 100644 --- a/skills/apk-reverse/evidence/tested-tool-versions.json +++ b/skills/apk-reverse/evidence/tested-tool-versions.json @@ -61,6 +61,12 @@ {"name": "lxml", "version": "6.0.2", "strength": "observed"}, {"name": "mitmproxy", "version": "12.2.3", "strength": "observed", "notes": "the library version"} ], + "dex_indexers": [ + {"name": "dexsql", "version": "0.1.0", "probe": "dexsql --version", "strength": "observed", "used_by": "the SQL query layer over dex and APK resources; registered as the dex_query_sql capability", "notes": "Windows amd64 release asset, 21,365,760 bytes, sha256 f354ebcf715de9a66ef8ebe08b2b36807ab58e25cd5a1c0812e497fc1fcfc448, downloaded from the project's release host (no build step). Class-definition sets were compared against the three indexers below on four archives (1,396 / 647 / 4 / 30,768 classes): identical on all four, zero differences in either direction. Full measurements, recipes and boundaries: references/dexsql-sql-query-layer.md."}, + {"name": "rasc (Rust ASC)", "version": "0.1.0", "probe": "rasc --version", "strength": "observed", "notes": "built from source by scripts/rasc_build.py; the class-set comparison quoted above is the same one rasc_build.py --verify performs between rasc and droidasc"}, + {"name": "droidasc (Python ASC)", "version": "0.1.1.post1", "probe": "python -c \"import importlib.metadata as m; print(m.version('droidasc'))\"", "strength": "observed"}, + {"name": "ddc", "version": "0.1.8", "probe": "ddc -V", "strength": "observed", "notes": "single Rust binary, not on PATH here; the measured calls pass it by explicit path"} + ], "proxy": [ {"name": "mitmdump (standalone binary in the local tools directory)", "version": "11.1.2 binary, bundling Python 3.13.1 and OpenSSL 3.4.0", "probe": " --version", "strength": "observed", "notes": "This is NOT the same version as the mitmproxy library above. A local workbench note recorded 12.2.3 against this binary's path; the binary reports 11.1.2. If a captured-flow shape matters, read the version from the binary you actually invoke."} ], @@ -98,6 +104,7 @@ "Kernel side: no kernel module was compiled or loaded at any version, on any kernel. The generator was exercised; the artefact was not.", "Gradle and the Android NDK: never used, at any version.", "Dart: the string-table formats were checked on two ABIs of one build (a Dart 3.6.0 snapshot), through a front end this skill does not ship.", - "Mitmproxy: the library and the standalone binary measured different versions (12.2.3 and 11.1.2); no capture in the record is tied to a mitmproxy release." + "Mitmproxy: the library and the standalone binary measured different versions (12.2.3 and 11.1.2); no capture in the record is tied to a mitmproxy release.", + "dexsql: only 0.1.0 on Windows amd64 was measured. The same release publishes Linux and macOS assets, and none of them was exercised; no other dexsql version was tested at all. The MCP surface was exercised to a tool listing, not from a real client session." ] } diff --git a/skills/apk-reverse/references/coverage-and-limits.md b/skills/apk-reverse/references/coverage-and-limits.md index e49b35f..97be066 100644 --- a/skills/apk-reverse/references/coverage-and-limits.md +++ b/skills/apk-reverse/references/coverage-and-limits.md @@ -32,6 +32,15 @@ but nobody here has paid for the counters yet. - **dex-level surgical patching**: equal-length byte edits and dexlib2 method rewrites, plus the header, verifier and alignment rules that decide whether the build loads (`dex-patching.md`, `byte-level-patching.md`, `patch-audit.md`). +- **Whole-APK dex indexing and query — locating a class or a reference before reading code.** Three + indexers ship as documented routes (`rasc`, `droidasc`, `dexsql`) and their class-definition sets + were compared against each other and against `ddc` on four archives (1,396 / 647 / 4 / 30,768 + classes): **identical on all four, zero differences in either direction**, and a string + cross-reference query agreed site for site. `dexsql` additionally exposes the dex as SQL — callers, + field accesses, numeric constants, debug lines, annotations, hiddenapi flags — plus the APK + resources and an MCP server (`rasc-and-droidsaw.md`, `dexsql-sql-query-layer.md`). **Boundary:** on a + hardened target all four return the same small class count because the payload is encrypted inside a + shell; no query language reaches that, and the unpacking routes own it. - **Repacking, signing, installing**, and the install refusals that look like a broken build (`repack-and-sign.md`). - **Packers, custom loaders and code virtualization**: identifying them, measuring the validation diff --git a/skills/apk-reverse/references/coverage/route-inventory.md b/skills/apk-reverse/references/coverage/route-inventory.md index b06d047..8738952 100644 --- a/skills/apk-reverse/references/coverage/route-inventory.md +++ b/skills/apk-reverse/references/coverage/route-inventory.md @@ -18,6 +18,7 @@ the evidence record condensed in `references/evidence-summary.md` §Where the fu | Client-side ads, promos, splash/popup/tab config | includes server-issued UI config that has no SDK to find | `ad-removal.md`, `server-config-and-updates.md` | | Membership / paywall / feature gates | decide whether it is *client-enforceable* at all, and say so when it is not | `membership-and-limits.md`, `account-gates.md` | | dex-level surgical patching | equal-length byte edits, dexlib2 method rewrite, header/verifier/alignment rules | `dex-patching.md`, `byte-level-patching.md`, `patch-audit.md` | +| Whole-APK dex indexing and query | locate classes and references **before** reading code: `rasc`/`droidasc` for `findrefs`, `dexsql` for SQL joins and aggregates (callers, field accesses, numeric constants, debug lines, annotations) — all four indexers cross-checked for identical class-definition sets on four archives | `rasc-and-droidsaw.md`, `dexsql-sql-query-layer.md` | | Repack, sign, install | including the install refusals that look like a broken build | `repack-and-sign.md` | | Packers, custom loaders, code virtualization | identification, the validation boundary, the routes that survive it | `packers.md`, `code-virtualization-and-custom-linkers.md` | | Java2C vs extraction shell vs JNI sinking | the misdiagnosis that sends you hunting a decrypted DEX that never exists | `java2c-and-jni-sinking.md`, `scripts/java2c_probe.py` | diff --git a/skills/apk-reverse/references/dexsql-sql-query-layer.md b/skills/apk-reverse/references/dexsql-sql-query-layer.md new file mode 100644 index 0000000..faf8a78 --- /dev/null +++ b/skills/apk-reverse/references/dexsql-sql-query-layer.md @@ -0,0 +1,201 @@ +# dexsql — the whole dex as SQL tables, and the APK's resources beside it + +Load this when: you are about to answer a question about a dex by grepping an export, or you already +use `droidasc`/`rasc` for `findrefs` and want the same answers as a relational query — or when you +want the artifact's manifest and resources in the same query surface as its code. + +**In one line:** a third indexer at the same stage as ASC, with the same class-definition answers +(measured identical on four archives), slower than `rasc` on a plain class list and faster than +`droidasc`, and one thing neither of the others has: the entire dex as SQL tables plus an MCP server. + +## What it is + +`LSPosed/dexsql` is a single self-contained binary (`dexsql-windows-amd64.exe`, 21.4 MB, v0.1.0). No +Python, no JVM, no SDK, no first-run index build. It parses the dex once into a relational model and +answers queries against it: + +```bash +dexsql tables --input app.apk # every table and column type +dexsql query --input app.apk --sql "SELECT count(*) AS n FROM methods" +dexsql query --input app.apk --interactive # keep the dataset loaded, read SQL from stdin +dexsql decode --input app.apk --class 'Lcom/x/Y;' # smali for a class or one method +dexsql prepare-cache --input app.apk --output cache # persistent cache.dxc2 for repeated queries +dexsql query-cache --cache-file cache/cache.dxc2 --sql "SELECT ..." +dexsql mcp # MCP server over stdio +``` + +Acquire it the same way as `ddc` — it is a release download, not a build: + +```bash +# https://github.com/LSPosed/dexsql-release/releases -> dexsql-- +``` + +**It is a registered capability.** `doctor.py` reports `dex_query_sql` as `OK` when the binary is on +PATH, on `APKREV_TOOLS`, or at the location this kit's own install step uses (`tools/_work/bin/`, +`DEXSQL`), and `BLOCKED` with the release URL as the next action when it is not. + +## The three indexers, and which one to reach for + +| | `rasc` | `droidasc` | `dexsql` | +|---|---|---|---| +| Answers class/reference questions | yes | yes | yes | +| Plain class list, 34.8 MB app | **0.055 s** | 0.200 s | 0.136 s | +| Reference query | `findrefs` (substring) | `findrefs` (substring) | SQL `WHERE value LIKE` | +| Read one class | `getclass` (Java) | `getclass` (Java) | `decode` (**smali**, not Java) | +| Relational model (callers, field accesses, numeric constants, debug lines, annotations, hiddenapi) | no | no | **yes** | +| APK manifest / resources / `resources.arsc` as queries | manifest only | manifest only | **yes, as SQL + MCP tools** | +| MCP server | no | no | **yes (13 tools)** | + +**The division of labour is now three-way:** `rasc` locates fastest, `droidasc` is the Python +cross-check, and `dexsql` answers the questions the other two cannot express — *which* methods call a +callee, which method holds a numeric constant above a threshold, what the debug line table says, which +annotations resolve on which owner. Reach for `dexsql` when the question is a **join or an +aggregate**, and for `rasc` when the question is a **class list**. + +**`decode` is not a decompiler.** It emits smali, and that is a feature: it is the bytecode-level +answer, which is what `dex-patching.md` tells you to decide with when a decompiler's Java looks +surprising. + +## The table model — what a query can ask + +~60 tables, grouped by what they answer (the full list is `dexsql tables`): + +| Group | Tables worth knowing | +|---|---| +| Identity / structure | `dex_files` (per-dex header fields, `checksum`, `signature_hex`, counts), `classes`, `class_defs`, `class_interfaces`, `fields`, `methods`, `protos`, `type_lists` | +| Code | `code_items` (`registers_size`, `ins_size`, `tries_size`), `method_opcodes`, `try_items`, `catch_handlers` | +| **Cross-reference** | `method_calls` (caller → callee), `method_field_accesses`, `method_strings` (method → literal), `method_used_types`, `method_numbers` (numeric constants), `method_using_numbers`, `method_refs` | +| Debug info | `debug_info`, `debug_positions` (address → line), `debug_locals`, `debug_parameters`, `debug_opcodes` | +| Annotations | `annotations`, `annotation_elements`, `annotation_values`, `annotation_elements_resolved` (owner + annotation + element flattened) | +| Runtime surface | `hiddenapi_flags`, `call_sites`, `method_handles` | + +The cross-reference group is the part that replaces an hour of grepping: `method_calls`, +`method_field_accesses`, `method_strings`, `method_used_types` and `method_numbers` are all +pre-joined views carrying `method`, `method_class` and `method_name` on the calling side. + +## Measured here + +Environment: Windows 10.0.26100, `dexsql 0.1.0` (Windows amd64 release asset, sha256 +`f354ebcf715de9a66ef8ebe08b2b36807ab58e25cd5a1c0812e497fc1fcfc448`), PowerShell 7. Four archives: +a 1.4 MB MASTG challenge (1,396 classes), a 4.3 MB hardened app (647), a 38.7 MB hardened app (4) and +a 34.8 MB app (30,768 classes / 268,446 methods). + +**Class-definition sets — identical to all three other tools, on every archive.** This is the check +that decides whether a new indexer may be trusted for location questions at all, and it is the same +comparison `rasc_build.py --verify` performs between `rasc` and `droidasc`: + +| Archive | dexsql | rasc | droidasc | ddc | verdict | +|---|---|---|---|---|---| +| 1.4 MB MASTG challenge | 1,396 | 1,396 | 1,396 | 1,396 | **identical** | +| 4.3 MB hardened | 647 | 647 | 647 | 647 | **identical** | +| 38.7 MB hardened | 4 | 4 | 4 | 4 | **identical** | +| 34.8 MB app | 30,768 | 30,768 | 30,768 | 30,768 | **identical** | + +Zero differences in either direction on all four — including the two archives where the count is +small because the payload is encrypted inside a shell, which is the target and not the tool. + +**Full class listing, best of 3, stdout to a file (the same shape for every tool):** + +| Archive | dexsql | rasc | droidasc | ddc | +|---|---|---|---|---| +| 1.4 MB / 1,396 classes | 0.035 s | 0.028 s | 0.149 s | 0.109 s | +| 4.3 MB / 647 classes | 0.031 s | 0.025 s | 0.139 s | 0.099 s | +| 38.7 MB / 4 classes | 0.040 s | 0.021 s | 0.156 s | 0.092 s | +| 34.8 MB / 30,768 classes | 0.136 s | **0.055 s** | 0.200 s | 0.156 s | + +So on a real app `dexsql` is **1.5× faster than `droidasc`** and **2.5× slower than `rasc`**; the +`rasc` and `droidasc` columns reproduce this repository's earlier figures (0.048 / 0.190 s on the +same 34.8 MB archive), which is what makes the row comparable rather than merely new. A count-only +query on the same archive is 0.056 s — the listing itself is most of the difference. + +**String cross-reference agrees with ASC.** On the 1.4 MB challenge, `SELECT DISTINCT method_class, +method_name FROM method_strings WHERE value LIKE '%http%'` returns exactly the three sites +`rasc findrefs … string http` and `droidasc findrefs … string http` return, and `ddc findrefs` agrees. +**The matching semantics differ and the difference is a trap:** `findrefs` matches a *substring*, +while SQL needs `LIKE '%…%'` — `value = 'http'` legitimately returns zero rows, because no method +holds a literal that *is* `http`. + +**`decode` agrees with `ddc disasm`.** `dexsql decode --method +'Lsg/vantagepoint/uncrackable3/MainActivity;->verifyLibs()V'` emits the same instruction sequence, in +the same order, as `ddc disasm … MainActivity.verify` — checked line by line on that method. + +**Persistent cache: a real feature with a small measured payoff here.** `prepare-cache` on the +34.8 MB archive takes 0.196 s and writes a 34.5 MB `cache.dxc2`. Repeating a query that needs the +full dataset — `method_calls` count, and a top-20 `GROUP BY` aggregate over it — went from 0.121 s to +0.098 s and from 0.470 s to 0.458 s. That is **under 20%, not an order of magnitude**, because the +parse-and-build path is already 0.12 s on this archive. Treat the cache as a way to hold one dataset +across many queries in a long session, not as a speed switch; on small archives it is a net loss +(cache read 0.071 s against a 0.042 s direct query on the 38.7 MB sample). + +**Read-only is enforced, and the error surface is uniform.** Missing class → `Error: not_found: +selected file, method, or class not found`, exit 1. Missing input, a non-dex input, an unknown column, +an unknown relation and a `DELETE` all exit 1 with a named cause; the write is refused. + +## Recipes worth keeping + +```sql +-- who calls what, ranked (the question `findrefs` cannot express) +SELECT callee_class, callee_name, count(*) AS n FROM method_calls +GROUP BY 1,2 ORDER BY n DESC LIMIT 20; + +-- every literal a method holds, instead of one needle at a time +SELECT method_class, method_name, value FROM method_strings +WHERE method_class LIKE 'Lcom/target/%' ORDER BY method_class; + +-- numeric constants above a threshold: config values, protocol magic, crypto parameters +SELECT method_class, method_name, number_type, long_value FROM method_numbers +WHERE long_value > 1000000 ORDER BY long_value DESC LIMIT 50; + +-- methods with no code (native, abstract) versus the ones that carry it +SELECT class_descriptor, name, method_kind, opcode_count FROM methods +WHERE opcode_count IS NULL OR opcode_count = 0; + +-- which owners carry a given annotation +SELECT owner, annotation, count(*) AS n FROM annotation_elements_resolved +GROUP BY 1,2 ORDER BY n DESC LIMIT 20; + +-- debug line table size: a cheap signal for "was this dex built with debug info" +SELECT count(*) AS positions FROM debug_positions; +``` + +`--sql` repeats, so several statements run against one loaded dataset; `--cache name=SELECT …` defines +a temporary table for the invocation; `--interactive` keeps the dataset loaded for a session. A +`method_idx` is DEX-local, so on a multi-dex APK carry `file_id` (from `dex_files`) into the join — +`-i` accepts several inputs and they share one dataset. + +**The MCP surface is the same model for an agent.** `dexsql mcp` speaks JSON-RPC over stdio and +publishes 13 tools — `open_dataset`, `list_tables`, `describe_table`, `query`, `decode`, +`get_manifest`, `get_manifest_components`, `list_resource_files`, `read_resource`, `query_resources`, +`list_datasets`, `close_dataset`, plus an embedded documentation catalog on `resources/list`. The +resource tools read `resources.arsc` and archive entries **without extracting them to disk**, and +every list-shaped result is paged by `next_offset`. Datasets are opened by path and are capped +(`--max-datasets`, `--max-input-bytes`, `--timeout-ms`), so a client that forgets `close_dataset` +degrades rather than grows. + +## Boundaries — what this does not settle + +- **A hit is a reference, not a call, and SQL does not change that.** `method_strings` proves a method + holds a literal; whether that method runs on the path you care about is still a disassembly or + runtime question (`dynamic-frida.md`). +- **`decode` gives smali, not Java.** When the question is Java-level, `ddc getclass` or `jadx` is + still the tool; `dexsql` is where you *locate* and where you check bytecode. +- **One version, one platform family was measured.** v0.1.0 on Windows amd64. Linux/macOS assets exist + in the same release but were not exercised here (`unverified`), and no other dexsql version was + tested at all. +- **The cache is not a speed switch** — see the measurement above; the honest claim is "hold a dataset + across queries", not "queries get fast". +- **`--trusted-input` skips input-consistency checks.** It exists for controlled build pipelines; do + not reach for it to make a suspicious file load. +- **A packed target stays packed.** On both hardened archives `dexsql` returns the same 4 and 647 + classes the other three tools return: the business dex is not in the file, and no query language + changes that. Take the question to the unpacking routes. +- **`quickened`/odex opcodes are unsupported** by `decode` (`odex` input is out of scope). +- **This is not a patching tool.** Nothing here writes a dex; `dex-patching.md` and + `byte-level-patching.md` own that, and `dexsql` is where you find the site to patch. + +## Provenance + +`https://github.com/LSPosed/dexsql-release` (release `v0.1.0`, published 2026-10-15; the repository +is the release host — it carries binaries, not source), retrieved and measured 2026-10-16. Nothing is +vendored into this repository: the binary lives in the gitignored `tools/` work area, and the +measurements above name the version, the platform and the hashes they came from. diff --git a/skills/apk-reverse/references/evidence-summary.md b/skills/apk-reverse/references/evidence-summary.md index 256af36..d33f35d 100644 --- a/skills/apk-reverse/references/evidence-summary.md +++ b/skills/apk-reverse/references/evidence-summary.md @@ -31,6 +31,7 @@ next** — and points at the one section that says "the full record is elsewhere | Dex method-level rewrite (dexlib2) | Documented, not measured: the smali round-trip blind spot is real and a tree round trip can pass every table check and still fail at load | partial | unverified | `references/dex-patching.md`, `references/patch-audit.md` | | Dex header integrity and verifier legality | Recompute order proven on one written file; `dex_classdiff` passing is necessary, not sufficient — it cannot see code-item damage | ok | observed | `references/patch-audit.md`, `scripts/dexutil.py`, `scripts/dex_check_verifier.py` | | Dex string-constant patch | Equal-length only; the shipped script refuses unequal lengths, so a different-length edit is a different, unmeasured route | partial | unverified | `references/byte-level-patching.md`, `scripts/dex_strpatch.py` | +| Whole-APK SQL query layer (dexsql) | Class-definition sets identical to rasc/droidasc/ddc on four archives, zero differences either way; a string xref query agrees site for site; `decode` emits smali, not Java, and the persistent cache is not a speed switch | ok | observed | `references/dexsql-sql-query-layer.md`, `references/toolchain.md` | | Repack, sign, install (single APK) | End to end: STORED and 4-byte-aligned `resources.arsc`, v1+v2+v3 true, install succeeded, on-device hash matched the local build, control build still fails the old way | ok | observed | `references/repack-and-sign.md`, `references/verification.md`, `scripts/repack.py` | | Split APK / App Bundle sets | Analyze, unified re-sign and merge measured on two real sets; merge correctly refuses a member carrying its own `resources.arsc` | partial | observed | `references/split-apk.md`, `scripts/repack.py` | | Third-party build audit | Not measured: the APK differ is on the list of scripts no pass has run | partial | unverified | `references/third-party-builds.md`, `scripts/apk_diff.py` | diff --git a/skills/apk-reverse/references/routing.md b/skills/apk-reverse/references/routing.md index 834acfb..9fb017e 100644 --- a/skills/apk-reverse/references/routing.md +++ b/skills/apk-reverse/references/routing.md @@ -112,6 +112,7 @@ _54 row(s) below the header._ | `references/long-task-discipline.md` | The task will run long, or you are resuming one. Live record, conclusion grading, drift checkpoints, **deliverable-form drift (rooted-only vs shippable)**, bound-your-waits, **captures-you-never-looked-at**, **long-context decay**, handover | | `references/pitfalls.md` | Always worth a skim before building. This is the failure catalogue. | | `references/rasc-and-droidsaw.md` | You are about to reach for the ASC indexer, or the Python one is the bottleneck: the Rust re-implementation, its measured speedup and **identical class sets**, the one class shape where it silently drops code, and how to build it | +| `references/dexsql-sql-query-layer.md` | You are about to answer a dex question by grepping an export, or you need a **join or an aggregate** rather than a class list: the whole dex as SQL tables (callers, field accesses, numeric constants, debug lines, annotations, hiddenapi), manifest and `resources.arsc` queries, the MCP server, the measured standing against rasc/droidasc/ddc, and the two traps (`decode` is smali; `=` is not a substring match) | | `references/coverage-and-limits.md` | You need to weigh a claim before trusting it: the evidence behind each covered item, the dependencies this skill does not ship, and the record of what was never exercised | | `references/coverage/route-inventory.md` | You must answer "**can this skill actually do X**" in one pass: the per-route inventory — covered by verified mechanism, covered only by a documented route, the dependencies not shipped, and what is out of scope | | `references/evidence-summary.md` | You need to know **whether a route was verified and how strong the evidence is**, from inside an installed copy: capability → one-line conclusion → strength → where the evidence lives | diff --git a/skills/apk-reverse/references/toolchain.md b/skills/apk-reverse/references/toolchain.md index 13e3dee..1c6e92c 100644 --- a/skills/apk-reverse/references/toolchain.md +++ b/skills/apk-reverse/references/toolchain.md @@ -35,6 +35,7 @@ cross-check with the program headers. | **`rasc`** (Rust ASC) | `rasc findrefs app.apk string ` | **the same index, 4–15× faster**, no Python runtime, a Rust DEX decompiler behind `getclass`. Reach for this FIRST when it is built — its one blind spot is documented in `rasc-and-droidsaw.md` | | **`droidasc`** (ASC) | `droidasc findrefs app.apk string ` | the original Python index, one `pip install` away, and the cross-check for `rasc`. Reach for it FIRST when `rasc` is not built | | **`ddc`** | `ddc app.apk -c ` | **fastest read of dex as Java, plus query subcommands** — reach for this SECOND, to *read*. See below | +| **`dexsql`** | `dexsql query --input app.apk --sql "…"` | **the whole dex as SQL tables** (callers, field accesses, numeric constants, debug lines, annotations) plus manifest and `resources.arsc` queries and an MCP server. Reach for it when the question is a **join or an aggregate** rather than a class list — see below | | `apktool` | `java -jar apktool.jar d/b` | whole-app decode including resources | | `jadx` | `jadx --no-res -d ` | readable Java for orientation; **not** a source of truth, and **not** a recon entry point (see below) | | `aapt2` | `aapt2 dump badging ` | manifest facts, package name, versions | @@ -213,6 +214,48 @@ matters: a full `ddc -o out/` export is a reasonable thing to *have*, but not a substitute for one indexed query, and it is the slower of the two by a wide margin on any APK worth analyzing. +### dexsql — the whole dex as SQL, and the resources beside it + +A third indexer at the same stage as ASC, and the one to reach for when the question is a **join or an +aggregate** rather than a class list. One self-contained binary (v0.1.0, 21.4 MB on Windows amd64, no +Python, no JVM, no index build), acquired as a release download from +`https://github.com/LSPosed/dexsql-release/releases`: + +```bash +dexsql tables --input app.apk # every table and its column types +dexsql query --input app.apk --sql "SELECT count(*) AS n FROM methods" +dexsql query --input app.apk --sql "…" --sql "…" # several statements, one loaded dataset +dexsql query --input app.apk --interactive # keep it loaded, read SQL from stdin +dexsql decode --input app.apk --class 'Lcom/x/Y;' # smali, not Java +dexsql prepare-cache --input app.apk --output cache # then: query-cache --cache-file cache/cache.dxc2 +dexsql mcp # MCP server over stdio +``` + +**What it is for.** `method_calls` (caller → callee), `method_field_accesses`, `method_strings`, +`method_used_types` and `method_numbers` are pre-joined views carrying the calling method's class and +name — so "which methods call `X`", "which methods hold a numeric constant above N", "what does the +debug line table say", "which owners carry this annotation" are single queries instead of greps. +`hiddenapi_flags` and the annotation tables have no equivalent in the other two indexers. + +**Where it sits against the other two, measured on four archives (identical class-definition sets on +all four, zero differences either way):** on a 34.8 MB / 30,768-class app a full class listing takes +`dexsql` **0.136 s** against `droidasc` 0.200 s and `rasc` 0.055 s — 1.5× faster than the Python +indexer, 2.5× slower than the Rust one. **So it does not replace `rasc` for a class list**; it answers +questions `rasc` cannot express. Its string cross-reference was checked against ASC's on a real +archive and agrees site for site. + +**Two traps.** `decode` emits **smali**, not Java — when the question is Java-level, `ddc getclass` +still owns it. And SQL's `=` is not `findrefs`: ASC matches a *substring*, so `WHERE value = 'http'` +legitimately returns nothing where `findrefs string http` returns three sites; use `LIKE '%…%'`. + +**The persistent cache is not a speed switch.** `prepare-cache` on the 34.8 MB archive writes a +34.5 MB `cache.dxc2` in 0.196 s, and repeating a full-dataset query against it moved 0.121 s → 0.098 s +— **under 20%**, because parse-and-build is already 0.12 s there. It is a way to hold one dataset +across many queries in a long session; on a small archive it is a net loss. + +Full table inventory, recipes, the MCP tool list and the boundaries: +`references/dexsql-sql-query-layer.md`. + **Classpath gotcha.** `baksmali`/`smali` need their dependency jars on the classpath together (`smali`, `antlr-runtime`, `stringtemplate`, `baksmali`, `dexlib2`, `util`, `jcommander`, `guava`). Missing one produces `ClassNotFoundException` that reads like a broken target. `scripts/smtool.py` diff --git a/skills/apk-reverse/scripts/capabilities.py b/skills/apk-reverse/scripts/capabilities.py index e5b8a15..2ced049 100644 --- a/skills/apk-reverse/scripts/capabilities.py +++ b/skills/apk-reverse/scripts/capabilities.py @@ -211,6 +211,17 @@ ATOMS = { 'est_minutes': 1, 'basis': 'measured', }, + 'tool:dexsql': { + 'name': 'dexsql (SQL over dex and APK resources)', + 'kind': 'tool', + 'tool': 'dexsql', + 'install': 'download the release binary for this platform from ' + 'https://github.com/LSPosed/dexsql-release/releases and put it on PATH ' + 'or on APKREV_TOOLS (no Python, no JVM, no crate, no build step)', + 'est_minutes': 2, + # Measured here: release download plus first query, on the Windows amd64 asset. + 'basis': 'observed', + }, 'tool:apktool': { 'name': 'apktool (resource round-trip)', 'kind': 'tool', @@ -356,6 +367,19 @@ CAPABILITIES = { 'note': 'one `pip install droidasc` away, and the cross-check for the Rust one: ' 'their class-definition sets are compared in scripts/rasc_build.py --verify', }, + 'dex_query_sql': { + 'name': 'Whole-APK SQL query layer over dex and APK resources (dexsql)', + 'required': ['tool:dexsql'], + 'optional': [], + 'note': 'one self-contained binary: no Python, no JVM, no index build. Its ' + 'class-definition set matched rasc, droidasc and ddc exactly on four archives ' + '(1,396 / 647 / 4 / 30,768 classes, zero differences either way). What it adds ' + 'is the relational model -- method_calls, method_field_accesses, method_strings, ' + 'method_numbers, debug_positions, hiddenapi_flags, annotation_elements_resolved ' + '-- plus APK resource queries and an MCP server. It is a third indexer at the ' + 'same stage as ASC, not a replacement for it: rasc is still the fastest for a ' + 'plain class list. See references/dexsql-sql-query-layer.md', + }, 'static_dex': { 'name': 'Static dex / zip / strings analysis', 'required': ['python39'], @@ -535,6 +559,24 @@ def extra_tool_dirs(): return [d for d in dirs if d and os.path.isdir(d)] +# Tools whose *documented install step in this kit* lands somewhere PATH does not know about. +# A machine that followed those instructions has the tool, and reporting it BLOCKED would make +# the capability table lie in the pessimistic direction -- the same reasoning as APKREV_TOOLS, +# applied to the tools acquired by a build or a release download rather than a package manager. +INSTALL_HINTS = { + 'rasc': [ + os.path.join(REPO_ROOT, 'tools', '_work', 'rust', 'target', 'release', + 'rasc' + ('.exe' if os.name == 'nt' else '')), + os.environ.get('RASC', ''), + ], + 'dexsql': [ + os.path.join(REPO_ROOT, 'tools', '_work', 'bin', + 'dexsql' + ('.exe' if os.name == 'nt' else '')), + os.environ.get('DEXSQL', ''), + ], +} + + class Probe: """Host facts, probed once per process and cached. @@ -559,15 +601,12 @@ class Probe: if name in self._tool_cache: return self._tool_cache[name] hit = shutil.which(name) - if not hit and name == 'rasc': - # rasc ships no artifact, so a machine that followed this kit's own build step has it - # under the repository's ignored work area rather than on PATH. Accepting that location + if not hit: + # The kit's own install step for this tool lands outside PATH (rasc is built into the + # ignored work area, dexsql is a downloaded release binary). Accepting that location # keeps the capability table from reporting BLOCKED at an agent that already did the - # work -- the same reasoning as APKREV_TOOLS, for the one tool whose install step is a - # build (scripts/rasc_build.py). - for cand in (os.path.join(REPO_ROOT, 'tools', '_work', 'rust', 'target', 'release', - 'rasc' + ('.exe' if os.name == 'nt' else '')), - os.environ.get('RASC', '')): + # work -- see INSTALL_HINTS. + for cand in INSTALL_HINTS.get(name, []): if cand and os.path.isfile(cand): hit = cand break