Absorb gaps from three public reverse-engineering skills

Batch 1 (full): Kotlin original-name recovery (new reference + zero-dep
kotlin_name_recovery.py mining @DebugMetadata/@Metadata.d2/jadx rename
comments, with -o/-p/--grep query modes); jadx advanced flags and a
Fernflower/Vineflower second-engine tier in toolchain.md; BuildConfig
sweep habit in recon.md; WebSocket/long-connection state-chain section
in protocol-reverse.md.

jsvmp-only (batches 2-3): A/B/C recovery ladder with black-box reuse
boundary and a wire-parameter family index in jsvmp-recovery.md; new
js-ast-deobfuscation.md pipeline layer (family detection, per-family
pass order, safe-rewrite rules, minimal bootstrap eval, residue metrics)
with js_obf_detect.py weighted detector.

Wiring: routing.md/SKILL.md symptom+index rows, coverage-and-limits,
route-inventory, evidence-summary, capability-matrix (+3 entries,
labels kept honest: scripts observed on fixtures, absorbed methodology
unverified). 12 new unit tests.

Also: silence two environment-dependent mypy errors (yaml/frida
sentinel assignments) and bump mypy python_version 3.9 -> 3.10 with the
trade-off recorded.

Gates: check_repo/check_refs/check_budget/check_commands/check_routing
PASS; pytest 787 passed; flake8 clean; mypy clean.
This commit is contained in:
dabao1955
2026-10-20 11:54:59 +08:00
parent 8f32bb5e29
commit aa9af54854
19 changed files with 1223 additions and 20 deletions
+5 -1
View File
@@ -31,7 +31,11 @@ import sys
try:
import yaml
except ImportError: # the skills CLI is the real authority; this is a local gate
yaml = None
# `ignore[assignment]` because the sentinel assignment is only ill-typed where the
# module IS installed (mypy then knows the name as Module); on a bare machine the
# override in mypy.ini makes it Any and the ignore is unused -- warn_unused_ignores
# is off, so both environments stay clean.
yaml = None # type: ignore[assignment]
ROOT = os.path.dirname(os.path.abspath(__file__))
SKILLS_DIR = os.path.join(ROOT, 'skills')
+11 -3
View File
@@ -17,11 +17,19 @@
# configuration, on a bare interpreter (no frida/capstone/tiktoken/PyYAML installed), so the
# overrides below are exercised rather than skipped.
#
# Verified locally on 2026-09-22: `mypy` over 55 files reports no issues with the overrides in place,
# on a venv containing only mypy and flake8.
# Verified locally on 2022-09-22 baseline; re-verified 2026-10-20 after the target bump below:
# `mypy` over 64 files reports no issues with the overrides in place, on an interpreter that
# also has yaml/frida installed (the ignore comments in check_repo.py and
# spawn_patch_detach.py keep both the installed and bare-machine sides clean).
[mypy]
python_version = 3.9
# 3.10, not 3.9: mypy releases from 2025 on dropped `python_version = 3.9` as a supported
# target and warn on every run. The honest cost: under a 3.10 target mypy stops rejecting
# 3.10-only syntax (match statements, runtime `X | Y`), so the SKILL.md "Python 3.9+"
# claim is no longer statically guarded -- nothing in the tree uses such syntax, and CI's
# oldest actually-executed interpreter is 3.11 (see ci.yml), which is the check that would
# catch a real 3.9 break at test time.
python_version = 3.10
warn_unused_ignores = False
[mypy-frida]
+2 -1
View File
@@ -234,12 +234,13 @@ unread in this repository.
| The trace segments themselves look corrupted — handler heads missing, tails glued into wrong segments | `references/native-vm-recovery.md` §7 Nested VMs — a flat segment accumulator discards the outer level; nested slices need a segment stack (`vm_handler_slice.py` flags the symptom) |
| You need to know **where two runs of the same operation diverge** (password/license/key), not just what executed | `ete-diff-tracing.md` — ETE hardware-trace gates plus the differential method; `scripts/trace_diff.py` also diffs Stalker logs |
| A generated KPM/LKM template is in front of you, or the kernel route feels like the plan | `kernel-event-injection.md` — the primitive map, the composition recipes, and the gates every kernel-side file must pass before it is a weapon |
| The traffic is protobuf/gRPC/QUIC, or a proxy sees TLS but requests still fail on a Flutter app | `references/protocol-reverse.md` — schema-less protobuf, frame capture, and native-side pinning |
| The traffic is protobuf/gRPC/QUIC, a proxy sees TLS but requests still fail on a Flutter app, **or the behaviour rides a long-lived WebSocket** | `references/protocol-reverse.md` — schema-less protobuf, frame capture, native-side pinning, and the long-connection state chain |
| Userspace hooks land and the app still dies: the check reads `/proc/self/status` through a raw `svc`, or runs before `JNI_OnLoad` | `references/kernel-and-environment-hardening.md` — what the next layer up and down can actually do, and when to stop |
| You must edit, repack, sign or inspect the APK **from the phone itself** | `references/on-device-tooling.md`, `scripts/mt_mcp_probe.py` |
| A captured body decodes to nothing readable, or you cannot tell whether a length-delimited field is a string, a nested message or a packed array | `protocol-reverse.md`. Protobuf on the wire (measured) — run `scripts/protobuf_decode_raw.py`; the candidate list and its `tie:` lines are the answer |
| Method bodies are present but decode as **private opcodes**, and you need the mapping rather than an explanation of why VMP is hard | `vmp-differential-analysis.md`, then `advanced-unpacking.md` for the shape diagnosis |
| The value you need (sign, token, risk parameter) is computed in **JavaScript** — a WebView or embedded JS engine — by an entry call fed a huge hex/base64 opcode literal: a JS virtual machine | `jsvmp-recovery.md` — de-obfuscate and **call it** first, then the stack/register-VM procedures and the four JS traps; `emulation-and-rpc.md` owns the call-don't-read route |
| You are reading a decompiled **Kotlin** app and every class name is a meaningless short token (`nq.e`, `C0123x`) | `kotlin-name-recovery.md` — the original FQNs survive R8 inside `@DebugMetadata`/`@Metadata`; `scripts/kotlin_name_recovery.py` rebuilds the mapping from any jadx export |
| A store build arrives as `base.apk` + `split_config.*.apk`, or a rebuilt build is refused **as a set** although every file verifies on its own | `split-apk.md` — one keystore across every member for `pm install-multiple`, and check that a merge is legal before trusting a merged single APK |
## Gates — clear these before you patch, in order
@@ -847,6 +847,53 @@
"three public write-ups of one reverse-engineering series, 2024-03..2024-09 (titles and authors anonymized; the absorbed record is the reference file itself)"
],
"limits": "The recovery procedures (recognition signals, the call-it-first route, stack- and register-machine reading, AST instruction extraction with register/immediate typing, the four JS-language evaluation traps, constant folding and control-flow recovery) are absorbed from three public write-ups and never executed by this repository on any real target -- all inferred. The shipped loader decoder is host-measured only against its own round trip (2,016 values exact, known varint/ZigZag byte layouts pinned, truncation raises) and decodes the write-up's own sample literal end to end; that is corroboration of the chain's shape, not a measurement on an independent target. Engine-specific delivery (WebView, QuickJS/JSC behind JNI, mini-program containers) was never exercised."
},
{
"capability": "js-ast-pre-deobfuscation",
"group": "unpacking",
"status": "partial",
"route_measured": false,
"strength": "observed",
"benchmark_rows": [],
"evidence": [
"references/js-ast-deobfuscation.md",
"scripts/js_obf_detect.py"
],
"evidence_not_shipped": [
"lwjjike/xbsReverseSkill (MIT), ast-deobfuscation skill: pipeline-config.js, safe-rewrite-rules.md, string-array-and-minimal-eval.md, control-flow-and-opcode-patterns.md, pattern-layering.md -- architecture and rules absorbed 2026-10-20, no code copied"
],
"limits": "The detection layer is observed against its own embedded fixtures only (four family fixtures, the hint-beats-content scoring rule, plain-code-is-generic, residue metric filter) -- no real obfuscated bundle was ever classified here. The rewriting passes (string-array restore, dispatcher inlining, array-CFF flattening, if-chain-to-switch, identifier renaming) need node plus @babel/parser/traverse/generator, which this skill does not ship; every rewrite is a documented route, unverified end to end. Residue metrics are textual proxies, valid for same-file before/after comparison only. The per-family pass-order caveat carried from the absorbed architecture (VM-heavy families skip late literal-inlining because it stalls large samples) is the upstream's measurement, not this repository's."
},
{
"capability": "kotlin-name-recovery",
"group": "dex",
"status": "partial",
"route_measured": false,
"strength": "observed",
"benchmark_rows": [],
"evidence": [
"references/kotlin-name-recovery.md",
"scripts/kotlin_name_recovery.py"
],
"evidence_not_shipped": [
"SimoneAvogadro/android-reverse-engineering-skill (Apache-2.0): recover-kotlin-names.sh and references/kotlin-name-recovery.md -- re-implemented 2026-10-20 as dependency-free Python"
],
"limits": "Parsing of all three recovery sources (@DebugMetadata c=, @Metadata d2 first non-stdlib descriptor, jadx 'renamed from' comments), the third-party skip list and the map-file round trip are observed on embedded fixtures only. The upstream recovery-rate claims (30-50% of classes overall, ~100% of *Repository/*ViewModel/*Impl, ~80% of data-class DTOs) are unverified here: no real obfuscated Kotlin app was run through the script. Method and field names are never recovered; heavily inlined shapes can surface under a wrong primary class, so every mapping row is a hint. The script reads decompiled sources, not the dex: a decompiler that drops annotations removes the source entirely."
},
{
"capability": "websocket-long-connection-analysis",
"group": "protocol",
"status": "partial",
"route_measured": false,
"strength": "unverified",
"benchmark_rows": [],
"evidence": [
"references/protocol-reverse.md"
],
"evidence_not_shipped": [
"715494637/reverse-skill (no license file): jsr-reverse/references/protocol-and-long-connection.md -- methodology only absorbed 2026-10-20, no text copied"
],
"limits": "The state-chain method (capture handshake, first business frame, heartbeat/ack, renewal, error; record the connection as a dependency chain; separate message families; split envelope from payload before body semantics) is absorbed from a browser-WebSocket reference and was never run on an Android long-connection target here. The Android capture points in the section (OkHttp WebSocket listener surfacing, native/QUIC transports never transiting Java) are inferred from documented APIs, not measured. Frame bodies that are protobuf do inherit the measured schema-free decoder from protocol-reverse.md section 1; that is the only measured part of the route."
}
]
}
}
@@ -141,10 +141,31 @@ its own strength note at the top.
- **JS VMP (jsvmp) recovery** — reading logic protected as a private JS opcode stream: recognition
signals, the call-it-first route (AST restore + environment fill), stack- and register-machine
reading procedures, AST instruction extraction with register/immediate typing, the four
JS-language traps, and constant-folding/control-flow recovery. Absorbed from three public
JS-language traps, and constant-folding/control-flow recovery; the A/B/C depth ladder and the
wire-parameter family index are later additions from a fourth public source (provenance in the
file). Absorbed from three public
write-ups; every procedure `inferred`, none run on a real target. The one shipped piece is the
loader decoder (base64 → varint → ZigZag), host-measured against its own round trip plus the
write-up's sample literal (`jsvmp-recovery.md`, `scripts/jsvmp_loader_decode.py`).
- **JS AST pre-deobfuscation (detection + pass order)** — the pipeline layer `jsvmp-recovery.md`
names as its step 1: weighted family detection and per-family pass ordering, safe-rewrite rules,
minimal bootstrap eval, residue metrics. Absorbed 2026-10-20 from a public MIT-licensed skill
(`lwjjike/xbsReverseSkill`); the rewriting passes need node+babel, which this skill does not
ship, so the end-to-end route is `unverified`. The shipped detector is `observed` against its
own embedded fixtures only (`js-ast-deobfuscation.md`, `scripts/js_obf_detect.py`).
- **Kotlin original-name recovery** — the mapping R8 cannot strip: `@DebugMetadata`/`@Metadata.d2`
carry original class FQNs into any decompiled export, and the shipped script rebuilds the
obfuscated→original map plus `--grep` annotation. Absorbed 2026-10-20 from a public
Apache-2.0-licensed skill (`SimoneAvogadro/android-reverse-engineering-skill`); parsing is
`observed` against embedded fixtures, the upstream recovery-rate claims are `unverified`, and
no real obfuscated app was run through it here (`kotlin-name-recovery.md`,
`scripts/kotlin_name_recovery.py`).
- **Long-connection (WebSocket) state-chain analysis** — capture beyond the first frame,
connection state chains, message-family separation, envelope/payload split. Absorbed
2026-10-20 from a public unlicensed JS-reverse skill's reference (methodology only); written
for browser targets and never run on an Android long-connection app here — `unverified`
end to end, with Android capture points `inferred` from documented APIs
(`protocol-reverse.md` §6).
- **Kernel-side event injection** — invisible watchpoints and register rewrites from a driver, as
generated templates with their gates; nothing kernel-side was ever compiled or loaded here
(`kernel-event-injection.md`, `scripts/kernel_instrument_scaffold.py`).
@@ -153,6 +174,10 @@ its own strength note at the top.
## Dependencies this skill does not ship — name them before the workflow starts
- **JS AST rewriting needs node + babel** (`@babel/parser`/`traverse`/`generator`) — the detector
and the pass order ship; the rewriting engine does not, and every rewrite pass is a documented
route until that stack is installed (`js-ast-deobfuscation.md`).
- **Dart AOT analysis needs a snapshot dump.** `dart-aot.md`'s workflow begins at `pp.txt`;
producing it requires a snapshot container resolver this skill does not contain and cannot
synthesize. `dart_pool_strings.py` reports **file** offsets while `pp.txt` and `dart_pprefs.py`
@@ -49,6 +49,9 @@ route was not measured on this repository's evidence**. Treat them as leads with
| Interpreter-trace de-virtualization | handler slicing + guest-opcode trajectory, calibrated on an open IR-VM's source — `vm_handler_slice.py` |
| Native VM recovery procedure | the absorbed method end to end: dispatcher separation, register anchoring, nested stacks, operand slicing, re-VM validation — `native-vm-recovery.md` |
| JS VMP (jsvmp) recovery | recognition, call-it-first, stack/register-VM reading, instruction extraction, the four JS traps — procedures absorbed, only the loader decoder shipped and round-trip-tested — `jsvmp-recovery.md` |
| JS AST pre-deobfuscation | family detection + per-family pass order + safe-rewrite rules; detector shipped and fixture-tested, rewriting needs node+babel (not shipped) — `js-ast-deobfuscation.md`, `js_obf_detect.py` |
| Kotlin original-name recovery | obfuscated→original class map from `@DebugMetadata`/`@Metadata` in any jadx export; parsing fixture-tested, recovery rates upstream-claimed — `kotlin-name-recovery.md`, `kotlin_name_recovery.py` |
| Long-connection (WebSocket) analysis | state chains, message families, envelope/payload split — absorbed methodology, never run on an Android target here — `protocol-reverse.md` |
| Kernel-side event injection | watchpoints and register rewrites from a driver, as gated templates — `kernel-event-injection.md` |
| Working from the phone itself | MT Manager edit/repack/sign and its APK MCP, LSPosed Manager, Termux — `on-device-tooling.md` |
| Publishing without publishing the target | the leak scanner, the do-not-anonymize list, and the graded precedent library — `desensitization-and-leak-scans.md`, `scripts/scan_leaks.py`, `precedents/` |
@@ -62,6 +62,9 @@ next** — and points at the one section that says "the full record is elsewhere
| Interpreter-trace de-virtualization | Dispatcher slicing, signature clustering and the opcode trajectory measured on synthetic interpreter traces; the calibration facts come from a public IR-VM's source, and no hardened binary was traced | partial | observed | `references/ete-diff-tracing.md` §5, `scripts/vm_handler_slice.py` |
| Native VM recovery procedure | The full method is absorbed from two public write-ups (dispatcher-layer separation, register anchoring with triple verification, nested segment stacks, STORE-backward slicing, clean-re-VM validation); no step was executed by this repository | partial | inferred | `references/native-vm-recovery.md` |
| JS VMP (jsvmp) recovery | Procedures absorbed from three public write-ups (stack and register machines) and never run on a real target; the shipped loader decoder (base64→varint→ZigZag) round-trips 2,016 values exactly, pins the known varint/ZigZag layouts and decodes the write-up's own sample literal end to end | partial | inferred | `references/jsvmp-recovery.md`, `scripts/jsvmp_loader_decode.py` |
| JS AST pre-deobfuscation | Detector self-tested on embedded fixtures (4 families, hint-beats-content, plain-is-generic) and residue metrics pinned; the rewrite passes themselves need node+babel and were never run here | partial | observed | `references/js-ast-deobfuscation.md`, `scripts/js_obf_detect.py` |
| Kotlin original-name recovery | All three annotation sources (DebugMetadata, Metadata.d2, jadx rename comments) parse correctly on pinned fixtures, third-party trees skipped, map files round-trip; upstream's 30–50%/≈100% recovery-rate claims not measured here | partial | observed | `references/kotlin-name-recovery.md`, `scripts/kotlin_name_recovery.py` |
| Long-connection (WebSocket) analysis | Absorbed state-chain methodology (handshake→heartbeat/ack→renewal, message families, envelope/payload split); no Android long-connection app was captured through it here | partial | unverified | `references/protocol-reverse.md` §6 |
| Kernel-side instrumentation/injection | The scaffold's generate/verify is measured and every kernel file ships unbuilt with its gate; the register-rewrite action is explicitly experimental and never ran anywhere | blocked | unverified | `references/kernel-event-injection.md`, `scripts/kernel_instrument_scaffold.py` |
| `svc` site scanning | Two independent decoders agreed on an identical 214-site set; a byte scan also matches data, so neighbour context decides | ok | observed | `references/kernel-and-environment-hardening.md`, `scripts/svc_scan.py` |
| On-device tooling (MT Manager MCP) | The service-down path is measured; the connected path needs the service started by hand | partial | observed | `references/on-device-tooling.md`, `scripts/mt_mcp_probe.py` |
@@ -0,0 +1,135 @@
# JS AST pre-deobfuscation — the pipeline layer before `jsvmp-recovery.md` reads anything
**Load this when** the JavaScript in front of you (pulled from a WebView, an embedded engine, a
mini-program bundle, `assets/www/`) is unreadable through *structure*, not just minification —
string tables, self-executing decoder wrappers, flattened control flow, if-chain dispatch — and
`jsvmp-recovery.md` §3's step 1 ("AST pre-deobfuscation") is the instruction you are executing.
This file owns **how that step is run as a pipeline**; the VM work after it stays in
`jsvmp-recovery.md`, and this file deliberately does not repeat it.
**What this answers:** which family the sample belongs to, in what order the rewrite passes run,
when a rewrite is *safe to apply at all*, and how you prove the batch helped. Contents:
§1 the three layers · §2 detection · §3 the pass library and order · §4 safe-rewrite rules ·
§5 string tables and minimal bootstrap eval · §6 residue and proof · §7 extending the library.
**The fence — this skill ships detection and order, not the rewriter.** The passes need a real JS
parser: node + `@babel/parser` + `@babel/traverse` + `@babel/generator`. That stack is one
`npm install` away, but this repository does not carry it; every *rewrite* below is a documented
route (`unverified` as end-to-end), while the **detector and its fixtures are `observed`**
(`scripts/js_obf_detect.py --selftest`). Re-parse after every batch — output that does not re-parse
is corruption, not progress.
**Provenance.** Absorbed 2026-10-20 from `lwjjike/xbsReverseSkill` (MIT), its `ast-deobfuscation`
skill — the pipeline-config step library with per-family ordering and weighted detection, and the
safe-rewrite-rules, string-array-and-minimal-eval, control-flow-and-opcode-patterns and
pattern-layering notes. No code or pattern corpus was copied; the architecture and the rules are
the absorption. The absorbed pipeline's own measured caveat
(late literal-inlining stalls on large VM-heavy samples, so that family skips it) is carried in §3.
## 1. Three layers, and the rule that keeps them honest
1. **Generic transform layer** — stable, low-risk rewrites that hold across unrelated samples.
2. **Detection / pipeline-selection layer** — score the sample, pick the pass order.
3. **Family adapter layer** — narrow passes for one vendor's reproducible quirk.
The rule (the reason the absorbed library stayed usable): **a family rule does not move into the
generic layer until it has recurred in at least two unrelated samples** with its safety boundary
still clear. The failure this prevents is generic passes accreting site-specific heuristics until
every run needs per-sample exemptions — at which point the pipeline is unmaintainable *and*
unreviewable, and mis-rewrites become undebuggable.
## 2. Detection — score before you choose the order
`scripts/js_obf_detect.py <file.js> [--hint TEXT] [--json]` implements the absorbed scoring model:
- a **hint** token matched against your `--hint` text or the file path scores **10** — a human
saying "this is the delivery platform's sign SDK" outranks any regex;
- each **content** regex hit scores **3** — `_0x` identifiers and a top-level string array
(string-array lineage), `while(!![]){switch(...)}` with `"3|1|2".split('|')` order tables
(flattening), a huge hex/base64 literal plus a `[Object, Math, …]` dependency array (jsvmp
entry — cross-check against `jsvmp-recovery.md` §2's table, which is the authoritative
recognition step), long literal-vs-variable if-chains (opcode dispatch).
Any score > 0 is a detection; the top scorer sets the pipeline; nothing scored means the generic
ladder, which is also where you fall back when family residue stays stable after two batches.
The huge-literal threshold is `--min-literal` (default 1500) — a parameter, not a boundary: lower
it for small samples, and say so in your record.
## 3. The pass library and the order that matters
Generic ladder (absorbed order): **normalize → prune fake branches → inline dispatchers → flatten
array-driven control flow → if-chain to switch → prune → normalize**, with identifier renaming
**last**, and a `string_array_pass` **first** when detection says the lineage uses one — every
later pass reads strings through the decoder, so metrics and rewrites on undecoded text are noise.
Per-order rules, each paid for by someone's stall:
- **Decode the string array before any control-flow work** (§5).
- **Never jump to full beautification because the code is ugly.** The stop condition is "the main
dispatcher is readable and the key cases are individually analysable", not "the file is pretty".
`jsvmp-recovery.md` §3's ladder decides how much deeper than that you ever go.
- **VM-heavy families skip late literal-inlining** — the absorbed architecture's own measured
example of a pass order that hangs a large sample; re-ordering per family is the point of the
detection layer, not a hack.
- **A pass that finds no match must exit clean**, not "best effort" — a visitor that half-applies
leaves the tree in a state no later pass expects. Structure mismatch ⇒ skip and report.
## 4. Safe-rewrite rules — when a node is *certainly* foldable
Replace **only when the answer to "is this certainly recoverable" is yes**; anything less is a
corruption lottery.
Certainly replaceable: array-literal indexing with a numeric literal and stable contents · calls
whose arguments are all literals **after** the minimal decoder environment exists (§5) · pure
expressions (`['a','b'].join('')`, `String.fromCharCode(...)`, `split/reverse/join` chains) ·
verified alias chains (`var a='x'; var b=a;`) · statically-true/false comparisons.
Never fold: anything depending on `this`/global/closure-external bindings or the host environment ·
getters/setters/Proxy objects/dynamic property access/side-effecting calls · alias chains whose
binding stability is unproven · comma/logical expressions where order matters · anything needing
cross-scope execution reasoning.
Judgement order: (1) does the result depend only on literals and stable local bindings?
(2) does evaluation have side effects? (3) does it depend on execution order, host state, or
environment? — **any unknown at any step ⇒ do not replace**; leave the region for runtime evidence.
And never re-fold an already-folded literal: putting `StringLiteral`s back through the evaluator is
how output destabilizes.
## 5. String tables and minimal bootstrap eval
The string-array family: the file's first statements build a decoder — a top-level IIFE rotating an
array, or 2–5 statements initialising one. Order:
1. Locate the array literal, the decoder function, and the alias chain.
2. **Static first**: array indexing, literal concatenation, numeric-argument calls.
3. Only when static stops: **minimal bootstrap eval** — execute *only* the first few statements
that construct the decoder, never the file; never in an environment with DOM/timers/network;
with the narrowest goal ("resolve numeric-argument calls to strings").
4. Delete the consumed bootstrap declarations, **re-parse**, then enter control-flow work.
Stop when the main table is restored and the decoder entry is clear; string tables beyond that are
low-value residue. For a jsvmp sample this is `jsvmp-recovery.md` §3 step 1, which halved the
write-up's sample before any VM reading.
## 6. Residue and proof — a batch helped, or it did not
Measure **before and after each batch** with `js_obf_detect.py`'s residue metrics (mangled-identifier
count, hex-literal density, comma-expression count, switch/case counts, longest line). They are
*textual proxies*, honest for same-file before/after comparison and for nothing finer — do not
quote them as AST counts. Two rules:
- **Compare against a reference where one exists.** If you hold the vendor's decoded output or a
known-good decode of one function, diff against it; residue dropping while diverging from the
reference is a loss.
- **A batch that did not move the metrics is a signal to stop**, not to re-run it louder — the same
two-strike shape as `SKILL.md` §Stop conditions, at pass granularity.
## 7. Extending the pattern library — the three-piece rule
Adding a family adapter means adding **all three, in the same change**: the rule document
(a `patterns/<family>.md` note — shape, safety boundary, measured caveats; the directory starts
empty and fills only when a real family is absorbed), the detector entries (hint tokens + content
regexes in `scripts/js_obf_detect.py`), and the pipeline entry (pass order in its `FAMILIES`
record). An adapter that exists in only one of the three places is invisible,
undetectable, or unmaintainable — and §1's promotion rule still governs what may reach the generic
layer.
@@ -7,9 +7,11 @@ receives a **giant literal** — thousands of hex characters or a base64 blob
globals, and everything behind it is one closed interpreter. That shape is **jsvmp**: the original
logic compiled to a private opcode stream and executed by a virtual machine shipped with the script.
This file owns reading it. The neighbouring layers own the rest: `references/emulation-and-rpc.md`
owns **calling** the routine instead of reading it, `references/vmp-differential-analysis.md` the
dex-layer VMP, `references/native-vm-recovery.md` the native interpreter loop, and
`references/framework-runtimes.md` deciding that the logic lives in JS at all.
owns **calling** the routine instead of reading it, `references/js-ast-deobfuscation.md` owns the
**pre-deobfuscation pipeline** (detection, pass order, safe-rewrite rules — the step §3.1 names),
`references/vmp-differential-analysis.md` the dex-layer VMP, `references/native-vm-recovery.md` the
native interpreter loop, and `references/framework-runtimes.md` deciding that the logic lives in JS
at all.
**Claim strength.** Absorbed from three public write-ups of one reverse-engineering series
(2024-03..2024-09; titles and authors anonymized in this record). One builds a minimal JS VM to
@@ -57,6 +59,39 @@ Two cautions. A big `switch` alone proves nothing (a state machine looks the sam
**read a cursor from a data buffer, dispatch on the value, and write the cursor back**. And the
presence of a readable interpreter does not mean you must read it — §3 first.
**A parameter-name index for the families this repository has on record** (`unverified` —
absorbed 2026-10-20 from a public JS-reverse case corpus; wire-level parameter names are kept as
the recognition signal, vendors anonymised per this repository's desensitization convention; none
of these was met on a real target here). Recognition aid only — confirm by §2's structural signals
before routing on it:
| Wire parameter / marker | Family shape | Typical carrier |
|---|---|---|
| `a_bogus`, `X-Bogus` | layered sign params, heavy env probing | short-video platforms, web + app |
| `mtgsig` | mini-program VMP with native assist | delivery/local-services apps |
| `sensor_data` | device-fingerprint collection + packing | risk-control SDKs (app-embedded) |
| `blackbox` | fingerprint blob, env-sensitivity | risk-control SDKs (app-embedded) |
| `h5st` | staged params with time windows | large e-commerce clients |
| `x-s`, `x-t` pair | jsvmp-generated sign pair | lifestyle-community platforms |
| `__zse_ck` | JS + wasm nested obfuscation | Q&A communities |
| `acw_sc__v2` | JS-computed cookie on first visit | e-commerce/ACW-fronted sites |
| `F14B3CEF*`-prefixed cookies, `$_ts` | 4/5/6-generation dynamic JS ("RS") | ticketing/airline/gov sites in WebView |
| `armorToken`, `Anti-in` | request-header token pair | fashion e-commerce |
| `aws-waf-token` | JS challenge token | AWS-WAF-fronted endpoints |
| `Reese84`, `___utmvc` | obfuscator.io-lineage challenge | airline/CDN-fronted sites |
| `hexin-v` | sign with heavy obfuscation | stock-data platforms |
| `ssxmod_itna` | multi-cookie interdependent chain | travel/booking sites |
A hit in this table says "expect a known shape, budget accordingly" — it does **not** say the
algorithm is known, and it never overrides a structural check. Two or three of these families show
up inside APKs as embedded WebView/H5 risk pages and mini-program containers rather than as the
app's own code — which is exactly why they surface here and not in `detection-and-anti-analysis.md`.
**Provenance note.** The A/B/C ladder at the end of §3 and this table are the only parts of this
file absorbed from outside the three write-ups the rest cites: both come from
`715494637/reverse-skill` (`jsr-reverse` skill; the repository ships no license, so methodology
only was absorbed, no text), retrieved 2026-10-20.
## 3. The cheaper question first: restore, fill the environment, call it
The stack-machine write-up's outcome, in its own order, and the step most skip:
@@ -65,7 +100,9 @@ The stack-machine write-up's outcome, in its own order, and the step most skip:
propagation. This halved the sample (~9,600 lines → roughly half) and made every later step
debuggable in an IDE instead of a browser's obfuscated view (the same function was 3,000+
unreadable lines in devtools). Debugging un-restored code in the browser is the slow path; do
not attempt VM analysis inside it.
not attempt VM analysis inside it. **Run as a pipeline, not ad hoc:**
`references/js-ast-deobfuscation.md` owns the family detection, pass order and safe-rewrite
rules, and `scripts/js_obf_detect.py` scores the sample.
2. **Fill the environment** ("补环境"): give the script the `window`/`document`/`navigator`/canvas
surface it probes. Canvas fingerprints are **hardware-linked** — vendors have collected real
device fingerprints to reject forged ones; a random canvas value may still be accepted (the
@@ -79,6 +116,28 @@ deliverable: a harness that runs the guest in your controlled environment. Read
you must understand or alter the algorithm itself, or when the environment the VM demands is more
expensive to fake than the VM is to read.
### How deep to go — the A/B/C ladder (absorbed; `inferred`)
"Call it first" is level **A**, and most tasks never leave it. The ladder below makes the upgrade
a *decision with evidence*, not a mood — its shapes come from a public JS-reverse recovery-strategy
note (see provenance above), and they match what the three write-ups actually did:
| Level | What you recover | You are here when… | You leave when… |
|---|---|---|---|
| **A — call the guest, extract only the opcodes that touch the target field** | the harness + a handful of critical-opcode semantics | the write-back boundary is known and a few opcodes explain the field | — (the deliverable is the harness) |
| **B — recover the dispatcher plus the critical state carriers** | the dispatcher loop, register/stack/context carriers, opcode families with state flow | a critical opcode cannot be interpreted without dispatcher semantics; the field explanation depends on register/stack flow; key branches cannot be judged without the carriers | never jump here because the code is *ugly* — ugliness is not an evidence type |
| **C — minimal decompilation / minimal re-interpreter** | a smallest verifiable re-execution of the needed path | multiple paths must be re-played, or batch/protocol rebuild requires an executable fragment, and A+B demonstrably cannot support it | — |
**Hard rules.** Start at A; never jump to C for aesthetics. A→B requires a *proven* dispatcher or
state-carrier dependency, not a suspicion. B→C requires a named downstream need (replay, batch,
protocol rebuild) — "the write-up rebuilt the whole ISA" is not that need until your task has one.
And the **black-box reuse boundary** decides whether reading is needed at all: reuse the module
whole when its input/output boundaries are known and the difficulty is the container or bridge
rather than the operator; recover deeper when the module rides heavy implicit shared state, replays
are unstable without the state carrier, or the module is *itself* another jsvmp. Write the decision
down in four lines — reuse mode / known inputs / known outputs / still-dependent state carriers —
so "we stopped at A" reads as a conclusion rather than a shortcut.
## 4. Locating the interpreter and clearing the porch
- **Find the entry by the literal, not by names.** Search for the huge constant being passed; follow
@@ -238,6 +297,7 @@ integers → ~14,000 instructions, with constants visible as bulk stores into fi
| Need | Tool / file |
|---|---|
| decode a jsvmp program literal (base64 → varint → ZigZag) | `scripts/jsvmp_loader_decode.py` (`--selftest` proves the round trip) |
| classify an obfuscated JS sample and get its pass order | `scripts/js_obf_detect.py`, then `references/js-ast-deobfuscation.md` |
| decide the logic lives in JS at all | `references/framework-runtimes.md` (runtime table; zero-hit Java hooks) |
| call the routine instead of reading it (the §3 route) | `references/emulation-and-rpc.md`; Frida RPC over a WebView/engine boundary |
| reach a JS engine behind JNI from the device | `references/dynamic-frida.md`; the JS bridge inputs are in `references/system-ipc-audit.md` |
@@ -252,6 +312,7 @@ the G1 sentence accordingly before sinking days into the ISA.
- [ ] entry recognised by the literal + dependency-array shape, not by a big `switch` alone
- [ ] §3 executed in order — AST restore, environment fill, call the function — before any VM reading
- [ ] the A/B/C stop depth stated with its evidence (which field-explanation or replay need forced the level) and the reuse boundary written down
- [ ] environment patches labelled: each one can fail server-side later (canvas especially)
- [ ] guard closures simplified with reference-checked replacements; "dead" branches probed, not cut
- [ ] stack vs register decided by the state idiom (§2 table), then the naming map applied
@@ -0,0 +1,96 @@
# Kotlin name recovery — original class names survive R8 inside annotations
**Load this when** you are reading a decompiled Kotlin app and the class names are meaningless
(`nq.e`, `a`, `C0123x`) — before you conclude the code is "unreadable" and before you sink hours
into cross-referencing by shape. One pass over the export recovers the *original* names for a
large share of the classes you actually want to read.
**What this answers:** which original class each obfuscated class was — from data R8 cannot strip,
not from guessing. What it does **not** answer: method names and field names (never recovered —
see Limits).
Contents: §1 why the leak exists · §2 the three sources · §3 the shipped script · §4 what you
typically get · §5 how it plugs into this kit's index-first discipline · §6 limits.
**Provenance.** Absorbed 2026-10-20 from `SimoneAvogadro/android-reverse-engineering-skill`
(Apache-2.0), its `recover-kotlin-names.sh` and `references/kotlin-name-recovery.md`; re-implemented
here as a dependency-free Python script. The recovery-rate numbers in §4 are the upstream author's
claims — `unverified` here. The script's parsing is `observed` against its own embedded fixtures
only (`--selftest`); no real obfuscated app was run through it in this repository.
## 1. Why the names survive
R8/ProGuard rename JVM symbols, but the Kotlin runtime needs metadata at runtime — for reflection,
coroutines (`@DebugMetadata` drives coroutine stack-trace recovery), and `data class` / serialization
behaviour (`@Metadata`). The annotations are **values the runtime reads**, so obfuscators keep them,
and the values contain the original fully-qualified names. This is a leak by necessity, the same
class of fact as "R8 cannot rename what the manifest references" (`recon.md` §5's anchor idea).
## 2. The three sources, in trust order
1. **`@DebugMetadata(c = "...")`** — emitted for nearly every `suspend` function's `SuspendLambda`
subclass. The `c` value is the original FQN **with `$`-suffix scope**:
`com.example.account.AccountRepositoryImpl$fetch$1` means the declaring class is everything
before the first `$`. Highest precision: a true declaring-class name, one per coroutine lambda —
and modern Kotlin apps are full of them.
2. **`@Metadata(d2 = {...})`** — every Kotlin class carries it. The `d2` array lists the file's
internal class references as JVM descriptors (`Lcom/example/Foo;`); the **first non-stdlib,
non-framework descriptor** is usually the file's primary class. A hint, not a declaration.
3. **jadx `/* renamed from: X */`** — jadx's own rename bookkeeping, emitted when `--deobf` or a
mapping renamed the class. Weakest: it records what *jadx* renamed from, which for an R8 target
is the obfuscated name — useful mainly for jadx-generated names (`C0123x`).
## 3. The shipped script
```bash
python scripts/kotlin_name_recovery.py <jadx-export>/sources # build the map
python scripts/kotlin_name_recovery.py --map <map-dir> -o nq.e # resolve one name
python scripts/kotlin_name_recovery.py --map <map-dir> -p com.example.feature # list a package
python scripts/kotlin_name_recovery.py --map <map-dir> --grep '"api/' <jadx-export>/sources
```
Outputs: `mapping.tsv` (`obf_fqn real_fqn source file`), `mapping.json`, and `by_package/`
(one file per original package). The `--grep` mode is the everyday one: it greps the export and
annotates **every hit with the real owning class**, so the export becomes navigable without opening
files one by one. Third-party trees (`kotlin.*`, `androidx.*`, `okhttp3.*`, …) are skipped — their
names are already real.
Resolving `-o nq.e` also prints **siblings** under the same short prefix (`nq.d`, `nq.f`): they are
frequently one original class's split lambdas/inner classes, which is often the context you wanted.
## 4. What you typically get — upstream's claims, `unverified`
| Class kind | Recovery |
|---|---|
| `*Repository` / `*Impl` / `*ViewModel` / `*UseCase` | ~100 % |
| plain `data class` DTOs | ~80 % |
| share of all classes in a real app | 30–50 % |
The shape is the point: the classes a *reading* task targets are exactly the annotated ones, so the
effective recovery on "the classes you came for" is much higher than the global share. Treat the
numbers as directional; this repository has not measured them.
## 5. How it plugs into this kit's order of work
This is a **reading aid for the export tier**, not a replacement for the index tier
(`SKILL.md` §Tooling — orient with an indexer, not an export): locate first with `rasc findrefs` /
`dexsql query`, then read the located class in the export with names restored. Two combinations that pay:
- **Mapping onto query results:** the index hands you `nq.e`; `-o nq.e` tells you it is
`com.example.feature.account.AccountRepositoryImpl` — which usually *is* the layer answer
(`recon.md` §3's "where does the behaviour live" becomes a lookup instead of an inference).
- **Endpoint hunting:** `--grep '"/' <export>` over the app's own packages gives every path literal
with its real owning class attached — a cheap first pass for `server-api.md`'s surface list.
## 6. Limits — say them before trusting a mapping
- **Method and field names are never recovered.** Metadata preserves class-level FQNs and some
signatures, not members. For members you still need call-site evidence (`find_refs.py`,
`dexsql` callers) or jadx-gui's interactive rename.
- **Pure-Java classes carry no `@Metadata`** and stay obfuscated; the fallbacks do not see them.
- **Heavily inlined shapes** (`@JvmInline value class`, top-level functions compiled into shared
`*Kt` classes) can surface under a wrong primary class; every mapping row is a hint, and §5's
cross-checks are the antidote.
- The script reads **decompiled sources**, not the dex: annotations are read as jadx printed them,
so a decompiler that drops annotations removes the source entirely. (The raw dex annotation
sections carry the same strings — `dexsql`'s annotation tables — as a cross-check when in doubt.)
@@ -305,7 +305,56 @@ therefore: identify the client first (`tls-and-cert.md` §Step 4 attribution), t
mechanism. Patching a library that was never consulting the store is work with no effect; installing
a system CA when a native verifier ignores it produces the same empty capture you started with.
## 6. Failure modes
## 6. WebSocket and long-lived connections — state chains, not packets
**Load this when** the behaviour you care about rides one long connection instead of request/response
pairs: a WebSocket (OkHttp `WebSocket`, `Java-WebSocket`, a native socket), SSE, or an app-held
TCP/QUIC stream. The unit of analysis changes: a single captured frame says almost nothing, because
its meaning is a function of the connection's accumulated state.
**Claim strength.** `unverified` as a method — absorbed 2026-10-20 from `715494637/reverse-skill`
(`jsr-reverse`'s long-connection reference; no license there, methodology only), where it was
written for browser WebSocket targets; the Android capture points below are `inferred` from
documented APIs, and nothing here was run on a real long-connection app in this repository.
**Capture more than the first packet.** The expensive mistake is analysing one business frame with
no idea which state produced it. Minimum evidence set, in order: **handshake** (and its response —
it usually mints the session), **first business frame**, a **heartbeat/keepalive** pair, a
**seq/ack** exchange, a **renewal/refresh** if tokens expire, and an **error/retry** frame if one
exists. Each later frame family is only interpretable against that chain.
**Record the connection as a state chain, not a frame list.** Write the dependencies down as a
chain — handshake response grants session → heartbeat carries seq → ack confirms → token near
expiry triggers renewal → renewal failure forces reconnect/risk state — because the question that
actually decides replayability is *which prior state a frame consumes*. A frame that embeds the
session id, a monotonic seq, and an ack of the previous server frame is **not replayable alone**;
that is a finding about the protocol, not a failure of your capture.
**Separate message families before naming anything.** Handshake/auth · heartbeat/keepalive ·
business · ack/retry/error · renewal. Only the business family is your target; the others are the
state machinery you must reproduce to *reach* the business family. Mislabelling a heartbeat as a
protocol error is the standard first misread.
**Split the envelope from the payload — envelope first.** Long-connection payloads are usually
framed: message **type**, **length**, **seq**, **time window**, session identifier, and
compression/encryption flags wrap the body. Recover that contract before touching body semantics
(and the body itself is very often a protobuf message — §1's decoder and the round-trip check apply
unchanged). A body read against a wrong envelope alignment produces plausible garbage, the same
failure as §7's wrong frame split.
**Android capture points** (`inferred`): an OkHttp `WebSocket` surfaces through its listener and
`okhttp3.internal.ws` — hookable from Java (`dynamic-frida.md`); a native or engine-internal socket
(QUIC, a game/push SDK) may never transit Java at all, and the §5 attribution rules apply — check
what is mapped (`scripts/lib_map.py`) before assuming you missed a hook. Note the transport itself
can be pinned exactly like §5's native verifier, and `tls-and-cert.md` still owns the trust-chain
attribution.
**Stop rule.** Stop when you can name the state chain, the message families, and the target
frame's dependency set, and the envelope decodes stably across several samples. If the real blocker
is still "which state did this frame consume", no amount of body decoding helps — go capture the
missing family first.
## 7. Failure modes
**A decode that looks valid but is not.** protobuf is self-synchronising enough that random bytes
often decode into a plausible-looking field list. Guard against it by (a) decoding several samples of
+2 -1
View File
@@ -197,9 +197,10 @@ Collect: base URLs, DoH/DoT lookups, path constants, CDN hosts, custom headers.
Treat **unique** strings as navigation aids — they can often be searched byte-wise in the dex to find the owning class (see `references/dex-patching.md` §finding-the-call-site).
Two patterns worth recognizing early:
Three patterns worth recognizing early:
- **Dynamic gateway**: the real API host is fetched at runtime (e.g. via a DNS TXT record over DoH). The hardcoded host in dex may be only a fallback. This affects server-side analysis, not patching.
- **Minimal headers / no request signing**: if auth is just `Authorization: Bearer` plus a static app-name header, then a repackaged client is not distinguishable to the server by request signature — which matters for `references/server-api.md`.
- **BuildConfig sweep** (absorbed 2026-10-20 from `SimoneAvogadro/android-reverse-engineering-skill`, Apache-2.0; the habit is `inferred`, its yield unmeasured here): every Gradle module compiles one generated `BuildConfig` class, and they are a cheap first source of `BASE_URL`/`ENDPOINT` constants, API keys, flavor flags and debug switches — one `--find 'BuildConfig|BASE_URL|FLAVOR'` pass over the string dump. Generated constants are inlined at their use sites even when the class itself is renamed, so the *values* surface in `--urls` output regardless; the sweep is what attributes them to a module.
## Recon output
+10 -5
View File
@@ -75,15 +75,16 @@ reference file is not named here, or when a script is not named here.
| The trace segments themselves look corrupted — handler heads missing, tails glued into wrong segments | `references/native-vm-recovery.md` §7 Nested VMs — a flat segment accumulator discards the outer level; nested slices need a segment stack (`vm_handler_slice.py` flags the symptom) |
| You need to know **where two runs of the same operation diverge** (password/license/key), not just what executed | `ete-diff-tracing.md` — ETE hardware-trace gates plus the differential method; `scripts/trace_diff.py` also diffs Stalker logs |
| A generated KPM/LKM template is in front of you, or the kernel route feels like the plan | `kernel-event-injection.md` — the primitive map, the composition recipes, and the gates every kernel-side file must pass before it is a weapon |
| The traffic is protobuf/gRPC/QUIC, or a proxy sees TLS but requests still fail on a Flutter app | `references/protocol-reverse.md` — schema-less protobuf, frame capture, and native-side pinning |
| The traffic is protobuf/gRPC/QUIC, a proxy sees TLS but requests still fail on a Flutter app, **or the behaviour rides a long-lived WebSocket** | `references/protocol-reverse.md` — schema-less protobuf, frame capture, native-side pinning, and the long-connection state chain |
| Userspace hooks land and the app still dies: the check reads `/proc/self/status` through a raw `svc`, or runs before `JNI_OnLoad` | `references/kernel-and-environment-hardening.md` — what the next layer up and down can actually do, and when to stop |
| You must edit, repack, sign or inspect the APK **from the phone itself** | `references/on-device-tooling.md`, `scripts/mt_mcp_probe.py` |
| A captured body decodes to nothing readable, or you cannot tell whether a length-delimited field is a string, a nested message or a packed array | `protocol-reverse.md`. Protobuf on the wire (measured) — run `scripts/protobuf_decode_raw.py`; the candidate list and its `tie:` lines are the answer |
| Method bodies are present but decode as **private opcodes**, and you need the mapping rather than an explanation of why VMP is hard | `vmp-differential-analysis.md`, then `advanced-unpacking.md` for the shape diagnosis |
| The value you need (sign, token, risk parameter) is computed in **JavaScript** — a WebView or embedded JS engine — by an entry call fed a huge hex/base64 opcode literal: a JS virtual machine | `jsvmp-recovery.md` — de-obfuscate and **call it** first, then the stack/register-VM procedures and the four JS traps; `emulation-and-rpc.md` owns the call-don't-read route |
| You are reading a decompiled **Kotlin** app and every class name is a meaningless short token (`nq.e`, `C0123x`) | `kotlin-name-recovery.md` — the original FQNs survive R8 inside `@DebugMetadata`/`@Metadata`; `scripts/kotlin_name_recovery.py` rebuilds the mapping from any jadx export |
| A store build arrives as `base.apk` + `split_config.*.apk`, or a rebuilt build is refused **as a set** although every file verifies on its own | `split-apk.md` — one keystore across every member for `pm install-multiple`, and check that a merge is legal before trusting a merged single APK |
_55 row(s) below the header._
_56 row(s) below the header._
## Reference index (no longer inline in `SKILL.md`)
@@ -97,7 +98,9 @@ _55 row(s) below the header._
| `references/code-virtualization-and-custom-linkers.md` | **No packer, dex is readable, and a re-signed build still dies** — whole classes turned `native`, a private loader with a mismatched SONAME, a self-decrypting payload, a Java-layer "signature killer", and the keep-it/drop-it deadlock |
| `references/java2c-and-jni-sinking.md` | **Whole classes are bare `native` declarations and you are about to hunt for a decrypted DEX** — Java2C has none, ever; the code is in a `.so`. Also the JNI boundary: why a `Java_*` symbol search fails silently |
| `references/vmp-differential-analysis.md` | Method bodies are present but decode as **private opcodes** — a real Dex VMP: the known-plaintext differential, which links can and cannot be automated, how to *prove* a derived opcode table, smali generation, and when the route is closed |
| `references/jsvmp-recovery.md` | The protected logic is **JavaScript** (WebView, embedded JS engine, mini-program): a jsvmp opcode literal plus an interpreter — recognition, the call-it-first route, stack- and register-VM reading, instruction extraction, the four JS-language traps, and the shipped loader decoder |
| `references/jsvmp-recovery.md` | The protected logic is **JavaScript** (WebView, embedded JS engine, mini-program): a jsvmp opcode literal plus an interpreter — recognition and the wire-parameter family index, the call-it-first route with its A/B/C depth ladder, stack- and register-VM reading, instruction extraction, the four JS-language traps, and the shipped loader decoder |
| `references/js-ast-deobfuscation.md` | The JS in front of you is unreadable through **structure** — string tables, decoder IIFEs, flattened control flow — and `jsvmp-recovery.md` §3's "AST pre-deobfuscation" step is next: family detection with `scripts/js_obf_detect.py`, the per-family pass order, safe-rewrite rules, minimal bootstrap eval, residue proof |
| `references/kotlin-name-recovery.md` | You are reading a decompiled **Kotlin** app and the class names are meaningless short tokens: the original FQNs survive R8 inside `@DebugMetadata` and `@Metadata` — `scripts/kotlin_name_recovery.py` builds the obfuscated→original map from any jadx export, with `--grep` annotating every hit by its real class |
| `references/framework-runtimes.md` | The UI is not native (Flutter / React Native / Unity / Cordova), or Java-layer hooks fire zero times while the UI clearly works |
| `references/dart-aot.md` | The logic lives in a Dart AOT snapshot (`libapp.so`): pinning the Dart version, building a matching decompiler, the object pool and reference indexing, register/boolean conventions, locating and patching Dart code |
| `references/native-and-so.md` | Patching in a `.so`, needing code to run before the app's own code, hand-built native payloads that crash inside the linker, or **deciding which library/ABI is actually loaded and executing** |
@@ -145,7 +148,7 @@ _55 row(s) below the header._
| `references/on-device-tooling.md` | Working **from the phone itself**: MT Manager edit/repack/sign and its built-in APK MCP, LSPosed Manager, Termux+frida, and on-device data inspection |
| `references/split-apk.md` | The target is a **split APK / App Bundle set** (`base.apk` + `split_config.*.apk`), or `pm path <PKG>` returned several files: reading a set, merge versus unified re-signing, and the install refusal each mistake produces |
_44 row(s) below the header._
_46 row(s) below the header._
## Script index (no longer inline in `SKILL.md`)
@@ -163,6 +166,7 @@ _44 row(s) below the header._
| `scripts/dex_strpatch.py` | Byte-level string constant patch with **string_ids ordering guard** |
| `scripts/dex_classdiff.py` | Compare two dex class tables (set + access flags) to prove a patch was surgical |
| `scripts/dex_strings.py` | Dump/extract strings and endpoints from dex without a decompiler |
| `scripts/kotlin_name_recovery.py` | Rebuild the **obfuscated→original Kotlin class-name map** from a jadx export: `@DebugMetadata`/`@Metadata.d2`/jadx rename comments, third-party trees skipped, `mapping.tsv`/`json`/`by_package/` outputs, then `-o`/`-p`/`--grep` query modes that annotate every grep hit with the real owning class; `--selftest` pins the three sources |
| `scripts/dart_pprefs.py` | Build/query the object-pool offset -> code-site index for a Dart AOT snapshot (arithmetic decode; seconds, not minutes) |
| `scripts/dart_pool_strings.py` | Recover string literals from a Dart AOT snapshot: framed entries, the one-byte vs UTF-16 split, file offsets, and a run-length noise filter |
| `scripts/dart_disasm.py` | Annotated windowed disassembly of Dart AOT code (pool + boolean annotations) plus a B/BL caller index |
@@ -201,6 +205,7 @@ _44 row(s) below the header._
| `scripts/protobuf_decode_raw.py` | **Schema-free protobuf decode**: hex / file / stdin to a JSON tree, every length-delimited field as a candidate set with ties labelled; `--reencode --check` for a byte-exact round trip |
| `scripts/vmp_diff_harness.py` | Differential hardening for Dex-VMP: build a labelled opcode-coverage fixture (**218 of 224 opcodes, measured**), derive a candidate private-opcode map from an original/hardened dex pair, verify it in a closed loop, render smali |
| `scripts/jsvmp_loader_decode.py` | Decode a **jsvmp** program literal (base64/hex → varint → ZigZag → signed opcode stream) with offsets, JSON output and a round-trip `--selftest`; raises on truncated input rather than decoding short |
| `scripts/js_obf_detect.py` | **Score a JS bundle's obfuscation family** (hint tokens weight 10, content regexes weight 3: string-array lineage, control-flow flattening, jsvmp entry, opcode if-chains), get the per-family pass order for `js-ast-deobfuscation.md`, plus textual residue metrics for before/after comparisons; `--selftest` pins the fixtures |
| `scripts/kernelsu_syscall_mask.py` | Generate a KernelSU/APatch syscall-masking scaffold: an installable userspace module plus KPM/LKM/eBPF kernel-side templates, each with its version gate and an explicit unverified label. **A userspace module cannot change a syscall return value** |
| `scripts/hw_watch.py` | **Invisible memory watchpoints / execution breakpoints** on arm64: plan (WCR/BCR encoders with the 8-byte-window rules, host-tested), emit (self-contained on-device C helper, compiles clean as static aarch64), gates, run (build+push+stream hits), decode (module+offset per hit). ptrace route: `TracerPid` stays visible — the ledger is in the reference |
| `scripts/trace_diff.py` | **Differential trace reduction**: reduce one trace to runs+histogram, or diff two traces into first-divergence / divergent regions / blocks-unique-to-each-side. Producer-agnostic (Stalker `BLK` logs or any address-per-line stream, incl. decoded cs_etm) |
@@ -215,4 +220,4 @@ _44 row(s) below the header._
| `scripts/svc_scan.py` | Name the syscall behind an inline `svc` and the segment it sits in — which decides whether a libc-level hook can observe the call at all. `--context` shows neighbours, because a byte scan also matches data |
| `scripts/anti_detect_probe.js` | Observer-only probe (patches nothing): path/loader/thread/kill hooks with **caller module + offset**, an environment self-report (`TracerPid`, frida-named mappings), and live streaming so a sub-second target still yields evidence |
_54 row(s) below the header._
_56 row(s) below the header._
+29 -1
View File
@@ -37,10 +37,38 @@ cross-check with the program headers.
| **`ddc`** | `ddc app.apk -c <Class>` | **fastest read of dex as Java, plus query subcommands** — reach for this SECOND, to *read*. See below |
| **`dexsql`** | `dexsql query --input app.apk --sql "…"` | **the whole dex as SQL tables** (callers, field accesses, numeric constants, debug lines, annotations) plus manifest and `resources.arsc` queries and an MCP server. Reach for it when the question is a **join or an aggregate** rather than a class list — see below |
| `apktool` | `java -jar apktool.jar d/b` | whole-app decode including resources |
| `jadx` | `jadx --no-res -d <out> <apk>` | readable Java for orientation; **not** a source of truth, and **not** a recon entry point (see below) |
| `jadx` | `jadx --deobf -d <out> <apk>` | readable Java for orientation; **not** a source of truth, and **not** a recon entry point (see below) |
| Fernflower / Vineflower | `java -jar vineflower.jar -dgs=1 -mpm=60 …` (via dex2jar) | the **second engine** for the classes jadx mangles — see below |
| `aapt2` | `aapt2 dump badging <apk>` | manifest facts, package name, versions |
| `zipalign`, `apksigner` | from build-tools | alignment and signing |
### jadx and a second engine — the readable-Java tier (`unverified` unless stated)
jadx stays what the table says: a viewer of located classes. Four flags change what you get,
absorbed 2026-10-20 from `SimoneAvogadro/android-reverse-engineering-skill` (Apache-2.0):
- **`--deobf`** renames obfuscated identifiers to *synthetic* readable ones and writes the mapping
to `<out>/deobf-mapping.txt`. It does **not** recover original names — for a Kotlin target the
original names live in annotations, and `references/kotlin-name-recovery.md` recovers them.
- **`--deobf-map <mapping.txt>`** consumes a real ProGuard/R8 mapping — and an APK sometimes
**ships one** (search `assets/` and the dex strings for `mapping.txt`/`.map` before assuming
there is none). When a mapping exists this beats every heuristic.
- **`--show-bad-code`** emits partially-decompiled bodies instead of error comments — useful
exactly where you need to see *something* of a class jadx struggles with.
- **`-e` / `--export-gradle`** exports a Gradle project (IDE navigation); **per-dex targeting**
(`unzip` the APK, run jadx on one `classes*.dex`) keeps cost down on multi-dex giants.
**Fernflower / Vineflower as the second engine.** Where jadx's output on a class is wrong or
empty (complex lambdas, generics, streams — the shapes a Kotlin app is full of), the analytical
decompiler often produces a readable body jadx cannot. It reads class files, not dex: the chain
is `d2j-dex2jar -f -o app.jar app.apk` → `java -jar vineflower.jar -dgs=1 -mpm=60 -ren=1
-e=$ANDROID_HOME/platforms/android-<N>/android.jar app.jar out/` → unzip the source jar. `-dgs=1`
decompiles generic signatures, `-mpm=60` bounds each method (a decompiler hang becomes a skip),
`-e=android.jar` gives it the framework types. **Run both engines and compare** when a class is
load-bearing: two independent bodies agreeing is cheap corroboration a single body cannot give.
All of this tier is `unverified` here — no APK was decompiled by this route in this repository —
and it stays a *reading* route: the patch decision still belongs to the byte/smali tier above.
### rasc — the same index in Rust, when you have built it
`rasc` is the `rust` branch of the same project (`MG1937/ASC`). Same CLI shape — `classes`,
+278
View File
@@ -0,0 +1,278 @@
#!/usr/bin/env python3
"""Detect which obfuscation family a JS bundle belongs to, by weighted content probes.
This is the detection layer of the JS AST pre-deobfuscation route documented in
`references/js-ast-deobfuscation.md`: before running any rewrite pass you want to know
*which family* you face, because the pass order differs per family (a wrong order stalls
or corrupts -- the absorbed architecture's own measured example: late literal-inlining
hangs on large VM-heavy samples, so that family skips it).
Design (absorbed, see the reference for provenance): every family carries
* ``hint`` tokens -- matched against the file path and the user-supplied ``--hint``
text; a hit scores 10, because a human saying "this is the delivery platform's SDK"
is stronger evidence than any content regex;
* ``content`` regexes -- matched against the source text; each hit scores 3.
Any family scoring > 0 is a detection; the top scorer is the recommended pipeline. No
hit means the generic pipeline, which is also the escalation target when residue stays
stable after the family passes.
What this script deliberately does NOT do: rewrite anything. The passes need a real JS
parser (node + @babel/parser + traverse + generator); this skill ships the detector and
the pass order, not the rewriting engine -- the fence is stated in the reference.
The residue metrics are *textual proxies* (identifier shapes, literal densities, chain
counts), not AST counts; they are honest for comparing before/after a pass batch on the
same file and for nothing finer.
Evidence: `observed` only for `--selftest` against embedded fixtures. No real obfuscated
target was ever run through this detector here; every threshold is a starting point, not
a boundary.
Examples:
python scripts/js_obf_detect.py bundle.js
python scripts/js_obf_detect.py bundle.js --hint 'delivery-platform sign sdk' --json
python scripts/js_obf_detect.py --selftest
"""
import argparse
import json
import os
import re
import sys
FAMILIES = [
{
"id": "ob_string_array",
"display": "string-array / obfuscator.io lineage",
"hint": ["obfuscator", "javascript-obfuscator", "ob.io"],
"pipeline": ["normalize", "prune_fake_branches", "string_array_pass",
"inline_dispatchers", "flatten_array_cff", "prune_fake_branches",
"normalize"],
"content": [
(r'\b_0x[a-f0-9]{3,}\b', "hex-mangled identifiers"),
(r'var\s+_0x[a-f0-9]+\s*=\s*\[', "top-level string array"),
(r'\b(?:push\s*\(\s*(?:shift|pop)\s*\(\s*\)\s*\)|shift\s*\(\s*\)\s*\))', "rotate/pull decoder idiom"),
(r'String\s*\.\s*fromCharCode', "charCode decoder"),
(r'\bparseInt\s*\(\s*[A-Za-z_$][\w$]*\s*\+\s*[^)]*?,\s*16\s*\)', "radix-16 decoder math"),
],
"notes": "Decode the string array before any control-flow work: every later pass "
"reads strings through it, and residue metrics on undecoded samples are noise.",
},
{
"id": "control_flow_flattening",
"display": "control-flow flattening (switch dispatch)",
"hint": ["flatten", "cff", "ollvm-js"],
"pipeline": ["normalize", "prune_fake_branches", "inline_dispatchers",
"flatten_array_cff", "if_chain_to_switch", "prune_fake_branches",
"normalize"],
"content": [
(r'while\s*\(\s*(?:!!\[\]|\(?!!\[\]\)?|!0|0x1|1)\s*\)\s*\{', "infinite-loop driver"),
(r'["\'][0-9]+\|[0-9|]+["\']\s*\.\s*split\s*\(\s*["\']\|["\']\s*\)', "sequence-string order table"),
(r'switch\s*\(\s*_?0x[a-f0-9]+', "switch on mangled var"),
],
"notes": "Recover the real block order from the state variable's assignment chain; "
"the dispatcher block is noise, not structure.",
},
{
"id": "jsvmp_entry",
"display": "jsvmp entry (opcode literal + interpreter)",
"hint": ["jsvmp", "js vmp", "sign", "risk"],
"pipeline": ["string_array_pass", "normalize", "prune_fake_branches",
"inline_dispatchers", "if_chain_to_switch"],
"content": [
(r'[_$][\w$]{0,12}\'?\]?\s*\(\s*[\'"][0-9a-fA-F]{%d,}[\'"]', "huge hex opcode literal"),
(r'[_$][\w$]{0,12}\'?\]?\s*\(\s*[\'"][A-Za-z0-9+/=]{%d,}[\'"]', "huge base64 opcode literal"),
(r'\[\s*(?:Object|Function|Math)\s*,', "globals dependency array"),
(r'_\$\w+', "_$-prefixed vendor identifiers"),
],
"notes": "Do NOT beautify the interpreter into 'readable' form before reading the "
"dispatcher: the pass order here stops at making the dispatch readable. "
"See references/jsvmp-recovery.md for the VM work itself.",
# The huge-literal thresholds are parameters, not constants: a selftest or a small
# sample legitimately lowers them.
"literal_min": 1500,
},
{
"id": "opcode_if_chain",
"display": "opcode if-chain dispatch",
"hint": ["if chain", "opcode chain"],
"pipeline": ["normalize", "prune_fake_branches", "if_chain_to_switch",
"prune_fake_branches", "normalize"],
"content": [
(r'if\s*\(\s*(?:0x[0-9a-f]+|\d+)\s*[=!]==?\s*[A-Za-z_$][\w$]*', "literal-vs-var comparison"),
(r'else\s+if\s*\(\s*(?:0x[0-9a-f]+|\d+)\s*[=!]==?\s*[A-Za-z_$][\w$]*', "chained else-if"),
],
"notes": "Rewrite to switch ONLY when the whole chain tests one discriminant against "
"literals; a mixed chain left as-is beats a mangled rewrite.",
},
]
def score_families(source_text, hint_text, path_text, literal_min):
"""Return [detection dicts] sorted by score desc then id; every family with score > 0."""
detections = []
haystack = " ".join(filter(None, [hint_text, path_text])).lower()
for fam in FAMILIES:
score = 0
reasons = []
for token in fam.get("hint", []):
if token.lower() and token.lower() in haystack:
score += 10
reasons.append("hint:" + token)
for rx_src, what in fam["content"]:
rx_src = _bind_literal_min(rx_src, literal_min)
if re.search(rx_src, source_text):
score += 3
reasons.append("content:" + what)
if score > 0:
detections.append({"id": fam["id"], "score": score, "reasons": reasons})
detections.sort(key=lambda d: (-d["score"], d["id"]))
return detections
def _bind_literal_min(rx_src, literal_min):
"""Families declare huge-literal thresholds with %d so callers can lower them."""
return rx_src % literal_min if "%d" in rx_src else rx_src
def residue_metrics(source_text):
"""Textual proxies for 'how obfuscated does this still look'. Compare same-file only."""
lines = source_text.splitlines()
identifiers = set(re.findall(r'\b_0x[a-f0-9]{3,}\b', source_text))
hex_lits = re.findall(r'\b0x[a-fA-F0-9]{2,}\b', source_text)
comma_expr = re.findall(r'(?<=[;{}])\s*[^;{}()]{1,80},[^;{}]{1,80}\s*(?=[;{}])', source_text)
switches = len(re.findall(r'\bswitch\s*\(', source_text))
cases = len(re.findall(r'\bcase\s+', source_text))
longest = max((len(ln) for ln in lines), default=0)
evals = len(re.findall(r'\beval\s*\(', source_text))
return {
"lines": len(lines),
"mangled_identifiers": len(identifiers),
"hex_literals": len(hex_lits),
"comma_expressions": len(comma_expr),
"switches": switches,
"cases": cases,
"longest_line": longest,
"eval_calls": evals,
}
def pipeline_for(family_id):
"""The ordered pass list for a family (or the generic ladder when nothing scored)."""
fam = next((f for f in FAMILIES if f["id"] == family_id), None)
if fam is None:
return ["normalize", "prune_fake_branches", "inline_dispatchers",
"flatten_array_cff", "if_chain_to_switch", "prune_fake_branches", "normalize"]
return list(fam["pipeline"])
def notes_for(family_id):
fam = next((f for f in FAMILIES if f["id"] == family_id), None)
if fam is None:
return "Generic ladder only. Escalate to a family adapter when residue is stable and reproducible."
return fam["notes"]
FIXTURES = {
"ob_string_array": (
"var _0x4ab1 = ['\\x68\\x65\\x6c\\x6c\\x6f', 'push', 'shift'];\n"
"(function (_0x1, _0x2) { var _0x3 = function (_0x4) { while (--_0x4) { _0x1.push(_0x1.shift()); } }; _0x3(++_0x2); }(_0x4ab1, 0x1a4));\n"
"var _0x9f = function (_0x1, _0x2) { _0x1 = _0x1 - 0x0; var _0x3 = _0x4ab1[_0x1]; return _0x3; };\n"
"console[_0x9f('0x0')](_0x9f('0x1'));\n"
),
"control_flow_flattening": (
"function f(_0x1, _0x2) { var _0x3 = '3|1|2'.split('|'), _0x4 = 0; "
"while (!![]) { switch (_0x3[_0x4++]) { case '1': var _0x5 = _0x1 + _0x2; continue; "
"case '2': return _0x5; case '3': _0x1 = _0x1 | 0; continue; } break; } }\n"
),
"opcode_if_chain": (
"function d(o) { if (0x31 === o) { return 1; } else if (0x32 === o) { return 2; } "
"else if (0x33 === o) { return 3; } else if (0x34 === o) { return 4; } return 0; }\n"
),
}
def _jsvmp_fixture(literal_min):
literal = "ab" * (literal_min // 2 + 10)
return "global['_$webrt_%s']('%s', [Object, Math, setTimeout, void 0, false, entry]);\n" % ("d2a1", literal)
def selftest(literal_min):
"""Each embedded fixture must select its own family as the TOP pick.
Exclusivity is NOT asserted: families genuinely overlap by design (hex-mangled
identifiers appear in the CFF fixture too), so the test pins the winner, not the
loser list. The hint-beats-content rule (10 vs 3) and the plain-code-is-generic
case are pinned as well, because both decide real routing.
"""
for fam_id, body in FIXTURES.items():
detections = score_families(body, "", "", literal_min)
assert detections, "no detection for %s" % fam_id
assert detections[0]["id"] == fam_id, (fam_id, detections)
jsvmp_body = _jsvmp_fixture(literal_min)
detections = score_families(jsvmp_body, "", "", literal_min)
assert detections and detections[0]["id"] == "jsvmp_entry", detections
# A hint alone is worth 10 and must beat a single content hit (3).
hinted = score_families("plain text, nothing obfuscated", "delivery-platform obfuscator", "x.js", literal_min)
assert hinted and hinted[0]["id"] == "ob_string_array" and hinted[0]["score"] == 10, hinted
# Plain code selects nothing: generic ladder.
plain = score_families("function add(a, b) { return a + b; }\n", "", "", literal_min)
assert plain == [], plain
residues = residue_metrics(FIXTURES["ob_string_array"])
# The 3+ hex-char filter counts _0x4ab1 but not short names (_0x1, _0x9f) -- pinned so
# the filter does not silently loosen to counting every _0x-prefixed token.
assert residues["mangled_identifiers"] == 1 and residues["eval_calls"] == 0, residues
print("selftest: 4 family fixtures + hint-beats-content + plain-is-generic + residues: ok")
return 0
def main(argv=None):
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
ap.add_argument("input", nargs="?", help="JS source file to classify")
ap.add_argument("--hint", default="", help="free text: target name, vendor, parameter name")
ap.add_argument("--json", action="store_true", help="machine-readable output")
ap.add_argument("--min-literal", type=int, default=1500,
help="char length that makes a literal 'huge' (default 1500; lower for small samples)")
ap.add_argument("--selftest", action="store_true", help="run the embedded fixture tests")
args = ap.parse_args(argv)
if args.selftest:
return selftest(args.min_literal)
if not args.input:
ap.print_usage()
print("classify: script bundle.js [--hint TEXT] families: %s" %
", ".join(f["id"] for f in FAMILIES))
return 2
if not os.path.isfile(args.input):
print("not a file: %s" % args.input, file=sys.stderr)
return 1
with open(args.input, "r", errors="replace") as fh:
text = fh.read()
detections = score_families(text, args.hint, args.input, args.min_literal)
best = detections[0]["id"] if detections else "generic"
result = {
"input": os.path.abspath(args.input),
"best": best,
"detections": detections,
"pipeline": pipeline_for(best),
"notes": notes_for(best),
"residues": residue_metrics(text),
}
if args.json:
print(json.dumps(result, indent=1))
else:
print("family: %s" % best)
for d in detections:
print(" %-3d %-22s %s" % (d["score"], d["id"], ", ".join(d["reasons"])))
if not detections:
print(" (no family scored; generic ladder)")
print("pipeline order: %s" % " -> ".join(result["pipeline"]))
print("note: %s" % result["notes"])
print("residues: %s" % json.dumps(result["residues"], sort_keys=True))
return 0
if __name__ == "__main__":
sys.exit(main())
@@ -0,0 +1,317 @@
#!/usr/bin/env python3
"""Rebuild an (obfuscated -> original) Kotlin class-name map from decompiled sources.
R8/ProGuard rename JVM symbols but cannot strip the Kotlin metadata the Kotlin runtime
itself needs for reflection and coroutines, so two annotations survive into any decompiled
Java and still carry the *original* fully-qualified names:
* ``@DebugMetadata(c = "com.example.Foo$bar$1", ...)`` -- emitted for nearly every
``suspend`` function's SuspendLambda; everything after the first ``$`` is inner/lambda
scope, so the prefix is the declaring class.
* ``@Metadata(d2 = {"...", "Lcom/example/Foo;", ...})`` -- every Kotlin class; the first
non-stdlib ``L...;`` descriptor in ``d2`` is usually the file's primary class.
* jadx's ``/* renamed from: X */`` comments as a last-resort fallback.
This is a *reading aid*, not ground truth: it feeds names back into a jadx export you are
already navigating, and every mapping is a hint. Method and field names are NOT recovered --
metadata preserves class-level FQNs only.
Provenance: absorbed 2026-10-20 from SimoneAvogadro/android-reverse-engineering-skill
(Apache-2.0), its ``recover-kotlin-names.sh`` + ``references/kotlin-name-recovery.md``;
re-implemented here as a dependency-free Python script. The upstream recovery-rate claims
("30-50% of classes, ~100% of *Repository/*ViewModel/*Impl") are the upstream author's and
are ``unverified`` in this repository. The script itself is measured only by ``--selftest``
on embedded fixtures (``observed`` for the parsing, nothing more).
Examples:
python scripts/kotlin_name_recovery.py <jadx-export>/sources # build the map
python scripts/kotlin_name_recovery.py --map <map-dir> -o nq.e # obf -> real
python scripts/kotlin_name_recovery.py --map <map-dir> -p com.example # list a package
python scripts/kotlin_name_recovery.py --map <map-dir> --grep '"api/' <jadx-export>/sources
python scripts/kotlin_name_recovery.py --selftest
"""
import argparse
import json
import os
import re
import sys
import tempfile
from collections import defaultdict
# @DebugMetadata(c = "com.foo.Bar$Inner$1", ...) -- `c` carries the original outer FQN.
RE_DEBUG = re.compile(r'@DebugMetadata\([^)]*?c\s*=\s*"([^"]+)"', re.S)
# @Metadata(... d2 = { "...", "Lcom/foo/Bar;", ... } ...) -- internal class refs of the file.
RE_DTWO = re.compile(r'@Metadata\([^)]*?d2\s*=\s*\{([^}]*)\}', re.S)
# A JVM class descriptor inside d2: Lcom/foo/Bar;
RE_LCLASS = re.compile(r'L([A-Za-z][\w/$]+);')
# jadx emits this comment above classes it renamed during deobfuscation.
RE_RENAMED = re.compile(r'/\*\s*renamed from:\s*([\w.$]+)\s*\*/')
# Third-party / framework trees whose names are already real, so a "recovery" there is noise.
SKIP_PREFIXES = (
"kotlin.", "kotlinx.", "androidx.", "android.", "java.", "javax.",
"com.google.", "com.facebook.", "com.appsflyer.", "com.datadog.",
"io.ktor.", "io.sentry.", "io.realm.", "okhttp3.", "okio.",
"com.squareup.", "com.bumptech.", "com.airbnb.",
"com.amplitude.", "com.segment.", "com.mixpanel.", "com.onesignal.",
"com.stripe.", "retrofit2.", "dagger.", "javax.inject.", "org.jetbrains.",
)
MAP_TSV = "mapping.tsv"
MAP_JSON = "mapping.json"
BY_PACKAGE = "by_package"
def recover_real_name(text):
"""Return (real_fqn, source) for one decompiled file's text, or (None, None).
Order is deliberate: @DebugMetadata is the most precise (a true declaring-class FQN),
d2 descriptors are a strong hint, jadx's rename comment is the weakest.
"""
m = RE_DEBUG.search(text)
if m:
# Everything after the first `$` is inner/lambda scope of the original source tree.
return m.group(1).split("$", 1)[0], "debug_metadata"
m = RE_DTWO.search(text)
if m:
for lm in RE_LCLASS.finditer(m.group(1)):
cand = lm.group(1).replace("/", ".").split("$", 1)[0]
# A real package has at least one dot and is not a stdlib/framework ref.
if "." in cand and not cand.startswith(SKIP_PREFIXES):
return cand, "metadata_d2"
m = RE_RENAMED.search(text)
if m:
return m.group(1).split("$", 1)[0], "renamed_comment"
return None, None
def mine_tree(sources_dir):
"""Walk every *.java under sources_dir; return {obf_fqn: {real, file, source}}."""
mapping = {}
counts = defaultdict(int)
for dirpath, _dirs, files in os.walk(sources_dir):
for fname in files:
if not fname.endswith(".java"):
continue
path = os.path.join(dirpath, fname)
rel = os.path.relpath(path, sources_dir)
obf = rel[:-len(".java")].replace(os.sep, ".")
if obf.startswith(SKIP_PREFIXES):
continue
try:
with open(path, "r", errors="replace") as fh:
text = fh.read()
except OSError:
continue
real, source = recover_real_name(text)
if real:
mapping[obf] = {"real": real, "file": path, "source": source}
counts[source] += 1
return mapping, dict(counts)
def default_map_dir(sources_dir):
"""The mapping lands next to the sources tree, matching an export-then-query flow."""
return os.path.join(os.path.dirname(os.path.abspath(sources_dir)), "kotlin-name-map")
def write_map(mapping, counts, out_dir):
os.makedirs(os.path.join(out_dir, BY_PACKAGE), exist_ok=True)
with open(os.path.join(out_dir, MAP_TSV), "w") as fh:
fh.write("obf_fqn\treal_fqn\tsource\tfile\n")
for obf in sorted(mapping):
entry = mapping[obf]
fh.write("%s\t%s\t%s\t%s\n" % (obf, entry["real"], entry["source"], entry["file"]))
with open(os.path.join(out_dir, MAP_JSON), "w") as fh:
json.dump({"counts": counts, "mapping": mapping}, fh, indent=1, sort_keys=True)
fh.write("\n")
by_pkg = defaultdict(list)
for obf, entry in mapping.items():
by_pkg[entry["real"].rsplit(".", 1)[0]].append((entry["real"], obf, entry["file"]))
for pkg, rows in by_pkg.items():
safe = pkg.replace(".", "_")
with open(os.path.join(out_dir, BY_PACKAGE, safe + ".tsv"), "w") as fh:
fh.write("real_fqn\tobf_fqn\tfile\n")
for row in sorted(rows):
fh.write("%s\t%s\t%s\n" % row)
return len(by_pkg)
def load_map(map_dir):
with open(os.path.join(map_dir, MAP_JSON), "r") as fh:
return json.load(fh)["mapping"]
def resolve_query(args, mapping):
"""Run one query mode against a loaded mapping; returns the lines to print."""
lines = []
if args.obf:
if args.obf in mapping:
lines.append("%s -> %s (%s)" % (args.obf, mapping[args.obf]["real"],
mapping[args.obf]["source"]))
else:
# Sibling classes under the same short prefix are useful context: they are
# frequently the same original class's split lambdas/inner classes.
prefix = args.obf.rsplit(".", 1)[0] + "."
siblings = sorted(k for k in mapping if k.startswith(prefix))
for sib in siblings:
lines.append("%s -> %s (%s)" % (sib, mapping[sib]["real"],
mapping[sib]["source"]))
if not siblings:
lines.append("not in mapping: %s" % args.obf)
elif args.package:
for obf in sorted(mapping):
real = mapping[obf]["real"]
if real == args.package or real.startswith(args.package + "."):
lines.append("%s\t%s" % (real, obf))
return lines
def grep_with_map(pattern, sources_dir, mapping):
"""Grep sources, annotating each hit with the real owning class when known."""
rx = re.compile(pattern)
hits = 0
for dirpath, _dirs, files in os.walk(sources_dir):
for fname in files:
if not fname.endswith(".java"):
continue
path = os.path.join(dirpath, fname)
rel = os.path.relpath(path, sources_dir)
obf = rel[:-len(".java")].replace(os.sep, ".")
try:
with open(path, "r", errors="replace") as fh:
for lineno, line in enumerate(fh, 1):
if rx.search(line):
real = mapping.get(obf, {}).get("real", "?")
print("%s:%d: %s // %s" % (rel, lineno, line.rstrip(), real))
hits += 1
except OSError:
continue
if not hits:
print("(no hits for %r under %s)" % (pattern, sources_dir))
FIXTURE_DEBUG = '''package nq;
@DebugMetadata(
c = "com.example.feature.account.AccountRepositoryImpl$fetch$1",
f = "AccountRepositoryImpl.kt",
l = {42, 51},
m = "invokeSuspend"
)
public final class a extends kotlin.coroutines.jvm.internal.SuspendLambda {
}
'''
FIXTURE_METADATA = '''package p0.q;
@Metadata(
mvp = 1,
d1 = {"..." },
d2 = {"", "", "Lcom/example/feature/login/LoginViewModel;", "", "", ""}
)
public final class b {
}
'''
FIXTURE_RENAMED = '''package z;
/* renamed from: com.example.core.network.ApiClient */
public class C0123x {
}
'''
FIXTURE_PLAIN = '''package z;
public class Untouched {
}
'''
def selftest():
"""Parse the three recovery sources plus a negative fixture; assert each outcome.
The fixtures pin the *shapes* (annotation layout, `$`-suffix scope, d2 descriptor list,
rename comment), so a regex regression fails here rather than on a real export.
"""
real, source = recover_real_name(FIXTURE_DEBUG)
assert real == "com.example.feature.account.AccountRepositoryImpl", real
assert source == "debug_metadata"
real, source = recover_real_name(FIXTURE_METADATA)
assert real == "com.example.feature.login.LoginViewModel", real
assert source == "metadata_d2"
real, source = recover_real_name(FIXTURE_RENAMED)
assert real == "com.example.core.network.ApiClient", real
assert source == "renamed_comment"
real, source = recover_real_name(FIXTURE_PLAIN)
assert real is None and source is None
with tempfile.TemporaryDirectory() as tmp:
src = os.path.join(tmp, "sources")
for rel, body in (("nq/a.java", FIXTURE_DEBUG),
("p0/q/b.java", FIXTURE_METADATA),
("z/C0123x.java", FIXTURE_RENAMED),
("z/Untouched.java", FIXTURE_PLAIN),
("kotlin/x/Keep.java", FIXTURE_DEBUG)):
path = os.path.join(src, rel)
os.makedirs(os.path.dirname(path), exist_ok=True)
with open(path, "w") as fh:
fh.write(body)
mapping, counts = mine_tree(src)
# kotlin.* is skipped as already-real: exactly three mappings must come back.
assert len(mapping) == 3, mapping
assert counts == {"debug_metadata": 1, "metadata_d2": 1, "renamed_comment": 1}, counts
out = os.path.join(tmp, "map")
packages = write_map(mapping, counts, out)
assert packages == 3, packages
assert load_map(out)["nq.a"]["real"] == "com.example.feature.account.AccountRepositoryImpl"
print("selftest: 3 recovery sources + skip-list + round trip through the map files: ok")
return 0
def main(argv=None):
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
ap.add_argument("sources", nargs="?", help="decompiled sources root (build mode)")
ap.add_argument("--out", help="map output dir (default: next to the sources dir)")
ap.add_argument("--map", dest="map_dir", help="map dir for query modes (-o/-p)")
ap.add_argument("-o", "--obf", help="resolve one obfuscated FQN (siblings listed as context)")
ap.add_argument("-p", "--package", help="list real classes under one original package")
ap.add_argument("--grep", help="regex; grep sources with the real class annotated per hit")
ap.add_argument("--selftest", action="store_true", help="run the embedded fixture tests")
args = ap.parse_args(argv)
if args.selftest:
return selftest()
if args.obf or args.package:
if not args.map_dir:
ap.error("-o/-p need --map <dir> (build it first with: script <sources-dir>)")
for line in resolve_query(args, load_map(args.map_dir)):
print(line)
return 0
if args.grep:
if not (args.map_dir and args.sources):
ap.error("--grep needs --map <dir> and a sources dir")
grep_with_map(args.grep, args.sources, load_map(args.map_dir))
return 0
if not args.sources:
ap.print_usage()
print("build: script <sources-dir> [--out DIR] query: --map DIR -o nq.e / -p com.pkg / --grep")
return 2
if not os.path.isdir(args.sources):
print("not a directory: %s" % args.sources, file=sys.stderr)
return 1
out_dir = args.out or default_map_dir(args.sources)
mapping, counts = mine_tree(args.sources)
packages = write_map(mapping, counts, out_dir)
total = 0
for dirpath, _dirs, files in os.walk(args.sources):
total += sum(1 for f in files if f.endswith(".java"))
print("mapped %d of %d .java files (%d original packages) -> %s" %
(len(mapping), total, packages, out_dir))
print("by source: %s" % json.dumps(counts, sort_keys=True))
print("next: --map %s -o <obf.Fqn> | -p <com.example.pkg> | --grep <pattern> <sources-dir>" % out_dir)
return 0
if __name__ == "__main__":
sys.exit(main())
@@ -55,7 +55,9 @@ import time
try:
import frida
except ImportError: # keep --help working on a machine without frida
frida = None
# Same measured asymmetry as check_repo.py's yaml: only ill-typed where frida is
# installed; unused-ignored on a bare machine (warn_unused_ignores is off).
frida = None # type: ignore[assignment]
DEFAULT_ADB = "adb"
+67
View File
@@ -0,0 +1,67 @@
"""Pinned-shape tests for the JS obfuscation family detector.
The detector (`skills/apk-reverse/scripts/js_obf_detect.py`) scores families by weighted
probes (hint token = 10, content regex = 3). These tests pin the routing decisions that
matter: each fixture selects its own family as the top pick, a hint outranks a single
content hit, plain code routes to the generic ladder, and the huge-literal threshold is
a parameter the caller can lower for small samples.
"""
import os
import sys
sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', '..', 'skills',
'apk-reverse', 'scripts'))
from js_obf_detect import ( # noqa: E402
FIXTURES, _jsvmp_fixture, pipeline_for, residue_metrics, score_families,
)
def test_each_fixture_wins_its_own_family():
for fam_id, body in FIXTURES.items():
detections = score_families(body, '', 'sample.js', 1500)
assert detections, 'no detection for %s' % fam_id
assert detections[0]['id'] == fam_id, (fam_id, detections)
def test_jsvmp_entry_needs_the_literal_threshold():
# The huge-literal probes only fire at the configured length: with the default 1500
# threshold a 210-char literal scores nothing for the literal itself, and the fixture
# is detected (if at all) through the dependency-array probe alone. Lowering
# --min-literal is the documented way to classify small samples.
body = _jsvmp_fixture(200)
default_detections = score_families(body, '', 's.js', 1500)
reasons = [r for d in default_detections if d['id'] == 'jsvmp_entry' for r in d['reasons']]
assert all('opcode literal' not in r for r in reasons), reasons
detections = score_families(body, '', 's.js', 200)
assert detections and detections[0]['id'] == 'jsvmp_entry', detections
assert any('opcode literal' in r for r in detections[0]['reasons'])
def test_hint_beats_a_single_content_hit():
# A human hint is worth 10; one content regex is worth 3. The routing consequence:
# a named vendor hint wins over incidental structural overlap.
detections = score_families('plain text', 'some obfuscator lineage', 'x.js', 1500)
assert detections[0]['id'] == 'ob_string_array'
assert detections[0]['score'] == 10
def test_plain_code_is_generic_ladder():
assert score_families('function add(a, b) { return a + b; }\n', '', 'x.js', 1500) == []
assert pipeline_for('generic')[0] == 'normalize' and pipeline_for('generic')[-1] == 'normalize'
def test_pipelines_differ_per_family():
# The whole point of detection: the string-array lineage decodes FIRST, the
# opcode-if-chain family goes straight to if-chain conversion.
assert pipeline_for('ob_string_array')[2] == 'string_array_pass'
assert 'if_chain_to_switch' in pipeline_for('opcode_if_chain')
assert pipeline_for('ob_string_array') != pipeline_for('opcode_if_chain')
def test_residue_metrics_are_textual_proxies():
r = residue_metrics(FIXTURES['ob_string_array'])
# The 3+ hex-char filter counts _0x4ab1 but not short names (_0x1, _0x9f).
assert r['mangled_identifiers'] == 1
assert r['eval_calls'] == 0
assert r['lines'] >= 3
+73
View File
@@ -0,0 +1,73 @@
"""Pinned-shape tests for the Kotlin name recovery parser.
The shipped script (`skills/apk-reverse/scripts/kotlin_name_recovery.py`) is measured by
its embedded fixtures; these tests pin the same shapes independently so a regex regression
fails in CI rather than on a real jadx export. Every fixture here is a *shape*, written by
this repository, not a fragment of any real target.
"""
import os
import sys
sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', '..', 'skills',
'apk-reverse', 'scripts'))
from kotlin_name_recovery import ( # noqa: E402
FIXTURE_DEBUG, FIXTURE_METADATA, FIXTURE_PLAIN, FIXTURE_RENAMED, mine_tree,
recover_real_name, write_map,
)
def test_debug_metadata_strips_dollar_scope():
# `$fetch$1` is inner/lambda scope; the declaring class is everything before the first $.
real, source = recover_real_name(FIXTURE_DEBUG)
assert real == 'com.example.feature.account.AccountRepositoryImpl'
assert source == 'debug_metadata'
def test_metadata_d2_takes_first_non_stdlib_descriptor():
real, source = recover_real_name(FIXTURE_METADATA)
assert real == 'com.example.feature.login.LoginViewModel'
assert source == 'metadata_d2'
def test_renamed_comment_is_the_weakest_fallback():
real, source = recover_real_name(FIXTURE_RENAMED)
assert real == 'com.example.core.network.ApiClient'
assert source == 'renamed_comment'
def test_plain_java_recovers_nothing():
# No annotation and no rename comment: the honest result is (None, None), not a guess.
real, source = recover_real_name(FIXTURE_PLAIN)
assert real is None and source is None
def test_debug_metadata_outranks_everything():
# A file carrying both sources must resolve through @DebugMetadata: it is a true
# declaring-class FQN, while d2 descriptors are a hint.
both = FIXTURE_DEBUG + '\n@Metadata(d2 = {"Lcom/example/other/WrongHint;"})\n'
real, source = recover_real_name(both)
assert real == 'com.example.feature.account.AccountRepositoryImpl'
assert source == 'debug_metadata'
def test_mine_tree_skips_third_party_and_maps_the_rest(tmp_path):
src = tmp_path / 'sources'
for rel, body in (('nq/a.java', FIXTURE_DEBUG),
('z/C0123x.java', FIXTURE_RENAMED),
('kotlin/x/Keep.java', FIXTURE_DEBUG)):
path = src / rel
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(body, encoding='utf-8')
mapping, counts = mine_tree(str(src))
# kotlin.* is skipped as already-real even though its fixture carries @DebugMetadata.
assert set(mapping) == {'nq.a', 'z.C0123x'}
assert counts == {'debug_metadata': 1, 'renamed_comment': 1}
out = tmp_path / 'map'
packages = write_map(mapping, counts, str(out))
assert packages == 2
tsv = (out / 'mapping.tsv').read_text(encoding='utf-8')
assert 'nq.a\tcom.example.feature.account.AccountRepositoryImpl\tdebug_metadata' in tsv
# by_package files are keyed by the ORIGINAL package, which is the lookup a reader wants.
assert (out / 'by_package' / 'com_example_core_network.tsv').exists()