From 6cffc3d2ac9cfe6b7cc1e3ca9047b4c320329f38 Mon Sep 17 00:00:00 2001 From: inliver233 Date: Mon, 21 Sep 2026 22:51:09 +0800 Subject: [PATCH] Absorption pass: take the mechanics from four external skill repositories, keep our discipline Four projects were cloned for comparison (reverse-skill, awesome-game-security, android-reverse-engineering-claude-skill, MobileRE-Skill) and read for what they do that this repository does not. None is vendored and none becomes a runtime dependency: what crosses over is a phase order, a checklist or a command pattern, with the source URL and the retrieval date recorded in the file that absorbed it. The rule applied to every candidate: does this make a judgement easier or a verification harder? The repository is a decision-and-verification layer, and the four things kept are all of that kind. new scripts/scan_leaks.py target identity scanned before publishing, with an explicit do-not-anonymize list and four RESULT=/exit states new references/precedents/ the positive half of the record: route including dead ends, a grade per assertion, measured pits, and a write-back checklist new scripts/svc_scan.py does a libc-level hook structurally see this call at all new scripts/anti_detect_probe.js observer-only: name the check, and report the environment it reads new references/desensitization-and-leak-scans.md new references/dart-aot.md sections the 32-bit snapshot's real string-record format check_repo.py now delegates the identity scan to scripts/scan_leaks.py and runs it over git ls-files, so the gate reads content instead of layout for the first time. An explicit file list cannot be narrowed by the default work-area exemptions -- that defect was caught by the gate itself. Two claims that this repository had already written down turned out to be wrong, and both are recorded as refuted rather than quietly rewritten: dart-aot.md the armv7 string table was documented as [len*2 u32le][UTF-16LE]. It is not. The same extractor keeps 4,980 chained entries on arm64 and zero on armv7, and the byte layer shows the 32-bit record is [header u32][byte-count u32le][UTF-8 payload]: two adjacent literals carry 0x46 (70) and 0x48 (72), exactly their byte lengths. The zero was a format mismatch, not an empty table. The engine-version banner on the same target is 66 bytes of "0.0.1" with a blank build id, so the documented regex returns a version-shaped lie. leak scan the pass that added the scanner found 26 strong hits inside its own evidence file on the day it was written -- the fixture's literal match values, quoted to prove the rules fire. The corpus may contain values; the published surface may not. Measured, not asserted. The check that fires was named on a public MASTG target: three arms with one variable, the control alive at the end and the probe arm dead in about 300 ms, with strstr("frida") at t=303 ms and no tombstone. Reproducibility is bounded honestly -- one full sequence in three arms, on a target whose whole window is sub-second. svc_scan.py's two independent decoders agree on an identical 214-site set, and ptrace-free dumping gained a negative boundary: a target whose dex is deflated inside its APK has no whole-image dex in memory to export, for reasons that say nothing about protection. Also in this pass: the detector-locating pipeline (detection-and-anti-analysis.md Step 3) with its branch conditions; the layered Java-to-svc descent and the dump's honesty boundary in advanced-unpacking.md; a sourced capability-gap list in toolchain.md where every row is graded URL-verified or measured; the RESULT=-token convention; and SKILL.md opening with an immediate-action block, because the symptom index and the gates were sitting in the part of a long file nobody reads. Gates: check_repo.py 0 problem, check_refs.py 0 dangling / 0 warning, check_budget.py 0 failure / 0 warning / 1 pre-existing note. The scan reports RESULT=leaks_found_strong_only, exit 0: four endpoint/weak hits, all of them addresses from the RFC 5737 documentation range in usage examples. --- README.md | 37 +- check_repo.py | 64 ++ .../EXTENSION-dart-aot-formats.md | 183 +++++ .../EXTENSION-desensitization.md | 454 +++++++++++ .../EXTENSION-detection-pipeline.md | 311 +++++++ docs/tool-verification/README.md | 15 + skills/apk-reverse/SKILL.md | 34 +- .../references/advanced-unpacking.md | 131 ++- .../references/coverage-and-limits.md | 43 +- skills/apk-reverse/references/dart-aot.md | 76 ++ .../desensitization-and-leak-scans.md | 222 +++++ .../references/detection-and-anti-analysis.md | 226 +++++- .../references/long-task-discipline.md | 26 + .../references/precedents/README.md | 97 +++ .../precedents/flutter-plus-shell-case-1.md | 74 ++ .../frida-spawn-hangs-zero-events-case-2.md | 76 ++ .../l1-equal-length-patch-case-4.md | 80 ++ .../logd-broken-module-never-ran-case-3.md | 70 ++ ...l-grep-finds-what-nobody-grepped-case-5.md | 81 ++ skills/apk-reverse/references/toolchain.md | 36 + .../apk-reverse/scripts/anti_detect_probe.js | 353 ++++++++ skills/apk-reverse/scripts/scan_leaks.py | 757 ++++++++++++++++++ skills/apk-reverse/scripts/svc_scan.py | 467 +++++++++++ tests/benchmark.md | 2 + 24 files changed, 3904 insertions(+), 11 deletions(-) create mode 100644 docs/tool-verification/EXTENSION-dart-aot-formats.md create mode 100644 docs/tool-verification/EXTENSION-desensitization.md create mode 100644 docs/tool-verification/EXTENSION-detection-pipeline.md create mode 100644 skills/apk-reverse/references/desensitization-and-leak-scans.md create mode 100644 skills/apk-reverse/references/precedents/README.md create mode 100644 skills/apk-reverse/references/precedents/flutter-plus-shell-case-1.md create mode 100644 skills/apk-reverse/references/precedents/frida-spawn-hangs-zero-events-case-2.md create mode 100644 skills/apk-reverse/references/precedents/l1-equal-length-patch-case-4.md create mode 100644 skills/apk-reverse/references/precedents/logd-broken-module-never-ran-case-3.md create mode 100644 skills/apk-reverse/references/precedents/manual-grep-finds-what-nobody-grepped-case-5.md create mode 100644 skills/apk-reverse/scripts/anti_detect_probe.js create mode 100644 skills/apk-reverse/scripts/scan_leaks.py create mode 100644 skills/apk-reverse/scripts/svc_scan.py diff --git a/README.md b/README.md index 1d6bcb0..e8d7d28 100644 --- a/README.md +++ b/README.md @@ -99,6 +99,13 @@ measurements. - Working a **real Dex VMP** with the known-plaintext differential — which links can be automated and which cannot, what a compiled fixture can and cannot reach, and how to *prove* a derived private-opcode table instead of asserting one. +- **Publishing what it learns without publishing the target** — a scanner that reports identity + shapes with their context, an explicit list of what must *not* be redacted (tools, libraries, + protocol fields, CVEs, hardening products, public crackmes) because redacting those destroys the + reusable part, and exit codes that gate a commit. +- Reading a **precedent** before repeating work this repository already converged on: the positive + half of the record, with the route including its dead ends, a grade on every assertion, and the + files the case says to write back to. ## Structure @@ -193,6 +200,12 @@ references/ loaded on demand, one topic each environment.md device/emulator setup, root, ADB, offline devices, log signals, emulator console control and recovery, preflight, look-at-the-screen verification.md the claim ladder; what "done" means + desensitization-and-leak-scans.md + publishing discipline: what must be desensitized and what must stay, + the do-not-anonymize list, the leak scanner and its exit states, and + the entry-point file as a prompt surface + precedents/ the positive case library: route including dead ends, a grade per + assertion, measured pit-falls, and the write-back checklist pitfalls.md the failure catalogue -- read before building advanced-unpacking.md the dump landed but the bodies are empty: extraction-shell diagnosis by trivial-body ratio, FART-style active invocation and why its classic hooks @@ -228,6 +241,12 @@ references/ loaded on demand, one topic each automated and which cannot (the upload is the bottleneck), the coverage a compiled fixture can reach, how to *prove* a derived private-opcode table, smali generation, and when the route is closed + coverage-and-limits.md the claim ladder applied to the skill itself: the evidence behind each + covered item, the dependencies this skill does not ship, and what was + never exercised + handoff-boundaries.md where this skill ends and another discipline begins: the JNI form + table, the packer-versus-loader split, and what "verified" means for + each of the four deliverable forms scripts/ parameterized, path-agnostic doctor.py run this first: capability report + per-script runnability, finds tools installed off-PATH or as runnable jars, and surfaces the @@ -330,6 +349,19 @@ scripts/ parameterized, path-agnostic kernelsu_syscall_mask.py generate a KernelSU/APatch syscall-masking scaffold: an installable userspace module skeleton plus KPM/LKM/eBPF kernel-side templates, each with its version gate and an explicit unverified label + scan_leaks.py scan a repository for target identity before publishing it: bundle ids + in manifest / `pm` / `ps` contexts, serial-shaped tokens, PATs, inline + appkey assignments, literal endpoints, host user paths. Exemptions for + everything that must stay (tools, libraries, CVEs, hardening products, + public crackmes, placeholders), findings carry their context, + `--show-exempt` prints why a hit was suppressed, exit 0/1/2 + svc_scan.py name the syscall behind an inline `svc` and the segment it sits in, + which decides whether a libc-level hook can observe the call at all; + `--context` shows neighbours because a byte scan also matches data + anti_detect_probe.js observer-only Frida probe (patches nothing): path/loader/thread/kill + hooks with caller module + offset, an environment self-report + (`TracerPid`, frida-named mappings), and live streaming so a sub-second + self-destructing target still yields evidence ``` ## Install @@ -454,7 +486,10 @@ Four tools live at the root and are not part of the installed skill: ``` check_repo.py every skill discovered, frontmatter valid, scripts runnable, - documented paths resolve, README paths explicit and existing + documented paths resolve, README paths explicit and existing, + and -- on the tracked surface only -- no target identity + (delegates the rules to skills/apk-reverse/scripts/scan_leaks.py so + there is one place to argue with the exemption list) check_refs.py every cross-reference that names a section of another document reaches a real heading in that document check_budget.py keep the always-loaded part from creeping: narrative lines in diff --git a/check_repo.py b/check_repo.py index 135a14a..fb67310 100644 --- a/check_repo.py +++ b/check_repo.py @@ -25,6 +25,7 @@ On the root README: import ast import os import re +import re import subprocess import sys @@ -235,6 +236,66 @@ def check_readme(): print(' BARE PATH %s' % m.group(0)) +def check_leaks(): + """No target identity on the committed surface. + + Path resolution, anchors and layout are all *structural* checks: they pass on a file that names + a live app and a real phone. This one reads content, and it is deliberately not reimplemented + here -- `scripts/scan_leaks.py` owns the rules and their exemption list, so there is one place + to argue with. The tracked-file list is passed in so a git-ignored work area (which may + legitimately hold real identifiers) can never produce a finding. + + A scanner error is reported as a note rather than a failure: an unreadable rule table is a + broken tool, not a leak, and conflating the two teaches people to ignore this line. + """ + scanner = os.path.join(SKILLS_DIR, 'apk-reverse', 'scripts', 'scan_leaks.py') + if not os.path.isfile(scanner): + print(' SKIP no scan_leaks.py (the leak gate is not installed)') + return + import subprocess + import tempfile + proc = subprocess.run(['git', 'ls-files'], cwd=ROOT, capture_output=True, text=True) + if proc.returncode != 0 or not proc.stdout.strip(): + print(' SKIP git ls-files unavailable (not a checkout?)') + return + with tempfile.NamedTemporaryFile('w', suffix='.txt', delete=False, encoding='utf-8') as fh: + fh.write(proc.stdout) + listing = fh.name + try: + run = subprocess.run([sys.executable, scanner, '--root', ROOT, + '--files-from', listing, '--max', '10'], + capture_output=True, text=True) + finally: + os.unlink(listing) + token = (run.stdout or '').strip().splitlines() + token = token[-1] if token else '' + if run.returncode == 2 or 'RESULT=error' in token: + print(' NOTE scan_leaks.py could not complete: %s' % (run.stderr or '').strip()[:160]) + return + if run.returncode == 0: + print(' no strong target identity on the tracked surface (%s)' % (token or 'RESULT=clean')) + return + # Non-zero: report the findings, which carry their own context. The scanner prints each hit as + # ":: [/] ", followed by its indented match/context + # lines. The list below keeps only the strong hits -- a weak hit is the scanner's business, not + # this gate's -- but a weak hit sitting between two strong ones may contribute its context pair, + # because attributing an indented line to the hit above it is heuristic. The exit code is the + # verdict; this block is for the reader. + print(' LEAK scan_leaks.py reported strong findings on the tracked surface:') + keep_detail = False + for line in (run.stdout or '').splitlines(): + stripped = line.strip() + if re.search(r'\[(package|device|token|appkey|path)/(strong|certain)\]', stripped): + print(' %s' % stripped) + keep_detail = True + elif stripped.startswith('[') and stripped.count(']') >= 2: + keep_detail = False + elif keep_detail and (stripped.startswith('match:') or stripped.startswith('context:')): + print(' %s' % stripped) + fail.append('leak scan: strong target identity on the tracked surface ' + '(run: python skills/apk-reverse/scripts/scan_leaks.py)') + + def main(): skills = discover_skills() print('== skills discovered ==') @@ -261,6 +322,9 @@ def main(): print('\n== README paths are explicit and exist ==') check_readme() + print('\n== tracked surface carries no target identity ==') + check_leaks() + print('\n== result: %d problem(s) ==' % len(fail)) for f in fail: print(' - %s' % f) diff --git a/docs/tool-verification/EXTENSION-dart-aot-formats.md b/docs/tool-verification/EXTENSION-dart-aot-formats.md new file mode 100644 index 0000000..ab8253e --- /dev/null +++ b/docs/tool-verification/EXTENSION-dart-aot-formats.md @@ -0,0 +1,183 @@ +# EXTENSION — Dart AOT string-table formats, measured on a real dual-ABI app + +Evidence file for the `dart-aot.md` changes in this pass. Every claim below carries a grade: + +- **observed** — a command was run here and its output is quoted in this file. +- **inferred** — follows from an observed fact or from documented mechanism; the step was not executed. +- **unverified** — assumed or reported elsewhere, not reproduced here. + +Target identity is replaced with ``; the artifact is referred to as "the sample". The technique +(tool, library, function, hardening product) is kept, per +`skills/apk-reverse/references/desensitization-and-leak-scans.md`. + +## 1. What was measured + +| Claim under test | Grade | Result | +|---|---|---| +| The arm64 Dart AOT string table is `[0x80\|(len<<1)\|two_byte][payload]` and the entries chain | **observed** | confirmed on 4,980 entries | +| The armv7 (32-bit) snapshot uses the *same* format | **refuted** | the same extractor keeps 0 entries; the tag byte is not present before the literal | +| An armv7 string record is `[length u32le][UTF-16LE chars]`, so the length field reads `len*2` | **refuted** | the u32 before the payload reads the **byte** count, and the payload is UTF-8 | +| The engine version banner always yields `x.y.z (stable) (…)` | **refuted** | a real `libflutter.so` carries `0.0.1` with a blank build id and no `(stable)` | +| An equal-length replacement of a data key is legal on both ABIs without recomputing the tag | **observed** (record layout), **unverified** (end-to-end device run) | the record is self-contained on both ABIs | + +Two of the five were assumptions that would have been written into the reference as fact. Both were +killed by a single command each, which is the whole argument for measuring before documenting. + +## 2. Sample and preparation + +The sample is the locally supplied "complex encryption and hardening" APK (360 shell + Flutter). +The two AOT snapshots were extracted from the zip without repacking: + +``` +$ python -c "import zipfile; z=zipfile.ZipFile(r'存在问题和例子\.apk'); \ + [print(n, len(z.read(n))) for n in ('lib/arm64-v8a/libapp.so','lib/armeabi-v7a/libapp.so')]" +lib/arm64-v8a/libapp.so 16352152 +lib/armeabi-v7a/libapp.so 17629776 +``` + +Work copies live under the ignored workspace `tools/_work/dart/`; nothing from this evidence pass is +committed except the reference text. + +## 3. The arm64 packed table — confirmed + +``` +$ python skills/apk-reverse/scripts/dart_pool_strings.py tools/_work/dart/app_arm64.so s64.tsv --min 6 +candidates : 17352 +kept (run >= 3) : 4980 (one-byte 4980 / utf16le 0) +below threshold : 12372 [--keep-isolated to keep them] +written s64.tsv +``` + +The tag arithmetic was checked directly against real literals in the image rather than only through +the script. For each literal, the byte immediately before it must equal `0x80|(len<<1)`: + +``` +$ python -c "... for m in (b'AdmobBanner', b'showSplashAd', b'AdPgl_Splash'): ..." +### app_arm64.so + AdmobBanner @0x4932e len=11 expected_tag=0x96 + tag_byte_at_-1=0x96 matches=True + showSplashAd @0x39deb len=12 expected_tag=0x98 + tag_byte_at_-1=0x98 matches=True + AdPgl_Splash @0x493ee len=12 expected_tag=0x98 + tag_byte_at_-1=0x98 matches=True +``` + +**observed**: three independent literals of two different lengths, three matching tag bytes. This is +also the assertion a patch script should perform (`dart-aot.md` §10), and it is what rejects a hit +that landed mid-string. + +## 4. The armv7 assumption — refuted, and what is actually there + +The starting hypothesis (carried over from an external case) was that the 32-bit snapshot stores a +string as `[tag 0x81][len*2 u32le][UTF-16LE chars]`. Two scanners written against that hypothesis +found nothing, which is itself the evidence: + +``` +$ python tools/_work/dart/probe3.py app_armv7.so raw # [8-byte hdr ending 0x81][u32le len][chars] +hits=0 chained_runs=0 entries_in_runs=0 + +$ python tools/_work/dart/probe3.py app_armv7.so packed # the arm64 tag scheme +hits=0 chained_runs=0 entries_in_runs=0 +``` + +The same extractor that keeps 4,980 entries on arm64 keeps **zero** on armv7: + +``` +$ python skills/apk-reverse/scripts/dart_pool_strings.py tools/_work/dart/app_armv7.so s7.tsv --min 6 +candidates : 1355 +kept (run >= 3) : 0 (one-byte 0 / utf16le 0) +``` + +A zero here is a **format mismatch, not an empty string table**, and that distinction is now stated in +`dart-aot.md` §7: reading it as "the strings were stripped" is exactly the wrong conclusion the same +section warns about for UTF-16. + +What the record actually looks like, from bytes around a known literal: + +``` +$ python -c "... b.find(b'SystemChrome') ..." +0x1ffb3c b'\x00\x00\x00\x008\x06U\x00F\x00\x00\x00\xce-\x8a@SystemChrome.setEnab...' +0x204ce4 b'\x00\x00\x00\x008\x06U\x00H\x00\x00\x00\xa45jVSystemChrome.setSyst...' +0x2185c4 b' ere8\x07U\x00J\x00\x00\x00\x8c]\xb4GSystemChrome.setPref...' +``` + +Read as `[header u32][length u32le][payload]`: `0x00550638` then `0x00000046` = **70**, which is the +byte length of `SystemChrome.setEnabledSystemUIMode` (36 chars) immediately followed by +`SystemChrome.setSystemUIOverlayStyle` (34 chars) — **70 bytes, UTF-8, not UTF-16**. The second +literal starts at the exact byte where the first ends, and its own length field reads `0x00000048` = +72, matching its own continuation. + +**observed**: `length == payload byte count`, payload is UTF-8, and in-place equal-length replacement +therefore keeps the record valid without recomputing anything. **refuted**: the `len*2` / UTF-16 form. + +The header word is a tags value with full 4-byte alignment, not a constant to hard-code: + +``` +$ python -c "... alignment histogram over the image ..." +alignment mod 4: {0: 13397} +``` + +``` +$ python tools/_work/dart/perf.py app_armv7.so +scanned 17629776 B in 0.94s using 4-byte stride + int.from_bytes +aligned-header string records: 582 +stride-1 micro scan over 17629776 B took 2.80s (found 2762) +``` + +The literal search plus a backwards length assertion is the tool-independent method, and it costs +under a second over a 17.6 MB image — so validating every candidate is affordable and the armv7 +limitation does not justify a hand-built table parser. **unverified**: whether a *different* armv7 +snapshot uses a different header word; the reference tells the reader to read it from the file. + +## 5. The engine version banner — a version-shaped lie + +``` +$ python -c "... rfind NUL before the ' on \"android_arm64\"' suffix ..." +end-of-version offset 0x110821 +version string start 0x1107f1 => total 66 +b'0.0.1 on "android_arm64"' +``` + +The banner exists, is 66 bytes long, and its numeric field is `0.0.1` with a blank build-id field — +no `(stable)`, no `(Mon …)`. A regex that takes `\d+\.\d+\.\d+` out of this returns a value that would +send a reader to build a Dart VM for `0.0.1`. + +**observed** (the byte string above), **inferred** (that this is a build-time population failure rather +than a deliberately rewritten banner). The actionable consequence is in `dart-aot.md` §1: assert the +`(stable) (…)` tail before using the number, and fall back to the snapshot hash otherwise. **The +sample's Dart version was not pinned in this pass** — that is an unverified item, not a finding. + +## 6. Cross-validation + +Every measurement above is a **raw byte search with an independent arithmetic assertion** (find the +literal in the file; check the byte or word immediately before it against the length of what follows). +It does not depend on `dart_pool_strings.py`, and the script's numbers are reported alongside the raw +probe rather than as the source of truth. The two agree on arm64 (4,980 kept entries, tag byte equal to +`0x80|(len<<1)` at every literal tested) and on armv7 (the packed scheme finds nothing, the byte layer +shows why). + +**Not obtained**: a second, independently implemented Dart snapshot parser. `blutter` was not built in +this pass, so the reference's `pp.txt`-side claims were not re-derived here, and the armv7 conclusion +rests on the byte layout rather than on a tool's agreement. That is the boundary of this evidence. + +## 7. What this pass changed in the repository + +- `skills/apk-reverse/references/dart-aot.md` — §7 gained the armv7 record format and the + "the armv7 zero is a format mismatch" warning; §1 gained the banner-shape assertion; §10 gained the + data-key-versus-API-path rule with its measured failure mode, and the longest-first / assert-the-prefix + rule. No new file, no new index row: the finding belongs in the file that already owns this layer. +- The equal-length-replacement failure (a 404 whose body is not JSON, `jsonDecode` throwing, the app + frozen on the launch logo) is documented as the concrete instance of the top-level constraint + "do not make an API fail to suppress a UI element". + +## 8. Not done, and known to be outstanding + +1. **No end-to-end device run.** The string-table patch was not repacked, signed and installed in this + pass, so "an equal-length key replacement suppresses the feature while the app still starts" is + `unverified` end-to-end even though the record layout that makes it legal is `observed`. +2. **No `blutter` build**, so no `pp.txt` comparison and no confirmation of the arm64 pool space from a + second implementation (§6). +3. **The sample's Dart version is unpinned** (§5); a structural probe or a snapshot hash is the next + experiment, not a guessed number. +4. **The armv7 header word is not characterised.** It is read from the file and used as a validator; + which component assigns it was not determined. diff --git a/docs/tool-verification/EXTENSION-desensitization.md b/docs/tool-verification/EXTENSION-desensitization.md new file mode 100644 index 0000000..59ffbdd --- /dev/null +++ b/docs/tool-verification/EXTENSION-desensitization.md @@ -0,0 +1,454 @@ +# Extension verification — desensitization and leak scans + +Covers the pass that added `skills/apk-reverse/scripts/scan_leaks.py` and +`skills/apk-reverse/references/desensitization-and-leak-scans.md`, plus the precedent library +under `skills/apk-reverse/references/precedents/`. + +Strength labels follow `docs/tool-verification/README.md`: **observed** (the exact command and +its output are reproduced here), **inferred** (follows from observed facts; the step itself was +not executed), **unverified** (assumed or reported elsewhere and not independently confirmed). + +Reference environment: Windows 11 host, Python 3.14.0, repository at `E:\apk-reverse` on branch +`main`. **No device, no network and no target sample were involved** — this pass is entirely +machine-local, because the thing worth measuring is what a scanner reports against a tree whose +contents are already known. Every identifier in the fixture and in every transcript below is +synthetic; the repository's convention of `` / `` / `` is used wherever a +real one would have appeared. + +Date of the recorded runs: 2026-09-21. The numbers in §3b and §4a were measured on that date, on +the tree this pass leaves behind; those two sections' transcripts are the ones this pass produced. + +**Headline: the planted-leak corpus is reported in full (30 findings, 10 further shapes admitted +through the exemption path), and the repository scan converges to zero strong/certain findings — +the residue is four `endpoint/weak` hits, all four an RFC 5737 documentation address, which is a +class this scanner reports on purpose. The first version of this file was itself the +counter-example to its own subject: it quoted the planted values to prove the rules fire, and the +scanner found that as a leak (§3b). The corpus keeps its values, because a corpus that cannot be +found proves nothing; the published document keeps none. The gate is not wired into anything yet; +that is the largest gap, and it is a Lead-side action.** + +--- + +## 1. What was built, and what it was measured against + +| Artifact | Role | Measured? | +|---|---|---| +| `skills/apk-reverse/scripts/scan_leaks.py` | the scanner: 17 rules in 6 categories, exemption table, four `RESULT=` tokens, `--fail-on strong\|any` | **observed** — every run below | +| `tools/_work/leak-precedent/gen_fixtures.py` | generates the planted corpus and the exemption corpus | **observed** — 7 files, re-runnable | +| `tools/_work/leak-precedent/fixture/` | the corpus itself (git-ignored work area) | **observed** | +| `skills/apk-reverse/references/desensitization-and-leak-scans.md` | the reasoning, the do-not-anonymize table, the failure modes | not executable | +| `skills/apk-reverse/references/precedents/**` | the case library, distilled from this repository's existing evidence | see the per-case strength tables | + +The fixture exists so the claim "the scanner reports planted leaks" is reproducible rather than +a transcript pasted once: + +``` +$ python tools/_work/leak-precedent/gen_fixtures.py +wrote E:\apk-reverse\tools\_work\leak-precedent\fixture\planted\package.txt (411 bytes) +wrote E:\apk-reverse\tools\_work\leak-precedent\fixture\planted\device.txt (218 bytes) +wrote E:\apk-reverse\tools\_work\leak-precedent\fixture\planted\token.txt (261 bytes) +wrote E:\apk-reverse\tools\_work\leak-precedent\fixture\planted\appkey.txt (212 bytes) +wrote E:\apk-reverse\tools\_work\leak-precedent\fixture\planted\endpoint.txt (146 bytes) +wrote E:\apk-reverse\tools\_work\leak-precedent\fixture\planted\path.txt (175 bytes) +wrote E:\apk-reverse\tools\_work\leak-precedent\fixture\exempt\benign.txt (1396 bytes) +== 7 fixture file(s) == +``` + +## 2. `--help`, and the rule table — observed + +``` +$ python skills/apk-reverse/scripts/scan_leaks.py --help +usage: scan_leaks.py [-h] [--root ROOT] [--only ONLY] [--exclude EXCLUDE] + [--format {text,json}] [--show-exempt] [--list-rules] + [--max MAX] [--quiet] [--fail-on {strong,any}] + [--no-color] + +Scan a skills repository for target identity that should not be published (bundle ids, device serials, tokens, SDK keys, literal endpoints, user paths). Exit 0 clean / 1 leaks_found / 2 error; prints RESULT= last. `--fail-on strong` (default) keeps the gate green when the only hits are documentation-range addresses. + +options: + -h, --help show this help message and exit + --root ROOT repository root to scan (default: the repo this script + lives in) + --only ONLY comma-separated categories to restrict to, e.g. + pat,appkey + --exclude EXCLUDE extra directory name to skip (repeatable) + --format {text,json} + --show-exempt also print findings that were suppressed by an + exemption + --list-rules print the rule table and exit + --max MAX stop after N findings (0 = no limit) + --quiet print only the RESULT token + --fail-on {strong,any} + which findings make the exit code non-zero: `strong` + (default) fails only on strong/certain findings, `any` + also fails on `weak` ones such as an address from the + RFC 5737 documentation range + --no-color accepted for compatibility + +Categories: package, device, token, appkey, endpoint, path +Exemptions (tool names, library names, CVE ids, hardening products, public +crackme names, -style placeholders, loopback/emulator addresses) are built +in and audited with --show-exempt. +``` + +`--list-rules` prints the table the exemption functions are attached to, which is what makes the +scanner arguable rather than opaque. It exits 0 and prints `RESULT=clean` (no files were read), +and the top of its output is: + +``` +$ python skills/apk-reverse/scripts/scan_leaks.py --list-rules +== rules == + bundle_pkg_attr package strong a `package=` bundle id in a manifest or transcript + bundle_pkg_decl package strong a bundle id in a `` line + bundle_pm_path package strong a bundle id in a `pm`/`am`/`monkey` invocation + bundle_process_line package strong a bundle id in a `ps`/`pidof`/`top` capture + bundle_component package strong a bundle id in a `component=`/`-n` activity reference + device_serial device strong an `adb devices`-shaped device serial + device_serial_context device strong a serial next to a device/install/`su` context word + device_serial_tabular device strong a bare 16-char token in a device-listing table + token_github_pat token certain a GitHub fine-grained personal access token + token_github_classic token certain a GitHub classic token + token_env_assignment token strong an inline API-key/token/secret assignment + token_authorization_header token strong a literal Authorization header value + appkey_assignment appkey strong an SDK appkey/appsecret assignment with a literal value + plain_addr endpoint weak a non-loopback literal IP:port + plain_addr_ipliteral endpoint weak a non-loopback bare IPv4 literal + user_path_posix path strong an absolute POSIX user-home path + user_path_windows path strong an absolute Windows user-profile path +RESULT=clean (exit 0) +``` + +## 3. The planted corpus — observed, every rule fires + +``` +$ python skills/apk-reverse/scripts/scan_leaks.py --root tools/_work/leak-precedent/fixture --format json +{ "root": "...\\tools\\_work\\leak-precedent\\fixture", "files_scanned": 7, + "findings": [ ... 30 ... ], "exempted": [], "errors": [], "result": "leaks_found" } +RESULT=leaks_found (exit 1) +``` + +aggregated (the `findings` array of the same JSON run, grouped by category): + +``` +files 7 +findings 30 {'appkey': 4, 'token': 5, 'device': 8, 'endpoint': 5, 'package': 5, 'path': 3} +``` + +and the 30 hits by position, rule and value **shape** — every field of the report except the +literal itself. Rule name, category, strength, position and count are the evidence; the quoted +value is a copy of the leak, and copying it into a published document re-creates what the scan +exists to prevent (§3b is where that rule was paid for): + +``` +planted/package.txt:2:16 bundle_pkg_decl 4-segment all-lowercase reverse domain +planted/package.txt:2:26 bundle_pkg_attr the same value, second rule on the same line +planted/package.txt:3:1 bundle_pm_path 3-segment all-lowercase reverse domain, after `pm path` +planted/package.txt:5:1 bundle_process_line 4-segment all-lowercase reverse domain, after `pidof` +planted/package.txt:6:15 bundle_component 4-segment all-lowercase reverse domain, after `component=` + +planted/device.txt:4:1 device_serial 16×[A-Z0-9], tab-separated from the word `device` +planted/device.txt:4:1 device_serial_tabular the same value, second rule on the same line +planted/device.txt:5:3 device_serial_context 16×[A-Z0-9], after `$ adb -s ` +planted/device.txt:5:10 device_serial the same value, second rule on the same line +planted/device.txt:6:2 device_serial_context 16×[A-Z0-9], after `[serial ` +planted/device.txt:6:9 device_serial the same value, second rule on the same line +planted/device.txt:7:1 device_serial_context 16×[A-Z0-9], after `device serial: ` (a second value) +planted/device.txt:7:16 device_serial the same value, second rule on the same line + +planted/token.txt:2:14 token_github_pat fixed provider prefix + 62×[A-Za-z0-9_] +planted/token.txt:3:15 token_github_classic fixed 4-char provider prefix + 36×[A-Za-z0-9] +planted/token.txt:4:1 token_env_assignment 20-char `AKIA`-prefixed key id after a key assignment +planted/token.txt:5:9 token_authorization_header 32×[A-Za-z0-9] after an `Authorization:` keyword +planted/appkey.txt:5:1 token_env_assignment the same 40-char secret, second rule on the same line + +planted/appkey.txt:2:1 appkey_assignment 24×[0-9a-f] after an appkey assignment +planted/appkey.txt:3:1 appkey_assignment 24×[0-9a-f], quoted, after a camelCase app-secret assignment +planted/appkey.txt:4:1 appkey_assignment 7 dash-separated 4-character uppercase groups +planted/appkey.txt:5:1 appkey_assignment 40-char mixed-case secret after a `SECRET_KEY` assignment + +planted/endpoint.txt:2:7 plain_addr an RFC 5737 documentation address, with `:port` +planted/endpoint.txt:2:7 plain_addr_ipliteral the bare form of the same address +planted/endpoint.txt:3:16 plain_addr an RFC 1918 private-range address, with `:port` +planted/endpoint.txt:3:16 plain_addr_ipliteral the bare form of the same address +planted/endpoint.txt:4:12 plain_addr_ipliteral an RFC 5737 documentation address, bare + +planted/path.txt:2:10 user_path_posix two-segment personal name after `/home/` +planted/path.txt:3:6 user_path_posix one-word personal name after `/Users/` +planted/path.txt:4:6 user_path_windows capitalized personal name after `C:\Users\` +``` + +Two rules firing on one line (`SECRET_KEY=` matches both an appkey and a token pattern) is kept +deliberately and is visible in the count; identical duplicates (same rule, same value, same +line) are collapsed. Eight of the 30 findings share their line with a second rule — one package +line, all four serial lines, the `SECRET_KEY` line and the two endpoint lines — which is why 30 +findings cover 22 distinct values (counted from the `findings` array of the JSON run above, not +by hand). + +### 3a. The exemption corpus — observed, and it is the half that is easy to get wrong + +The same run with `--show-exempt` shows what was *not* reported from +`fixture/exempt/benign.txt`, and the reason: + +``` +$ python skills/apk-reverse/scripts/scan_leaks.py --root tools/_work/leak-precedent/fixture --show-exempt +== suppressed by exemption: 10 == + exempt/benign.txt:13 [user_path_windows] -> placeholder user + exempt/benign.txt:17 [plain_addr] 127.0.0.1:27042 -> loopback/unspecified/emulator address + exempt/benign.txt:17 [plain_addr_ipliteral] 127.0.0.1 -> loopback/unspecified/emulator address + exempt/benign.txt:17 [plain_addr] 0.0.0.0:8080 -> loopback/unspecified/emulator address + exempt/benign.txt:17 [plain_addr_ipliteral] 0.0.0.0 -> loopback/unspecified/emulator address + exempt/benign.txt:17 [plain_addr] 10.0.2.2:8080 -> loopback/unspecified/emulator address + exempt/benign.txt:17 [plain_addr_ipliteral] 10.0.2.2 -> loopback/unspecified/emulator address + exempt/benign.txt:18 [plain_addr] 127.0.0.1:8787 -> loopback/unspecified/emulator address + exempt/benign.txt:18 [plain_addr_ipliteral] 127.0.0.1 -> loopback/unspecified/emulator address + exempt/benign.txt:18 [plain_addr_ipliteral] 169.254.169.254 -> loopback/unspecified/emulator address +``` + +The earlier pass recorded 6 suppressed hits, all `plain_addr`; the current run reports the paired +`plain_addr_ipliteral` for each exempt address as well, which is what the two same-line rules +look like when nothing is being collapsed. The count moved from 6 to 10 for that reason and not +because the exemption table changed — nothing in it was touched by this pass. + +Everything else on that file — tool names, library names, function names, dex/ELF constant +identifiers, protocol field names, CVE ids, hardening product names, the public crackme and its +URL, platform and SDK packages, `com.example.*` / `probe.synthetic.*` and the ``/``/ +`` placeholders — produced **no finding at all**, which is the intended outcome for the +do-not-anonymize list and is also the check that the two synthetic fixture prefixes named in +`references/desensitization-and-leak-scans.md` really are exempt: `com.example.app` sits on line +83 of that file and is absent from both the findings and the suppressed list, because +`BENIGN_PACKAGE_PREFIXES` contains `com.example`, `org.example`, `net.example`, `io.example` and +`probe.synthetic`. The file was written to hold every category of that list, one line each, so a +regression there shows up as a new finding rather than as a silent behavioural drift. + +`169.254.169.254` (the cloud metadata endpoint) is exempted **only** in this documented-endpoint +form. It is a real address that matters in a different context; the exemption is a judgement +recorded in `BENIGN_IP_PREFIXES`, not a pattern match, and it is the entry most worth +re-examining if this list is ever copied to another repository. + +### 3b. The leak this file carried — found by the scanner it documents + +The first version of this section listed all 30 hits **with their values**, quoted out of the +JSON, because a report with the value in it is the concrete kind. The Lead then ran the tool +against the repository and got `RESULT=leaks_found`, exit 1, with **26 strong/certain findings +and 14 weak ones — all 26 strong ones inside this file**, at lines 113–146 of the version that +existed then: the 5 bundle-id hits, 2 distinct device serials (12 hits), 2 provider-format tokens, +2 inline assignment values, 2 SDK appkeys and 3 host user names it had just finished quoting. +Nothing else in the tracked surface produced a strong hit. + +That is the failure mode this whole pass exists to prevent, committed by the pass itself, and it +states the boundary in one line: **the corpus is where identity is allowed; the published surface +is not.** `tools/_work/leak-precedent/` is git-ignored and excluded from the default scan root so +that it *can* hold the values — a corpus that does not contain them cannot demonstrate the rules +firing. This file is not in that category: it is tracked, it is read, and every string in it is +published. + +The correction, applied in this pass: + +1. The hit list in §3 now carries rule, category, strength, position, count and a **structural + description** of each value, in the repository's own placeholder notation. No evidence was + dropped: all 30 positions, all 17 rule ids, both strength labels and the per-category counts + are still here, and the corpus still produces all 30 when it is scanned directly (§3). +2. The two version-shaped literals in §4's false-positive table are now written in + `...` form. They are legitimate values, but leaving them would put two *different* + classes in the weak residue instead of one, and an unexplained second class is how a weak hit + starts getting ignored. +3. `references/desensitization-and-leak-scans.md` records the rule this produced — a report is a + copy of what it found, so the report is scanned like any other published surface — and its + exit table now carries the `leaks_found_strong_only` state. + +The scanner's own verdict on the corrected tree is in §4a. The lesson is the one the reference +file already made about entry points, applied to an evidence file: **run the scan on the file you +are about to publish, not on the file you started from** — and then do not paste the thing it +found back into that file. + +## 4. The repository scan — observed, and the false positives that produced it + +The first run against the real tree reported **28 findings**, of which 27 were wrong. The +sequence is recorded in full because the corrections are the useful part: every one of them was +found by running the scanner, not by reasoning about it. + +``` +$ python skills/apk-reverse/scripts/scan_leaks.py # first run +== leak scan: 118 file(s) scanned under E:\apk-reverse == + findings by category: device=4, endpoint=20, package=4 +== result: 28 finding(s) == RESULT=leaks_found (exit 1) +``` + +| # | Reported | Why it was wrong | Fix | +|---|---|---|---| +| 1 | a `jdk`-version-shaped run (`…\jdk-...`) ×19, across eight files in `docs/tool-verification/` | it is the JDK version in the toolchain's own path; the token `jdk` sits immediately before the match | a 30-character context window before the literal must not end in a version word (`jdk\|java\|python\|frida\|…`) | +| 2 | `0000000040001000`, `0000000000714240` | tombstone register/`pc` values — 16 hex digits, all digits | an all-digit 16-character token is suppressed; a real platform serial carries at least one letter | +| 3 | `0807060504030201` ×3 | a protobuf `fixed64` fixture in `EXTENSION-protobuf-raw.md` and in `scripts/protobuf_decode_raw.py` | same rule as #2 | +| 4 | `sg.vantagepoint.uncrackable3` | the published package name of an OWASP MASTG crackme | `PUBLIC_TARGET_PREFIXES` (public crackme/benchmark families are reuse, not identity) | +| 5 | `probe.synthetic.vmpdiff` | the synthetic manifest inside `vmp_diff_harness.py` | added `probe.synthetic` to the fixture prefixes | +| 6 | `args.package.spli` (twice truncated to `spli`) | a property lookup in `scripts/spawn_patch_detach.py`; the exemption window was measured from the *match* start, which an unbounded `[^\n]*?` prefix pushes past the `args.` | the window starts at the **capture group** (`m.start(group)`), and the exemption reason is printed so the path can be audited | +| 7 | `...` | an SDK version *directory* (`…/com.kwad.components.tachikoma/.../base-1_apk`) inside a private-storage path | a version-directory rule: a `.`-separated digit run enclosed in path separators is not an address | + +Two rows in that table are quoted in placeholder form rather than as the literal the first run +printed — row 1's four-part JDK version run and row 7's four-part SDK version run. Neither names +anybody: both are version numbers, and both were rewritten in this pass for the reason given in +§3b, because leaving them would put a second class (version runs, not addresses) into the weak +residue, where an unexplained class is the first thing a reader learns to skip. The shape is what +each row is about, and it survives the rewrite intact. + +### 4a. What this pass inherited, and what it leaves — observed + +Inherited state, reproduced at the start of this pass (same tree, before the §3b correction): + +``` +$ python skills/apk-reverse/scripts/scan_leaks.py +== leak scan: 128 file(s) scanned under E:\apk-reverse == + findings by category: appkey=2, device=12, endpoint=14, package=5, path=3, token=4 +== result: 40 finding(s) -- each one is a line to look at, not a verdict == +RESULT=leaks_found (exit 1) +``` + +26 of the 40 are strong/certain (`package` 5, `device` 12, `token` 4, `appkey` 2, `path` 3) and +every one of those 26 is a line of this file's former §3 hit list, at lines 113–146. The other 14 +are weak: 5 more lines of that same block (its endpoint rows), 2 version strings in §4's own +table, 3 RFC 5737 range literals in +`skills/apk-reverse/references/desensitization-and-leak-scans.md`, and `scripts/tls_check.py`'s +usage example. + +Post-correction, on the tree this pass leaves behind: + +``` +$ python skills/apk-reverse/scripts/scan_leaks.py --quiet +RESULT=leaks_found_strong_only (exit 0) + +$ python skills/apk-reverse/scripts/scan_leaks.py +== leak scan: 129 file(s) scanned under E:\apk-reverse == + findings by category: endpoint=4 + + skills/apk-reverse/references/desensitization-and-leak-scans.md:81:10 [endpoint/weak] plain_addr_ipliteral + match: + skills/apk-reverse/references/desensitization-and-leak-scans.md:81:26 [endpoint/weak] plain_addr_ipliteral + match: + skills/apk-reverse/references/desensitization-and-leak-scans.md:81:45 [endpoint/weak] plain_addr_ipliteral + match: + skills/apk-reverse/scripts/tls_check.py:44:23 [endpoint/weak] plain_addr_ipliteral + match: + +== result: 4 finding(s) -- each one is a line to look at, not a verdict == + all are `weak` (the documented address ranges); the gate is green under the default --fail-on strong, and `--fail-on any` is the strict pass +RESULT=leaks_found_strong_only + +$ python skills/apk-reverse/scripts/scan_leaks.py --quiet --fail-on any +RESULT=leaks_found (exit 1) +``` + +The four residual hits, by position, rule and class. Their individual values are deliberately not +reprinted here — that is §3b's correction, and re-printing them is how this file leaked the first +time. All four are the same class: + +| position | rule / strength | class | +|---|---|---| +| `skills/apk-reverse/references/desensitization-and-leak-scans.md:81` ×3 | `plain_addr_ipliteral` / weak | the three RFC 5737 documentation blocks, named inside that file's own statement that they are *not* exempt | +| `skills/apk-reverse/scripts/tls_check.py:44` | `plain_addr_ipliteral` / weak | the script's usage example, an RFC 5737 address | + +Reading: **zero strong/certain findings and zero unexplained hits on the tracked surface; four +weak ones, one class, both sources named.** The residue is the price of the design decision +recorded in §"Deliberately *not* exempt" of the reference file: the documentation that states the +decision has to name the ranges, and the ranges are exactly what the rule reports. That is why +the gate's pass condition lives in `--fail-on` and not in the exemption table, and why +`--fail-on any` stays red on this repository on purpose. + +The file count is 129 here, 128 in the inherited run above and 119 in the first pass: the tracked +surface grows while several writers work on it, and this pass added no file to the scanned +surface, so the count is a property of the run rather than of the leak. The number that is +comparable across the runs is the strong/certain one: **26 inherited, 0 now**. The inherited +figure also reproduces the Lead's own catch (§3b) exactly, which is the point of quoting the +transcript rather than the conclusion. + +Two work artifacts of this pass live beside the corpus in the same git-ignored directory, and +neither is part of the scanned surface: `EXTENSION-desensitization.before.md` (the pre-correction +snapshot of this file, kept so that the correction is auditable in both directions) and +`fixture-scan-before.txt` (a captured run of the fixture scan). Both contain the planted values +by construction, exactly like the corpus, and both are excluded from every default run — the work +area is where identity is allowed to exist; that is the whole of §3b's boundary. + +## 5. The exit-code contract — observed, all four states + +``` +$ python skills/apk-reverse/scripts/scan_leaks.py --root LICENSE --quiet +RESULT=clean (exit 0) + +$ python skills/apk-reverse/scripts/scan_leaks.py --quiet +RESULT=leaks_found_strong_only (exit 0) # the repository tree, weak hits only + +$ python skills/apk-reverse/scripts/scan_leaks.py --quiet --fail-on any +RESULT=leaks_found (exit 1) # the same tree, strict pass + +$ python skills/apk-reverse/scripts/scan_leaks.py --root tools/_work/leak-precedent/fixture --quiet +RESULT=leaks_found (exit 1) + +$ python skills/apk-reverse/scripts/scan_leaks.py --root tools/_work/nope-not-here +error: root does not exist: E:\apk-reverse\tools\_work\nope-not-here [stderr] +RESULT=error (exit 2) + +$ python skills/apk-reverse/scripts/scan_leaks.py --only nope +error: unknown category (or categories): nope [stderr] +RESULT=error (exit 2) +``` + +The token and the exit code are deliberately separate channels: a wrapper that only greps +stdout cannot tell "clean" from "the scanner never ran", and one that only reads `$?` cannot +explain itself in a log. + +`--fail-on` is what keeps those two channels from contradicting each other on a tree that is +documentation-shaped. Under the default (`strong`) a tree whose only findings are `weak` reports +`leaks_found_strong_only` and exits 0: the findings are still printed in full, they simply do not +fail the gate. Under `--fail-on any` the same tree reports `leaks_found` and exits 1, which is the +right verdict at publish time and the wrong one for a habit that runs on every change — a gate +that is red on its own design documentation is a gate people disable. + +## 6. What was not done + +- **The `RESULT=` token convention was undocumented when this pass started, and it is documented + now.** The Lead added `references/long-task-discipline.md` §Every script answers with a token, + not with prose during this pass, so the scanner is no longer the convention's only user. That + section's vocabulary is generic (`clean`, `patched`, `success`, `crash`, `timeout`, + `unavailable`, …) and does not spell out `leaks_found` / `leaks_found_strong_only`; whether + they belong in the enumeration is a Lead call. Verified with + `grep -n "RESULT=" skills/apk-reverse/references/long-task-discipline.md` → one match, line 70. +- **Nothing is wired into a gate.** No pre-commit hook, no CI job, no line in the maintenance + three-command habit. The scanner has never run automatically; every result above is a manual + invocation. Recorded as `unverified` in the sense that matters: whether it *would* be run is + untested. +- **No scan of the git-tracked list specifically.** The scan walks a filesystem root, so its + coverage is "the default surface under this root", not "the files that exist in HEAD". A file + that is present locally but ignored, or tracked but absent, is not distinguished. +- **No check of binary or archive content.** `*.apk`, `*.png`, `*.pcap` and `*.zip` are not + opened. Identity inside a screenshot or a capture file is real and this pass does not address + it. +- **No git-history scan.** A string deleted from the working tree remains in the history; nothing + here inspects commits. This pass did check the one case it created: `git status --short` shows + `docs/tool-verification/EXTENSION-desensitization.md` as `??` (untracked), so the values §3b + removed were never committed and nothing was written into history. That is a check of this + file, not a history scan. +- **No second-language rule set.** Identifiers that are not reverse-domain shaped (a nickname, a + local-part email, a CJK app name) are not covered. The rules are shape-based by construction. +- **No cross-repository validation of the exemption list.** The benign prefixes were derived from + this tree's own contents (a grep over its Markdown for reverse-domain tokens). Whether the same + list is right for another repository is `unverified`, and the reference file says so. +- **The `--max` / `--only` / `--exclude` paths were exercised only in the forms shown above.** + `--max` was not driven at all. + +## 7. One-line deliverable statement, and the open items + +**Deliverable (verified):** `python skills/apk-reverse/scripts/scan_leaks.py` scans a repository +root for six categories of target identity, reports each hit with a limited context window and a +strength label, exits 0/1/2 with a matching `RESULT=` token, reports all 30 planted leaks in the +reproducible corpus at `tools/_work/leak-precedent/fixture` (7 files, 10 further shapes admitted +through the exemption path), and against this repository's 129-file tracked surface returns zero +strong/certain findings — the residue is four `endpoint/weak` hits, all RFC 5737, both sources +named in §4a — so the default gate is green (`RESULT=leaks_found_strong_only`, exit 0) while +`--fail-on any` stays available as the strict pass (`RESULT=leaks_found`, exit 1). + +**Open items for the Lead:** + +1. Wire the scan into the maintenance gate (`check_repo.py`, `check_refs.py`, `scan_leaks.py`) + and record in `README.md` that the third one's exit 1 is a *content* finding, not a + mechanical failure like the other two. +2. Decide on a history scan. Nothing in this pass looked at commits, and a leak removed from the + tree is still in them. diff --git a/docs/tool-verification/EXTENSION-detection-pipeline.md b/docs/tool-verification/EXTENSION-detection-pipeline.md new file mode 100644 index 0000000..08507c1 --- /dev/null +++ b/docs/tool-verification/EXTENSION-detection-pipeline.md @@ -0,0 +1,311 @@ +# Extension verification — detection pipeline, layered descent, ptrace-free dumping + +Covers the pass that added the §Step 3 locating pipeline and the §Step 2B boundaries to +`references/detection-and-anti-analysis.md`, and the §What this route cannot do / +§Is this dex image real / §Layered descent sections to `references/advanced-unpacking.md`. +It also ships `scripts/svc_scan.py` and `scripts/anti_detect_probe.js`. + +Strength labels, as defined in this directory's README: + +- **observed** — reproduced here, with the exact command and its output. +- **inferred** — follows from observed facts, but the step itself was not executed. +- **unverified** — assumed, or reported by a tool and not independently confirmed. + +**Target identity is deliberately absent.** The hardened sample appears as ``; the +public MASTG challenge apps appear by their package names because they are public +published targets, not a user's app. No keys, signatures or user data are recorded. + +## 0. Environment and what was decided before measuring + +| Item | Value | +|---|---| +| Device | Android 11 (API 30), arm64-v8a, kernel **4.14.186**, Magisk alpha + root, toybox shell | +| Host | frida 16.7.19 / frida-tools 13.7.1; capstone; lief; pyelftools | +| Instrumentation server | **a `frida-server` was already running on the device** (`frida-server -l 0.0.0.0:27099`, pid 31859) with a leftover `adb forward tcp:27099` | +| Decision taken | **Reuse it rather than restart it.** A restart is a device-wide variable and the process had a live abstract socket (`unix:abstract=/frida-...)`; it was never started by this pass, so nothing about it is attributed here. Cost: the server's listening port and its path are part of the environment a detector can see — see §4, where a probe confirms frida artefacts *are* visible in the target's own maps | +| Root hiding | `magisk --denylist status` = *enforced*, list **empty** → no app had root hidden. Not changed in this pass | + +`frida-ls-devices` cannot be used as a liveness check in this harness (it requires a +Windows console and dies with `NoConsoleScreenBufferError`); the Python binding is the +reliable route, and that is what every measurement below used: + +```python +dev = frida.get_device_manager().add_remote_device('127.0.0.1:27099') +dev.query_system_parameters() +# {'arch': 'arm64', 'os': {'version': '11', 'id': 'android'}, 'api-level': 30, 'access': 'full'} +``` + +## 1. `scripts/svc_scan.py` — measured, with the count cross-checked + +`--help` works; a real call was made on five inputs. T3 cross-check first, because the +whole script is a decode claim: + +``` +json sites = 214 +hand word-scan = 214 +set difference = [] +``` + +The hand word-scan is a 6-line script that matches the `svc` encoding +(`(w & 0xFFE0001F) == 0xD4000001`) with no dependency on capstone, so the two readings +are genuinely independent implementations. **observed.** + +Device libraries, scanned from copies pulled with `su -c cp`: + +| Input | Sites | by_group | Notable | +|---|---|---|---| +| `/apex/…/bin/linker64` (1,451,392 B) | 214 | termination=4, evasion=5, discovery=8, unresolved=6 | the 4 termination sites are `exit`/`exit_group` at `0xda284`/`0xda2a4` | +| `/apex/…/lib64/bionic/libc.so` (1,193,024 B) | 215 | termination=4, evasion=6, discovery=8, unresolved=6 | `kill @0xc5b94`, `tgkill @0xc5bb4`, `ptrace @0xc5bd4`, `mprotect @0xc5f54`, `exit_group @0xc7054`, `exit @0xc7074` | +| `assets/libjiagu_a64.so` from the sample (1,171,896 B) | 21 | **unresolved=21** | every site is inside `scvtf`/`udf`/NEON byte soup — data, not instructions | +| `assets/libjiagu.so` (32-bit, 861,436 B) | 11710 | unresolved=11116 | scanning both PT_LOAD segments mixes code and data; the site count is not usable without inspecting neighbours | + +**The interpretation that matters, and it is the reason the script exists:** the four +termination sites in `libc.so` are libc's *own* exported implementations, so a hook on +`exit`/`kill` does see callers that go through libc. The shell library has **no** +termination site and 21 data artefacts — so on this target a libc-level hook is *not* +structurally excluded, and a missing exit event would be a finding about the hook. +**observed.** + +Two defects were found by running it and both are fixed in the shipped script: + +- `capstone` raises `CS_ERR_DETAIL` unless `detail = True` is set; the first run died on the + first hit (`AttributeError`-equivalent at `insn.operands`). +- `pf_x` is **not** usable as a filter on this platform. The ROM's own `libc.so` and + `linker64` carry `pf_x` set on their code segment but a first version of the probe read + `p_flags` from the wrong struct offset (reading `p_offset`'s low word instead), which + reported `pf_x=0` for everything. The shipped version reports per-segment `pf_x` and does + not filter on it by default: `--code-only` is opt-in precisely because a device that + clears the flag would otherwise return zero hits silently. + +## 2. The hardened sample: what it does now, and why nothing was dumped from it + +The previous pass recorded this sample as a clean self-exit on a 7-18 s cycle with no +tombstone. **That is no longer the shape.** Measured this pass, twice, 20 samples at 1 Hz +after `monkey` launch: + +``` +22:28:19.923 D/OppoExSurfaceFlinger: hecate traverse … Splash Screen com.gentle.ppcat#0 +22:28:21.684 W/ActivityManager: Process ProcessRecord{… 14905:com.gentle.ppcat/u0a623} failed to attach +22:28:21.686 I/ActivityManager: Killing 14905:com.gentle.ppcat/u0a623 (adj -10000): start timeout +``` + +- **observed:** the platform kills the process on **`failed to attach` / `start timeout`** + roughly **2 s** after start, i.e. the app never finishes starting. +- **observed:** an attempt to harvest memory inside that lifetime failed on both tries — + `RESULT=NO_PID` and `RESULT=NO_PID` with `t_pid=9.99`, so a 10 s window is already longer + than the process lives. Two strikes, no third variant: **the ptrace-free dump was not + obtained from this sample and this file does not claim one.** +- **observed, and a new signal rather than a repeat:** a tombstone is now produced, and its + fault is not in the packer's own code but in the runtime, reached from JIT output: + +``` +signal 11 (SIGSEGV), code 2 (SEGV_ACCERR), fault addr 0x75f16ff028 +backtrace: + #00 pc 0000000000574028 /apex/com.android.art/lib64/libart.so + (art::Runtime::SetJavaDebuggable(bool)) + #01 pc 0000000000022d58 /memfd:jit-cache (deleted) +``` + + A self-terminating `SIGABRT` would be the expected shape for an anti-analysis kill; a + `SEGV` in `Runtime::SetJavaDebuggable` reached **from JIT-compiled Java** is not, and the + honest reading is limited: **the app now crashes during startup, in the runtime, on a + path a Java method entered.** Whether that is the packer's environment check firing or a + consequence of the kill arriving mid-startup is **unverified** — the sample's own + start/stop behaviour changed between passes and no control run with root hiding + configured was made, because changing root-hiding state is a device-wide variable this + pass deliberately did not touch. + +## 3. Clean-target baseline: state, liveness, and a harness artefact avoided + +- **observed:** `owasp.mstg.uncrackable1` launched with `monkey` stayed alive at pid 15310 + for **more than a minute** of repeated checks — the stable baseline every dynamic + experiment needs, and the reason the arms below were run on it and on L3 rather than on + the hardened sample. +- **observed:** `owasp.mstg.uncrackable3` runs two processes (zygote64 + 32-bit), had its + own integrity log line, and stayed up. +- **observed, and the most expensive-looking failure that turned out to be free:** the + first attach to L1 hung and then failed — + `attach/spawn failed: unexpectedly timed out while waiting for signal from process with PID 15310` + — and a second attach said `process not found`. The process was still in `ps` and + `dev.enumerate_processes()` still listed it, but: + +``` +$ adb shell 'cat /proc/15310/stat | cut -d" " -f3' +D +``` + + **`D` (uninterruptible disk sleep)** is why. A userspace attach needs the process to run. + The one-line control that settled it in a single command: attaching to a *different* pid + from the same device succeeded (`attach 15974 OK`). **observed** — the failure was the + environment, not a detector, and this is now Stage 0 of the reference's pipeline. + +**Harness self-check applied before interpreting anything** (the discipline that a spawn +which is never resumed produces zero events and reads as a hang): the driver +(`tools/_work/detect/run_anti_detect_probe.py`, a work-file, not shipped) resumes +**explicitly** after the script is loaded, and records whether it did. Both arms below +report `resumed=True`, so "no events" cannot be an un-resumed spawn here. + +## 4. The three arms, on one target — and the detection actually found + +All arms: `owasp.mstg.uncrackable3`, `--spawn`, script loaded before resume. + +| Arm | Script | Result | +|---|---|---| +| **A. baseline** | none (launch, no instrumentation) | process alive and stable across a minute of sampling (§3) | +| **B. control** | attached/spawned, resumed, **no script loaded** | **`alive_at_end=True`**, survived the full window | +| **C. experimental** | `scripts/anti_detect_probe.js` (observer-only) | **`alive_at_end=False`**, process gone before the first heartbeat | + +Control vs. experimental differs in exactly one variable — the probe script — so the death +is attributable to instrumentation. Then the observer's own stream named the check: + +``` +t=0 [armed] script armed +t=207ms [thread_create] pthread_create entry=0x76dc031d78 +t=238ms [strcmp_probe] strcmp("Ldalvik/system/ClassExt;") +t=245ms [env_probe] open("/system/fonts/Roboto-Regular.ttf") +t=254ms [loader] android_dlopen_ext("…/oat/arm64/base.odex") +t=303ms [strstr_probe] strstr("frida") <-- the check +t~350ms process gone, heartbeats=0, no tombstone, no crash/ANR record +``` + +- **observed:** the target **searches for the string `frida`** (`strstr("frida")`) at ~300 ms + of its life. This is the "detection really did happen" datum the task asked for: not an + inference from a death, a recorded call with a timestamp. +- **observed:** an earlier arm of the same shape also captured the kill itself — + `[termination] raise(...) <- …` at **t=352 ms**, i.e. a signal-based self-abort roughly + 50 ms after the `strstr`. No tombstone was written and no `has died: crash` record + appeared, which is the clean-self-exit signature. +- **observed, and the reason the arm is reported as unstable:** three attempts at the same + arm produced *one* full sequence, then two runs where the script received **zero** + messages before the process died (`live=0, messages=0, heartbeats=None`). On a target that + kills itself in ~300 ms the observable window is sub-second and **not reproducible arm to + arm**. One successful attribution is therefore not a pipeline, and the reference says so. + +What the probe also recorded about the *environment*, before any check ran: + +``` +tracerpid=0 frida_named_maps=4 (four mappings whose names match /frida/i) +status: Name: tg.uncrackable3 ; State: S (sleeping) ; Seccomp: 2 +/proc/net/tcp -> Permission denied /proc/self/task -> Is a directory +``` + +**observed:** the target's own `/proc/self/maps` contains **four frida-named mappings** while +`TracerPid` reads **0**. So a `TracerPid`-only check would pass while a maps-or-thread-name +check would not — which is exactly the "what can this process actually see" question the +Stage 1 self-report exists to answer, and it is why the probe reports both. + +**One defect in the probe, fixed once and then left:** caller attribution printed +`bt-unavailable: cannot read property 'context' of undefined` on every line. Cause: the +`this` binding inside the anonymous `Interceptor` callback was not what the helper assumed; +the fix is to pass `this.context` explicitly. The measured consequence is worth stating +because it is a *plumbing* failure that reads like a hooking failure: the events and their +timings were correct throughout, only the "which module called this" half was missing, and +one of the three arms still showed no events at all. `observed`. + +## 5. ptrace-free dumping: the route, its boundary, and a cross-check that passed + +### 5.1 What the route yields when the target has a whole dex in memory + +Not re-measured here: the earlier pass's 17 ART dex mappings exported from `/proc//mem` +without any instrumenter present (`docs/tool-verification/EXTENSION-rootdump.md`). This pass +adds the **negative boundary** the earlier pass could not state, and it needed a clean +target to state it: + +``` +# both public MASTG targets, ~2,400 map lines each, one adb round-trip +owasp.mstg.uncrackable1: maps_lines=2427 named_dex=0 apk_views=2 +owasp.mstg.uncrackable3: maps_lines=2377 named_dex=0 apk_views=2 +``` + +- **observed:** `named_dex=0`. Neither target produces an + `[anon:dalvik-classes.dex extracted in memory from …]` mapping, so the filename-driven + export has **nothing to export** — and this says nothing about those targets being + protected. +- **observed, the explanation:** their `classes.dex` is **deflate-compressed inside the + on-device APK** (`compress_type=8` read from the APK pulled off the device), so the + `dexopt` "map the dex directly" mechanism cannot apply to it. +- **observed:** the `r--s` views that *do* exist point at APK **file offsets**, and the + bytes there are the APK's own bytes — at `0x11000` on L1 that content is the ZIP + **central directory** (`PK\x01\x02`), not a dex. +- **observed, and a trap that produced a false "DIFFERENT":** on L3 the view + `76dc28e000-76dc299000` has file offset `0x15a000` = 1,417,216 B, while the mapping is + 44 KiB (45,056 B) long — so the range reaches 1,462,272, i.e. **past the end of the + 1,460,555-byte APK**. The file-side read came back short (1,015 B) and the two sha256s + described different byte counts. Compute the length before suspecting the target. + +### 5.2 The reproducibility cross-check, which passed + +Two independent read paths over the same address range must agree byte-for-byte before a +capture is evidence: + +| Range | Read via `/proc//mem` | Read from the file | Verdict | +|---|---|---|---| +| L3 `765a89c000-765a8de000` (264 KiB, file offset `0x6000`) | `sha256=2eafd7cca73630c05d0b0c1b4a7709f21ee24540b5f2ba388ad3467af645aa75` | same | **IDENTICAL** | +| L1 `76dfb14000-76dfb15000` (4 KiB, file offset `0x11000`) | `sha256=74846b8cb02a764efef158194745466cf0c8ad2c1c3fecbb2bcf7cf9c58b6465` | same | **IDENTICAL** | + +**observed.** The comparison is only meaningful because both sides covered the same byte +count; the one range that differed was the one whose length exceeded its backing file. + +### 5.3 The cross-check that was **not** run, and why it is not silently absent + +The task's cross-validation requirement is satisfied for `svc_scan.py` by two independent +decoders agreeing on an identical 214-site set (§1). For a **dump**, the equivalent check +needs two producers that emit the *same* artifact. What this pass has is `/proc//mem` +(successful, §5.2) and, on the hardened sample, `frida-dexdump` — which was **refused** by +that sample in the previous pass (`script has been destroyed`, +`docs/tool-verification/EXTENSION-device-run.md`). The sample's lifetime then shrank to ~2 s +(§2), so the second producer could not be obtained on any target here: + +| Second-producer option | Status | +|---|---| +| `frida-dexdump` on the hardened sample | **unverified** — refused in the earlier pass; not retried here on a 2 s process | +| `frida-dexdump` on L1/L3 | **unverified, and expected to be empty** — `named_dex=0`, so there is no whole image for any dumper to find | +| `process_vm_readv(2)` as a second backend | **not run** — the on-device helper would have needed a second binary on the device, and the byte-identity question it answers was already answered against the file path | + +**Statement of the gap, so the next reader does not have to infer it:** the byte-identity +check between two *dumpers* is **unverified** in this repository. What is `observed` is +byte-identity between two *read paths* on the same bytes, and a 214-site decode agreement +between two independent decoders. Those are weaker claims than "two dumpers cross-checked", +and they are the ones this file makes. + +**A related statement that is deliberately negative:** two dumpers agreeing is not evidence +of *recovery*. Against an extraction shell, two agreeing dumps are two copies of the same +skeleton. The judge for that is the platform decoder, per +`advanced-unpacking.md` §Establish "the bodies do not decode" with a decoder you did not write. + +## 6. What was not established + +- **Not** a recovered payload from the hardened sample. Its process lives ~2 s and never + reaches a state where its payload is loaded; the attempt hit the two-strike limit and was + re-routed to the stable targets. +- **Not** a fix for the sample's startup failure. Root hiding was left alone — configuring + `magisk --denylist` is a device-wide change and this pass was the only lock holder, which + is not the same as being free to change every shared variable. +- **Not** a reproducible end-to-end anti-instrumentation pipeline on a fast detector. The + full sequence was captured once out of three arms and the reference labels it that way. +- **Not** a general claim about `strstr("frida")` being the only check, or about MASTG + challenges, or about Android 11. One device, one ROM, one session, with other experiments + sharing the device. +- **Not** a kernel-layer result. Kernel 4.14.186 < 5.10 closes the eBPF route; the layered + descent in the reference stops at the `svc` boundary and says so. + +## 7. Reproduction one-liners + +``` +python skills/apk-reverse/scripts/svc_scan.py tools/_work/detect/libc.so --context 2 +python skills/apk-reverse/scripts/svc_scan.py tools/_work/detect/linker64.bin --json + +adb shell 'su -c "sh /data/local/tmp/apkmaps.sh owasp.mstg.uncrackable3"' # named_dex / apk_views shape +adb shell 'su -c "sh /data/local/tmp/apkviews_verify.sh owasp.mstg.uncrackable3"' +adb shell 'su -c "sh /data/local/tmp/detlaunch.sh /data/local/tmp/dd1"' + +python tools/_work/detect/run_anti_detect_probe.py --pid --watch-ms 8000 --out arm.json +python tools/_work/detect/run_anti_detect_probe.py --pkg owasp.mstg.uncrackable3 --spawn \ + --script skills/apk-reverse/scripts/anti_detect_probe.js --out arm_probe.json +``` + +The on-device helper scripts (`apkmaps.sh`, `apkviews_verify.sh`, `detlaunch.sh`) and the +driver live under `tools/_work/detect/` and are **work files, not shipped deliverables** — +they exist so the commands above are reproducible, and their content is summarised in the +sections that cite them. The two shipped scripts are `scripts/svc_scan.py` and +`scripts/anti_detect_probe.js`. diff --git a/docs/tool-verification/README.md b/docs/tool-verification/README.md index 6dadc81..4c2e977 100644 --- a/docs/tool-verification/README.md +++ b/docs/tool-verification/README.md @@ -70,6 +70,21 @@ those are called out in the matrix rather than buried here. | `EXTENSION-split-apk.md` | Two real split sets through both branches: unified re-signing, the merge that is refused by design, the merge that succeeds, and the signer chain this host actually needs | | `EXTENSION-protobuf-raw.md` | A schema-free decoder cross-checked against the official runtime and a real DataStore container, with the packed-boundary and proto3-zero ambiguities reproduced on real bytes | +## Absorption-pass record + +A later pass absorbed mechanics from external skill repositories (`reverse-skill`, +`awesome-game-security`, `android-reverse-engineering-claude-skill`, `MobileRE-Skill`), keeping each +project's own discipline: only the phase, the checklist and the command pattern cross over, with the +source URL and the access date recorded, and nothing vendored as a runtime dependency. These files +live here rather than in the skill because two of them corrected a claim the repository had already +written down. + +| File | Covers | +|---|---| +| `EXTENSION-desensitization.md` | The leak scanner's rules and exemptions, its four `RESULT=` states and exit codes, the escaping false positives, and **the pass that found its own evidence file leaking 26 strong hits** — the fixture's literal match values, quoted to prove the rules fire | +| `EXTENSION-dart-aot-formats.md` | Two string-table claims tested against a real dual-ABI Flutter app: the arm64 packed scheme confirmed against literal tag bytes, and the assumed armv7 `len*2`/UTF-16 form **refuted** — the 32-bit record is `[header u32][byte-count u32le][UTF-8]`, and the extractor's zero is a format mismatch rather than an empty table | +| `EXTENSION-detection-pipeline.md` | Naming the check that fires: three arms on a public MASTG target (control alive, probe arm dead in ≈300 ms via `strstr("frida")`, with the `TracerPid=0` versus 4-frida-mappings asymmetry), `svc_scan.py`'s two decoders agreeing on an identical 214-site set, the layered Java-to-svc descent, and the ptrace-free dump boundary on targets whose dex is deflated inside the APK | + ## Adding a run Record the tools, versions and exact commands, and label each claim. Keep target-specific facts out: diff --git a/skills/apk-reverse/SKILL.md b/skills/apk-reverse/SKILL.md index e76fc90..35fa7ae 100644 --- a/skills/apk-reverse/SKILL.md +++ b/skills/apk-reverse/SKILL.md @@ -8,6 +8,21 @@ description: "Reverse engineer, debloat, de-ad, patch, or re-sign Android APKs, Goal: reach a **verified, installable, still-working artifact** fast — and avoid the whole class of mistakes that destroy an APK while looking completely healthy. +## Immediate — the four things to do before anything else + +Placed at the top on purpose: the longer this file gets, the less its middle is read, and everything +below is explanation for these four moves. + +**1. Classify before choosing a route** (§Start here's thirteen questions; **R4** decides whether you +are editing the right layer at all — a wrong branch produces an artifact that builds, runs, and does +the wrong thing). **2. Clear the four gates in order, with their pass criteria** (§Gates: G1 names the +deliverable form in one testable sentence *before* any work, G2 the machine's real capability, G3 the +code location, G4 baseline and control — "I understand the idea" is not clearing one). **3. A symptom +you cannot explain is a stop signal** — search §Symptom index for the shape *before your next attempt* +and load the row's file; those rows cost hours precisely because the answer was already written down. +**4. Two strikes on one shape of attempt → back to classification**, not a third variant +(§Stop conditions), and never report done without §What "done" means. + ## How to use this file This file is a **procedure with gates**, not background reading. Three things are mandatory: @@ -124,7 +139,9 @@ and de-obfuscation (`native-dbi-and-deobfuscation.md`) · protocol reversing bey (`kernel-and-environment-hardening.md`) · split APK / App Bundle sets (`split-apk.md`) · schema-free protobuf decoding (`protocol-reverse.md`) · Dex-VMP differential analysis (`vmp-differential-analysis.md`) · kernel-module templates and their version gate -(`kernelsu_syscall_mask.py`) · working from the phone itself (`on-device-tooling.md`). +(`kernelsu_syscall_mask.py`) · working from the phone itself (`on-device-tooling.md`) · publishing what +you learned without publishing the target, and reading a graded precedent before repeating work +(`desensitization-and-leak-scans.md`, `scripts/scan_leaks.py`, `precedents/`). **Dependencies this skill does not ship** — the Dart AOT workflow needs a snapshot dump this skill cannot produce, and naming which front end you used matters. Details: @@ -191,6 +208,8 @@ unread in this repository. | Deleting a library fixes validation but yields `UnsatisfiedLinkError: dlopen failed: library "X" not found` | `code-virtualization-and-custom-linkers.md` §the deadlock that eats hours | | Whole classes appear as bare `native` declarations with no body | `java2c-and-jni-sinking.md` — read it **before** dumping memory: if this is Java2C there is no DEX to find, at any point in the process lifetime. A handful of `native` methods in an otherwise ordinary dex is JNI sinking, not this | | A `Java_*` search over a hardened library returns nothing at all | `java2c-and-jni-sinking.md` §The JNI boundary — why a symbol search fails silently — dynamic registration, or `-fvisibility=hidden`. The check that works is "exports `JNI_OnLoad` and zero `Java_*`" | +| You are about to publish an evidence file, a transcript or a README that quotes real work | `references/desensitization-and-leak-scans.md` — run `scripts/scan_leaks.py` **before** it is committed; the hit list is a set of lines to look at, and `--show-exempt` is where the wrong suppressions show | +| A hooking module appears to have run but its log tag is silent, and you are about to record "it never loaded" | `references/precedents/logd-broken-module-never-ran-case-3.md` — a broken `logd` delivers nothing on `logcat` while the module's whole run sits in LSPosed's file log; read both channels | | A library's **SONAME does not match its filename** | `code-virtualization-and-custom-linkers.md`, `native-and-so.md` | | Your edit had **no effect at all**, with no error | `server-config-and-updates.md` §3 (the value may be server-sent), then `packers.md` §map the validation boundary | | Process **hangs** with no crash record, or dies to a `uid 0` killer | `native-tamper-and-suicide.md` §the rule (you probably made a terminate path *not return*) | @@ -205,6 +224,10 @@ unread in this repository. | You took screenshots but drew the conclusion from logs or from the patch itself | `long-task-discipline.md` §captures you never looked at are not evidence | | You are about to re-run an experiment whose result you already recorded | `long-task-discipline.md` §long-context decay | | A script will not start, or a tool "is missing" | `scripts/doctor.py`, then `toolchain.md` §"not on PATH" is not "not installed" | +| A hook or probe reports **no events at all**, and you are about to call it detection | `scripts/anti_detect_probe.js` for the environment self-report first, then `detection-and-anti-analysis.md` §Step 3: locating the check — the order of search from Stage 0 | +| `attach` hangs and then fails **while the process is still in `ps`** | `detection-and-anti-analysis.md` §Step 3 Stage 0 — check for `D` in `/proc//stat`, and attach a *different* pid as a one-line control before blaming the target | +| The app exits with no tombstone, no crash and no ANR record | `detection-and-anti-analysis.md` §Step 3 — a clean self-exit means the check ran before your hooks existed; the branch conditions there say which Stage | +| A dump region validates as the wrong thing, or an `r--s` view of `base.apk` looks like a dex | `advanced-unpacking.md` §What this route cannot do, and how to tell before you spend the window | | Feature-scoped network failure (login/register/pay) while the rest works | `tls-and-cert.md` — do not assume your patch caused it | | Everything works but **every signed request fails** after repack | `signature-derived-keys.md` | | A re-signed build **runs fine, renders its whole UI and logs no error — but one feature silently never loads**, and `dumpsys`/DNS/logcat show **no request for it at all** (not a rejected request: *no request*) | `code-virtualization-and-custom-linkers.md` §what the native check actually reads — refusing **before** the request is built. Not the row above: "sent and rejected" and "never sent" have different owners | @@ -456,7 +479,7 @@ Load only what the current step needs. | `references/dart-aot.md` | The logic lives in a Dart AOT snapshot (`libapp.so`): pinning the Dart version, building a matching decompiler, the object pool and reference indexing, register/boolean conventions, locating and patching Dart code | | `references/native-and-so.md` | Patching in a `.so`, needing code to run before the app's own code, hand-built native payloads that crash inside the linker, or **deciding which library/ABI is actually loaded and executing** | | `references/native-tamper-and-suicide.md` | The process dies on its own (no Java stack, or a native crash that looks like a bug); you are about to neutralise a `kill`/`exit`/`abort` path; or a hardened library's sections/function boundaries look wrong | -| `references/detection-and-anti-analysis.md` | The app fights back: it dies after you attach, refuses to run, detects root/hook/debugger, or your dynamic tool does not work here. **Read the first section before escalating** — the answer is usually to switch to static | +| `references/detection-and-anti-analysis.md` | The app fights back: it dies after you attach, refuses to run, detects root/hook/debugger. **Cost-first (A/B/C), plus the order of search** (see its Step 3 stage funnel) and when to go static | | `references/toolchain.md` | Choosing or invoking tools, something is not installed (including "not on PATH but present on disk"), a tool's output smells wrong, or you need to know which tools exist only as a GUI | | `references/ad-removal.md` | Task involves ads, trackers, sponsored cards, splash/interstitial/reward | | `references/updates-and-forced-upgrade.md` | The patched build must **keep working over time**; the app has any version check, forced-upgrade dialog, self-update installer, or hot-update/resource channel. Load this for essentially every build you intend to ship. | @@ -476,8 +499,10 @@ Load only what the current step needs. | `references/long-task-discipline.md` | The task will run long, or you are resuming one. Live record, conclusion grading, drift checkpoints, **deliverable-form drift (rooted-only vs shippable)**, bound-your-waits, **captures-you-never-looked-at**, **long-context decay**, handover | | `references/pitfalls.md` | Always worth a skim before building. This is the failure catalogue. | | `references/coverage-and-limits.md` | You need to weigh a claim before trusting it: the evidence behind each covered item, the dependencies this skill does not ship, and the record of what was never exercised | +| `references/desensitization-and-leak-scans.md` | You are about to publish anything derived from real work — evidence, a transcript, a README — or a leak scan reports a hit: what must be desensitised versus kept, the do-not-anonymize list, and how the scan gates a commit | +| `references/precedents/` | You are about to do a kind of work this repository has already converged on — a hardened Flutter target, a zero-event trace, a module whose log is silent, an equal-length patch taken to the device. Positive cases with graded evidence and dead ends | | `references/handoff-boundaries.md` | A routing decision is about to cross into another discipline: the JNI form table, the packer-versus-loader split, and what "verified" means for each of G1's four deliverable forms | -| `references/advanced-unpacking.md` | **The dump landed but the bodies are empty** (an extraction shell), or bodies decode as private opcodes (a real VMP): measuring the stub ratio, why FART's hooks died on Android 12-16, the root-side dump, and where recovery stops | +| `references/advanced-unpacking.md` | **The dump landed but the bodies are empty** (an extraction shell), or decode as private opcodes: the stub-ratio measurement, why FART's hooks died on Android 12-16, the root-side dump, its **boundary**, the layered descent, and where recovery stops | | `references/lsposed-and-modules.md` | The client-side logic is reachable but **a rebuilt APK is refused**: delivering a system-level hook module instead, its gradle-free build chain, scope configuration and verification, and the layer a Java module cannot reach | | `references/emulation-and-rpc.md` | You need to **call** a routine rather than change the app — a signing routine, a token, a cipher: emulated execution (Unidbg/Unicorn) with its environment-filling cost, versus service-ifying the live function over Frida RPC | | `references/native-dbi-and-deobfuscation.md` | A native function is an OLLVM state machine, or you need instruction-level execution evidence: Stalker traces, the trace-to-CFG route, the Stalker/QBDI/emulation decision, and the **measured** zero-event and crash boundaries | @@ -541,3 +566,6 @@ All scripts are parameterized and path-agnostic; pass paths explicitly. Run `--h | `scripts/protobuf_decode_raw.py` | **Schema-free protobuf decode**: hex / file / stdin to a JSON tree, every length-delimited field as a candidate set with ties labelled; `--reencode --check` for a byte-exact round trip | | `scripts/vmp_diff_harness.py` | Differential hardening for Dex-VMP: build a labelled opcode-coverage fixture (**218 of 224 opcodes, measured**), derive a candidate private-opcode map from an original/hardened dex pair, verify it in a closed loop, render smali | | `scripts/kernelsu_syscall_mask.py` | Generate a KernelSU/APatch syscall-masking scaffold: an installable userspace module plus KPM/LKM/eBPF kernel-side templates, each with its version gate and an explicit unverified label. **A userspace module cannot change a syscall return value** | +| `scripts/scan_leaks.py` | Scan a repository for target identity before it is published: bundle ids in manifest/`pm`/`ps` contexts, serial-shaped tokens, PATs, inline appkey assignments, literal endpoints, host user paths. Built-in do-not-anonymize exemptions, context-bearing findings, `--show-exempt` prints why a hit was suppressed, `--fail-on strong\|any`, exit 0/1/2 with a `RESULT=` token | +| `scripts/svc_scan.py` | Name the syscall behind an inline `svc` and the segment it sits in — which decides whether a libc-level hook can observe the call at all. `--context` shows neighbours, because a byte scan also matches data | +| `scripts/anti_detect_probe.js` | Observer-only probe (patches nothing): path/loader/thread/kill hooks with **caller module + offset**, an environment self-report (`TracerPid`, frida-named mappings), and live streaming so a sub-second target still yields evidence | diff --git a/skills/apk-reverse/references/advanced-unpacking.md b/skills/apk-reverse/references/advanced-unpacking.md index a3da7f4..b417e3c 100644 --- a/skills/apk-reverse/references/advanced-unpacking.md +++ b/skills/apk-reverse/references/advanced-unpacking.md @@ -28,7 +28,12 @@ end to end: 17 ART dex mappings exported from `/proc//mem`, page-alignment included, validated with `scripts/dex_dump_validate.py`; the evidence, the device-shell traps and the throughput numbers are recorded in `docs/tool-verification/EXTENSION-rootdump.md`. The `frida-dexdump` refusal that -motivates that section is in `docs/tool-verification/EXTENSION-device-run.md`. The +motivates that section is in `docs/tool-verification/EXTENSION-device-run.md`. A later +pass added the route's **negative boundary** — what it yields nothing against, and how +to tell before spending a process window — plus the reproducibility check between read +paths and the layered-descent table; those are measured on the clean MASTG targets and +on a hardened sample, with commands and hashes in +`docs/tool-verification/EXTENSION-detection-pipeline.md`. The FART/active-invocation half of the file is still **inferred** — no extraction-shell target has been spliced end to end here, and the two candidate samples shipped in `repos/CyReverse` turned out not to be extraction shells at all (their two "shell" @@ -381,6 +386,125 @@ loaded ART dex while the observer watched — which is the honest boundary of ev independently; the `code_off` cross-check and the class-count sanity check in §Measuring extraction are what catch that, not the magic search. +### What this route cannot do, and how to tell before you spend the window + +The sections above are the how. This is the boundary, and it is worth reading *before* +the first export, because the failure is silent: you export a region, it validates as +something else, and the conclusion "the payload is not resident" is drawn from a wrong +capture. Two measured obstacles and one measured non-obstacle. + +**A dex that is still compressed never materialises as an image.** `dexopt` can only +map a dex directly when the archive stores it uncompressed, so for an APK whose +`classes.dex` is stored with **deflate** the method never applies — and nothing in +memory is a whole dex. Measured on the clean MASTG targets (`owasp.mstg.uncrackable1`, +`owasp.mstg.uncrackable3`, both with `classes.dex` at `compress_type=8` in the +*on-device* APK): **zero** `[anon:dalvik-classes.dex extracted in memory from …]` +mappings across ~2,400 map lines each, i.e. `named_dex=0`. There was nothing to +export, and the route's negative says nothing about those targets being protected. + +**An `r--s` view of `base.apk` is not a source of dex bytes.** Both targets showed +exactly that shape — a read-only shared mapping of the APK — and it is the shape that +looks like an answer and is not one. Exported and compared byte-for-byte with the same +offset in the file: + +| View | Compared with | Result | +|---|---|---| +| `76dfb14000-76dfb15000` (L1, file offset `0x11000`, 4 KiB) | the APK at `0x11000` | **identical** (sha256 `74846b8c…` both) — it is the APK's own bytes, and at that offset they are the ZIP *central directory* (`PK\x01\x02`), not a dex | +| `765a89c000-765a8de000` (L3, file offset `0x6000`, 264 KiB) | the APK at `0x6000` | **identical** (sha256 `2eafd7cc…` both) | +| `76dc28e000-76dc299000` (L3, file offset `0x15a000`, 44 KiB) | the APK at `0x15a000` | **different** — and the reason is not tampering: the mapping range runs *past the end of the file* (offset `0x15a000` > 1,460,555 B), so the file-side read is short and the two hashes describe different byte counts | + +Two things follow, and the second one generalises. First, **the two read paths agree +exactly whenever they cover the same bytes** — `/proc//mem` and a straight file +read returned identical sha256 on both full-length comparisons, which is the +reproducibility check worth running before treating any capture as evidence. Second, +**when they disagree, compute the length before suspecting the target**: a VMA can +legitimately extend beyond its backing file, and "the bytes differ" then means "you +compared different amounts of data", not "something rewrote the file". + +**The non-obstacle, measured:** when the instrumentation route is refused, nothing here +needs fixing — root reads `/proc//mem` with no `ptrace` and no agent, and on the +test device that produced 17 real, parseable dex images from a hardened process +(`docs/tool-verification/EXTENSION-rootdump.md`). The correction this pass adds is +narrower than that claim: **the route's yield is a property of the target, not of the +route.** It yields everything the packer leaves as a whole image; it yields nothing +against a target whose dex never becomes one. + +### Is this dex image real? Two independent readers, or a claim you have not earned + +The route above produces candidate images; something else has to judge them, and the +judgement has to come from a reader you did not write. A single reading of a hard problem +is not evidence, and this file already carries the lesson in the opposite direction — +`§Establish "the bodies do not decode" with a decoder you did not write` is the case where +a hand-written decoder *invented* a finding. + +Three checks, in increasing cost: + +1. **Two read paths over the same bytes must agree byte-for-byte.** `/proc//mem` at + `(start, start+size)` versus the file or a second export. Measured above: identical + sha256 on every comparison that covered the same length. Do this **before** + interpretation — a capture that cannot be reproduced is not a capture. +2. **A producer you did not write must parse it.** `dexdump` + (`E:\tools\android-14\dexdump.exe`), `baksmali` or `jadx` — and quote *its* failure + count, not your own parser's. `scripts/dex_dump_validate.py` ranks and screens + candidates; it does not decide whether a body is real dalvik. +3. **Only then interpret** `stub%` / `emptied%` / class counts, per + §Measuring extraction instead of guessing. + +**Cross-checking two dumpers is only meaningful when both produce the same artifact.** A +byte-identity comparison is a real check between two implementations reading the same +address space (`/proc//mem` versus `process_vm_readv(2)`, or two independent export +tools); it is an empty check between a whole-image dump and a per-method `code_item` +harvest, because those are different artifacts and a byte difference says nothing about +either. State which kind of comparison you ran. A *cross-tool* agreement also does not +prove recovery: on a shell that stubs bodies, two agreeing dumps are two copies of the +same skeleton — which is why check 2's parse count and §Measuring extraction's ratios +both have to pass. + +## Layered descent: Java → JNI → Native → libc → syscall + +**Read this when an upper layer has gone silent.** "My hook produced no events" is not a +finding about the target until you know which layer is capable of seeing the code that +runs. The upper layers are cheaper and more informative, so the order is deliberate; the +mistake is skipping down for reassurance instead of on a signal. + +| Layer | What it sees | What it cannot see | The signal to descend | +|---|---|---|---| +| **Java** (framework hooks) | methods, objects, arguments — the algorithm in its readable form | anything the app does in a library; anything the framework compiled out of reach | no invocation ever fires, or only the first call and then nothing | +| **JNI boundary** | argument and return values of a *named* export; the boundary itself | argument *meaning*; any logic past the call | outcomes are correct but you cannot see why | +| **Native, named exports** (`Module.findExportByName`) | entries of exported functions; the call site that reached them | the function body's control flow (that is `native-dbi-and-deobfuscation.md`) | correct answers, opaque body; a `.so` that never calls its own exports | +| **Native, internal** (module base + offset) | code with no symbol at all | how you found the offset in the first place | a library whose exports are a thin shell over internal work | +| **libc** (`open`, `read`, `mmap`, `strstr`, `pthread_create`, `exit`, …) | every call that *goes through* libc, with caller attribution | calls that do not; anything above libc | an event count that contradicts observed effects | +| **`svc #0` sites** | nothing at runtime until you patch the site — it is a *static* layer | kernel-side: unreachable here, see the gate below | libc shows no such syscall while the effect happens | + +Three rules that keep the descent honest: + +- **Descending is cheap to state and expensive to trust.** A lower layer has less + context, so an event there needs *more* support to mean something: `open()` fired says + nothing on its own, while `open()` with a caller of `libX.so+0x1cef8` names a detector. + Always carry the caller offset down, or you have traded meaning for volume. +- **The bottom is static, not dynamic.** Below libc there is nothing to hook — a + `svc #0` site is found by scanning the library (`scripts/svc_scan.py`) and changed by + patching bytes. Measured: the ROM's `libc.so` carries exactly **4** termination sites + (`exit`, `exit_group`, `kill`, `tgkill`) and those are libc's own exported + implementations, which is *why* a libc hook sees callers at all; the same scan on a + shell library returned 21 byte-scan "sites" that were all data (no syscall-number load + near any of them). Read the neighbours before believing a count. +- **"Down" does not mean "more capable".** Measured on a real target, the decisive + detection ran at the **libc** layer — `strstr("frida")` at ~300 ms of process life, + followed within milliseconds by a clean self-exit with no tombstone — while the code + that recovered the caller chain sat in **JIT-compiled** runtime output, i.e. a + *higher* layer that has no file offset. The layers are not a ladder of access; they are + different observation points, and the useful one is wherever the target's check + happens to be. + +**The kernel layer is not reachable on this device, and the gate is a kernel version.** +eBPF tracing (`stackplz`, kprobes, `seccomp` filters pushed from userspace) requires +kernel **5.10+**; the measured device is **4.14.186**, so that entire layer is closed +here regardless of tooling. `kernel-and-environment-hardening.md` maps the route and its +version gate; do not spend a session on the layer before checking `adb shell uname -r`. +The consequence for this file: the descent above stops at the `svc` boundary, and a check +that runs *inside* the kernel is out of reach of both dynamic and static userspace work. + ## The honest boundary: real Dex VMP The last shape is the one to say out loud: if the dump's method bodies are intact but @@ -459,6 +583,11 @@ for the island itself. | Dumper runs, output dir empty | Storage path + `open()` check, then entry-type probe | §Why the classic hooks die on Android 12-16 | | App dies under the invoker | Shell counter-measures; narrow the force list | §Why the classic hooks die on Android 12-16, and `detection-and-anti-analysis.md` | | frida attach refused (`script has been destroyed`) | Root-side `/proc//mem` dump, atomic inside one process lifetime | §Dumping when frida is refused | +| Target has no `[anon:dalvik-classes…]` mapping at all (`named_dex=0`) | Its dex is compressed and never becomes a whole image — the route has nothing to export here; this is not evidence of protection | §What this route cannot do, and how to tell before you spend the window | +| An `r--s` view of `base.apk` looks like a dex region | Compare it with the file at the same offset and check the length: it is the APK's own bytes (often the ZIP central directory), and a range may exceed the file | §What this route cannot do, and how to tell before you spend the window | +| Two captures of the same address differ | Compare byte *lengths* first; a VMA can extend past its backing file | §What this route cannot do, and how to tell before you spend the window | +| A hook on the upper layer stops firing | Descend one layer on a named signal, not for reassurance, and carry the caller offset down | §Layered descent: Java → JNI → Native → libc → syscall | +| The effect happens but libc shows no such syscall | Static: scan the library for an inline `svc` site before assuming evasion | §Layered descent: Java → JNI → Native → libc → syscall | | Sample self-exits and restarts with no instrumenter present | The packer's own check, not your hook; shorten every step to one lifetime | §Dumping when frida is refused | | Methods are `native` stubs with a `Java_*` export | JNI sinking — read the function in the `.so` | `java2c-and-jni-sinking.md` | | Bodies present, decode as nonsense | Real VMP — **but confirm with `dexdump` first**, then differential oracle or walk away | §Establish "the bodies do not decode" with a decoder you did not write | diff --git a/skills/apk-reverse/references/coverage-and-limits.md b/skills/apk-reverse/references/coverage-and-limits.md index 8432b06..53aed0b 100644 --- a/skills/apk-reverse/references/coverage-and-limits.md +++ b/skills/apk-reverse/references/coverage-and-limits.md @@ -121,7 +121,11 @@ its own strength note at the top. problem — measure the ratio, name the recovery mechanism, stop when the target is a real VMP — but ships no modified ART runtime, no private-bytecode decompiler and no opcode-mapping derivation, and says so rather than presenting a memory dump as a recovery. `detection-and-anti-analysis.md` - decides by cost and often concludes "switch to static"; it is not a catalogue of evasion. + decides by cost and often concludes "switch to static"; it is not a catalogue of evasion. It now + also carries an order of search, plus a measurable environment self-report and an observer-only + probe — and the boundary that keeps those from becoming an evasion project is stated in the same + file (its Step 2B, two boundaries this file adopts): observers and interceptors stay separate + modules, and the cost of each added module is what decides when to leave for static. - **Kernel development.** `kernel-and-environment-hardening.md` maps the kernel-side route and names the version gate that decides whether it exists on your device; building and shipping a kernel module is outside this skill. @@ -169,6 +173,43 @@ trace of a real target). common shape there is *the tool was measured, the route was not* — read those files before treating any of the newer documents as a verified path. +**The detection pass added two tools and three boundaries, and its evidence is mixed.** Recorded in +`docs/tool-verification/EXTENSION-detection-pipeline.md`: + +- `svc_scan.py` — **measured**, and its whole claim is cross-checked: two independent decoders + (a hand-written word scan and the capstone-based scan) returned an identical 214-site set on a + device `linker64`, set difference empty. That agreement is what makes its output usable as + evidence about a *target's* library rather than about the scanner. +- `anti_detect_probe.js` — **measured, observer-only by contract** (it patches nothing, so a run + that uses it still describes the target it was pointed at). On a public MASTG challenge target it + recorded `strstr("frida")` at ~300 ms of process life followed by a clean self-exit, and it + reported the target's own view of the environment (`TracerPid=0` while **four** frida-named + mappings were visible in `/proc/self/maps`). The same arm captured the full sequence **once out of + three runs** — on a target that dies in ~300 ms the window is sub-second and not arm-to-arm + reproducible, which is why the reference labels it that way rather than as a pipeline. +- `scan_leaks.py` — **measured against a planted corpus and against this repository**. On the corpus + every rule fired (30 findings across 6 categories) and the exemption list produced zero false + positives on identifiers that must stay (tool names, SDK packages, dex constant identifiers, CVEs, + hardening products, public crackme names, placeholders). On this repository it found **26 strong + hits on its own evidence file**, which is the strongest evidence available that the class of leak it + targets is not visible by hand. `observed`. +- **The Dart AOT string-table formats** — the arm64 packed scheme is confirmed at the byte level + (tag byte equal to `0x80|(len<<1)` at every literal checked, 4,980 chained pool entries), and the + previously documented armv7 form is **refuted**: the 32-bit record is + `[header u32][byte-count u32le][UTF-8 payload]`, and the extractor's zero for that ABI is a format + mismatch rather than an empty table. `observed` for the formats; `unverified` for an end-to-end + patched build, because none was repacked and installed in that pass. +- **`observed`:** for the `ptrace-free` route, byte-identity between two *read paths* + (`/proc//mem` versus the backing file) on the same byte range, and the route's negative + boundary — a target whose dex is deflate-compressed inside its APK has **no** named dex mapping to + export (`named_dex=0` on both MASTG targets), so the route yields nothing there for reasons that + say nothing about protection. +- **`unverified`:** byte-identity between two independent *dumpers*. The second producer + (`frida-dexdump`) is refused by this repository's hardened sample, whose usable lifetime collapsed + to ~2 s this pass, and the clean targets have no whole-image dex for any dumper to find. The + weaker read-path check is what was run instead, and the distinction is stated in the evidence file + rather than papered over. + **Where the record lives.** `docs/tool-verification/README.md` indexes it; `tests/benchmark.md` is the public-target regression matrix. Neither ships inside the installed skill (they sit at the repository root), so an installed copy of the skill carries the claim but not its evidence file — diff --git a/skills/apk-reverse/references/dart-aot.md b/skills/apk-reverse/references/dart-aot.md index 8380619..14a5c16 100644 --- a/skills/apk-reverse/references/dart-aot.md +++ b/skills/apk-reverse/references/dart-aot.md @@ -24,6 +24,17 @@ print(m.group(1).decode() if m else 'not found') # e.g. b'3.9.0 (stable) (Mon If the string is absent, fall back to the snapshot hash the loader reports at runtime, or to the engine build id in `libflutter.so`'s version string. +**The regex above can return a version-shaped lie — assert the shape, do not just take the digits.** +Measured exception, on a real `libflutter.so`: the full banner string is +`0.0.1 on "android_arm64"` — a 66-byte engine version +record whose numeric field is `0.0.1` and whose **64-character build-id field is blank**, with no +`(stable)` marker and no `(Mon …)` timestamp at all. Anchoring only on `\d+\.\d+\.\d+` matches the +truncated prefix of a banner that was never populated. The discriminator that works: find the +` on ""` suffix, take the version string from the preceding NUL, and check that a +`(stable) (…)` tail is actually present **before** feeding the number to a VM-compiling tool +(§2's Route B). If it is missing, you cannot pin the version from this file — say so and use the +snapshot hash or a structural probe (§2's Route A), rather than building a decompiler for `0.0.1`. + ## 2. Get a decompiler that matches that version You need a tool that resolves the snapshot container — something that turns `libapp.so` into named @@ -69,6 +80,17 @@ release artifacts, so there is nothing prebuilt to fetch for a given Dart versio `build/blutter_dartvm__`) and try again; on the sample above the fault never reproduced after a relink — including for the unchanged binary — and three subsequent runs each completed in ~5.9 s. **One retry, then a relink, before you blame the target.** +- **Two Windows build failures that are not your target's fault, both measured `~0.2 h` each.** + *The compiler is not found even though `vcvars64` ran*: `%PATH%` expands when `cmd` parses the + line, so a `set PATH=...;%PATH%` in the same command overwrites the environment `vcvars` just + installed. Use delayed expansion (`cmd /V:ON` with `set PATH=...;!PATH!`) or run `vcvars64` in a + separate `cmd` invocation. *`string(REPLACE "/EHsc" ...)` aborts with "not enough arguments"*: + the `REPLACE` call is unguarded, so it breaks when `CMAKE_CXX_FLAGS` is empty — add an + `if(CMAKE_CXX_FLAGS)` guard, and patch **both** the template and the generated copy, or the next + configure regenerates the broken one. A prebuilt binary offered by a mirror is not a shortcut if + it is built for `aarch64`-Linux (the Termux target): check its ELF machine before spending time on + it — that mismatch cost `1 h` on the precedent this section is derived from, against `0.2 h` for + either compile fix above. Outputs of interest (blutter): @@ -212,6 +234,40 @@ Consequences that decide whether your search works: python dart_pool_strings.py libapp.so strings.tsv --min 3 ``` +### The 32-bit ABI does not use this format — and the tool will say so + +**On a 32-bit (`armeabi-v7a`) snapshot the packed tag scheme above does not exist, and the string +table is not laid out as a walkable chain at all.** `dart_pool_strings.py` enforces the chaining +constraint, so on armv7 it reports `kept (run >= 3): 0` — every candidate is isolated and discarded. +That zero is a **format mismatch, not an empty string table**: the same extractor over the arm64 +snapshot of the same app kept 4,980 chained entries. Do not read the armv7 zero as "the strings were +stripped". + +Measured on one app shipped with both ABIs (arm64 16,352,152 B / armv7 17,629,776 B): + +| ABI | Where a literal sits | How to find it | +|---|---|---| +| arm64 | inside the packed string table, `[0x80\|(len<<1)][payload]` | `dart_pool_strings.py`; the chain constraint makes it reliable | +| armv7 | **inline in the read-only data**, as its own record | a plain ASCII search for the literal — it is *not* UTF-16, so a byte search is the whole method | + +For armv7, the record a literal sits in is **4-byte aligned** and reads +`[header u32][length u32le][SAFE payload]`, where `length` is the **byte** count of the payload +(measured: an ASCII literal 12 chars long carried `12`, and a 70-char literal carried `70`). The +header u32 is a class/tags word that varies between snapshots — on the sample above it took the +values `0x00550238` and `0x00560238`, both 4-byte aligned, 13,397 occurrences. Treat the header as +a **validator, not a constant**: assert `length == payload length` and that the payload is +all-printable, then read the header word from the file rather than hard-coding it. Scanning the whole +17.6 MB image at 4-byte stride costs **0.94 s**, so validating every candidate is affordable. + +Consequence for patching: because the armv7 payload is **UTF-8 and length-explicit**, an equal-length +ASCII replacement is legal on both ABIs without recomputing the tag — the one-byte tag encodes the +length only on arm64, which is why the same patch script can serve both if it asserts the prefix it +found instead of assuming which encoding it is looking at. + +**Patch every ABI the app ships, or the device may run the one you left alone.** A phone that +prefers `arm64-v8a` still has the `armeabi-v7a` snapshot on disk, and a build that patches only one +of them is a controlled experiment, not a deliverable. + ## 8. Reading AOT code: three signatures that carry most of the weight Once you can disassemble a window with pool annotations, most business logic resolves into these @@ -289,6 +345,26 @@ Use this to find function boundaries when you need to delimit one. - **`libapp.so` inside an APK is usually deflate-compressed**, so in-place byte patching of the zip entry is not possible — patch the extracted file, then replace the whole entry (`repack-and-sign.md`). +- **Replacing a string in place is safe; replacing the *wrong* string is not.** The highest-value + equal-length target on a server-driven app is a **data key** — a JSON field name, a slot key, a + reporting label. Rename it to equal-length noise and the client looks up a key the server never + sends, so the feature goes quiet while every other request keeps working. This is the cheapest + surgical form of the "do not make an API fail" constraint at the top level of `SKILL.md`: the + request still succeeds, only the client's interpretation of it changes. +- **Never replace an API path or URL string.** It looks like the same kind of string and it is not. + Measured failure, on a real build: replacing the ad-fetch path produced a **404 whose error body is + not valid JSON**, the app's startup flow called `jsonDecode` on it, `FormatException` propagated + out of the Future that builds the home screen, and the app **stayed on the launch logo forever** — + `logcat` showed only `E flutter`. The ad did disappear; so did the app. Isolate it the way that + case did: a **re-sign-only control** (no string edits) plus `adb logcat -d | grep "E flutter"`, so + a startup failure is attributable to the patch rather than to the packer. Full precedent: + `references/precedents/`. +- **Longest string first when the targets are substrings of each other.** In a byte search + `welfare_ad` also matches inside `welfare_ad_top`, and `ad_click:` inside `ad_click:exp`; replacing + the short one first corrupts the longer entry. Sort candidates by descending length, and assert the + prefix byte you expect **immediately before** each hit — on arm64 that byte is `0x80|(len<<1)`, on + armv7 the length is the u32 immediately before the payload — so a hit that lands mid-string is + rejected instead of patched. ## 11. Traps specific to this layer diff --git a/skills/apk-reverse/references/desensitization-and-leak-scans.md b/skills/apk-reverse/references/desensitization-and-leak-scans.md new file mode 100644 index 0000000..a943b56 --- /dev/null +++ b/skills/apk-reverse/references/desensitization-and-leak-scans.md @@ -0,0 +1,222 @@ +# Desensitization and leak scans — what must leave, and what must stay + +A skill repository is published, and the material feeding it is real work: device transcripts, +packet captures, `ps` listings, log excerpts, an hour of shell history. Target identity leaks +into that material one line at a time — a bundle id in the middle of a `pm path` output, a +device serial echoed before a `dumpsys`, an SDK key pasted while debugging a download — and +**no structural check can see it.** `check_repo.py` validates layout, `check_refs.py` validates +anchors; both pass on a file that names a live app and a real phone. + +The failure this file prevents is two-sided, and the second side is the one that gets +forgotten: + +- **Under-redaction**: a published repository carries a target's identity, a device serial, or + a credential, and nobody notices until someone greps for it a year later. This is what + happened here once, by hand, after a pass had already been committed. +- **Over-redaction**: the same pass strips the *reusable* material along with it — the tool + names, the hardening-product names, the public crackme names, the protocol field names — and + the document stops being able to teach anything. A repository that has been scrubbed into + uselessness has failed at the same job from the other side. + +A rule that only handles the first side produces "a document about nothing". A rule that only +handles the second produces a leak. Both get fixed by the same artifact: an explicit list of +what is *exempt*, in code, that can be read and argued with. + +## Strength labels + +- **observed** — a command was run here and its output is quoted in the text or recorded in + `docs/tool-verification/`. +- **inferred** — follows from an observed fact or from documented mechanism; the step itself + was not executed. +- **unverified** — assumed, or reported elsewhere, and not reproduced in this repository. + +Claim strength in this file means *how well the statement about desensitization was measured*, +not how confident the writing sounds. + +## The two rules, stated so they can be argued with + +**1. Redact what identifies the target; keep everything that identifies the technique.** A +bundle id, a device serial, a host user path, an appkey, a live token, a non-loopback endpoint +are identity. A tool name, a library name, a function name, a protocol field name, a CVE id, a +hardening product name, a public crackme name and its URL are *technique* — they are how the +document transfers skill, and they are exactly what a naive redaction pass deletes first. + +The test that separates them: **would removing this string make the sentence less able to +teach?** `frida`, `apktool`, `libjiagu`, `UnCrackable-Level1`, `com.stub.StubApp`, +`RegisterNatives`, `proto3` all fail the test — removing them costs knowledge and protects +nothing, because they name a product, a tool or a public fixture. ``, ``, +`` pass — they carry the sentence's structure and none of its identity. And +`com..app` also passes the test, which is why a scanner must *report* the reverse +domain shape and let a human see what it is; it cannot decide. + +**2. A finding is reported with limited context, not a line number.** A report that says +`foo.md:412` makes the fixer reopen the file, read around the line, and decide what to delete — +per finding. A report that quotes ~48 characters either side means the decision is made from +the report itself. This is the same argument `verification.md` §Reporting makes for logs: an +artifact that does not contain the decisive information sends its reader back to the source. + +## What must be kept — the do-not-anonymize list + +Every entry here has been observed in this repository's own documents, and every one of them +is a wrong answer from a shape-only scanner. The list is materialized in +`scripts/scan_leaks.py` as `BENIGN_PACKAGE_PREFIXES`, `PUBLIC_TARGET_PREFIXES`, +`BENIGN_IP_PREFIXES` and the `CODE_IDENTIFIER_HINTS` table, so it can be extended with a +one-line edit and audited with `--show-exempt`. + +| Category | Examples | Why it is exempt | +|---|---|---| +| Tool and library names | `frida`, `apktool`, `jadx`, `libart.so`, `libfoo.so` | naming a tool is the content; the name is public and identical for every reader | +| Function and symbol names | `RegisterNatives`, `findExportByName`, `pthread_mutex_lock`, `JNI_OnLoad` | API vocabulary, not identity — and a large fraction of the reverse-engineering technique is exactly these names | +| DEX/ELF constant identifiers | `findExportByName`, `ImmutableDexFile`, `installPackageLI`, `LoadPackageParam` | 16-character camelCase runs that a serial-shaped regex matches; they are code constants | +| Protocol field names | `proto3`, `varint`, `packed repeated`, `length-delimited` | wire-format vocabulary | +| CVE identifiers | `CVE-2024-31317`, `CVE-2021-44228` | public advisory ids; redacting them destroys the reference | +| Hardening product names | `com.stub.StubApp`, `libjiagu.so`, `libDexHelper.so`, `com.secneo` | a *product* signature. It is what lets a reader recognise a shell, and it names no target | +| Public crackme / benchmark targets | `UnCrackable-Level1.apk`, `sg.vantagepoint.uncrackable3`, MASTG and its URLs | deliberately published fixtures; their package names are part of the published exercise | +| Platform and SDK packages | `com.android.settings`, `com.google.android.gms.ads`, `com.qq.e.ads.PortraitADActivity`, `androidx.work.WorkManager` | inventory. "Which SDK does this app carry" is a finding; the SDK's own name is not a secret | +| This repository's own fixtures | `com.example.*`, `probe.synthetic.*`, a probe module's package | synthetic by construction | +| Placeholders | ``, ``, ``, ``, ``, ``, `C:\Users\\...` | the redaction mechanism itself; a scanner that reports these is unusable | +| Loopback / unspecified / emulator addresses | `127.0.0.1:27042`, `0.0.0.0:8080`, `10.0.2.2:8080`, `169.254.169.254` (as a documented endpoint) | the frida and ADB idioms are these addresses; they identify nothing | + +**Deliberately *not* exempt, and observed as a finding here**: the RFC 5737 documentation +ranges (`192.0.2.0/24`, `198.51.100.0/24`, `203.0.113.0/24`). They are the correct way to write +a synthetic address, so a hit there is a signal rather than noise. Suppressing the range outright +would also hide the case where someone pastes a real address that happens to look synthetic. + +This repository pays for that decision with the *only* findings it produces, and the price is +recorded rather than argued away: **four `endpoint/weak` hits and nothing else**, measured on the +tracked surface with file count 129. Three of the four are the three blocks named in the sentence +above — this file stating the decision is itself the hit — and the fourth is +`scripts/tls_check.py`'s own usage example, a benign context the report makes obvious in one line. +Every other category is at zero. That ratio is why the gate's pass condition lives in +`--fail-on` and not in an exemption table: the documentation of the decision *is* the residue, so +a gate that failed on it would be failing on its own rationale. The transcript is in +`docs/tool-verification/EXTENSION-desensitization.md`. + +**What is a leak here, in contrast** — the shapes worth reporting: `package="…"` in a manifest +line, a bundle id after `pm path` / `pidof` / `ps -A` / `component=`, a 16-character +`[A-Z0-9]` run standing alone or beside a device word, `github_pat_…`/`ghp_…`, an inline +`APPKEY`/`appSecretKey`/`SECRET_KEY` assignment with a literal value, a non-loopback `IP:port`, +and `/home//`, `/Users//`, `C:\Users\\`. + +## What a scan can and cannot decide + +Decide in code: whether *this shape* is present, and whether a known-benign pattern explains +it. Never decide in code: whether the surrounding document is identity-bearing. + +Three consequences, each observed by writing the scanner for this repository: + +- **A range of four digits separated by dots is an IP address only if nothing nearby says + "version".** `JDK 17.0.4.1`, `build-tools 34.0.0` and `frida 16.7.19` are the shapes the first + version of the scanner reported twenty times across `docs/tool-verification/`. Context + (`jdk`, `python`, `frida`, … immediately before the match) separates them. +- **A 16-character uppercase token is a device serial only if it carries a letter.** Stack + addresses from a tombstone (`0000000040001000`) and hex protocol fixtures + (`0807060504030201`) are all-digit and were reported until that rule was added. +- **A reverse-domain string preceded by `args.` is a property lookup.** `args.package.split(".")` + in a script — observed in `scripts/spawn_patch_detach.py` — is not a bundle id, and the + exemption needs the *line*, which is why the exemption function takes it. + +Each of those is a **false positive that a reader would have had to investigate**. Three of them +were found by running the scanner against this repository, not by reasoning about it — which is +the argument for making the scan a gate rather than a document. + +## The scanner, and where it belongs in the pipeline + +`scripts/scan_leaks.py` — standard library only, `--root` explicit or defaulting to its own +repository, categories `package` / `device` / `token` / `appkey` / `endpoint` / `path`, text and +JSON output, and a fixed exit-code contract: + +| Exit | Token | Meaning | What a script does | +|---|---|---|---| +| 0 | `RESULT=clean` | nothing outside the exemption list | continue | +| 0 | `RESULT=leaks_found_strong_only` | findings exist, but every one of them is `weak` (an address from the documented range) | continue, and read the report — it is the publish-time checklist | +| 1 | `RESULT=leaks_found` | at least one `strong`/`certain` finding, or any finding under `--fail-on any` | fail the job and print the report | +| 2 | `RESULT=error` | bad usage, unreadable root, or a read error with no findings | fail the job as an infrastructure fault, not a content fault | + +The token is the machine-readable line; the exit code is the interface. Keeping them separate +matters because a wrapper that only greps stdout cannot distinguish "clean" from "the scanner +never ran", and a wrapper that only checks `$?` cannot explain itself in a log. + +Which findings fail the gate is chosen by `--fail-on`, and the two settings are two different +questions: + +- `--fail-on strong` (default) — only `strong`/`certain` findings fail. This is the maintenance + habit's setting, and it exists because of the RFC 5737 decision above: if the documented ranges + are reported on purpose, this repository is never `clean`, and a gate that is red on its own + rationale is a gate people learn to disable. +- `--fail-on any` — every finding fails, `weak` ones included. This is the publish-time setting, + and the one to run against the file you are about to publish rather than the tree you are + working in. + +Neither setting changes what is *reported*: weak findings are printed under both, and only the +exit code differs. The strict pass is therefore a decision recorded in a command, not a surprise. + +Placement, and the reasoning is about *when* the wrong answer is cheap: + +- **Not in the pre-commit hook alone.** A commit-time gate is the cheapest place to catch a + leak, but this repository's evidence files are written by long passes that commit once at the + end; a hook would fire on a tree nobody has finished writing. +- **In the maintenance gate, next to the other two checkers.** The habit that already exists is + `python check_repo.py && python check_refs.py` before a commit; a third line costs nothing and + runs at the moment the material is about to become permanent. Its exit 1 is a *content* + finding to read, not an error to retry — unlike the two checkers, whose failures are + mechanical. +- **Before publishing an evidence file**, once, by hand, with `--show-exempt`. The exemption + audit is where the interesting mistakes are: a value suppressed for the wrong reason looks + exactly like a value that was never there. +- **On the report itself, after it is written and before it is published.** A report that quotes + the hits in order to be concrete is a *copy* of the leak, in a file that is tracked and read. + This is not hypothetical here: the first version of the desensitization extension record did + exactly that, and the scanner flagged its own evidence file on the next run. Run the scan on + the artifact you are about to ship, which for an evidence file means the file itself. +- **In CI, on the published tree only.** A scanner run over a git-ignored work area will report + the work area — which is where the target identity legitimately lives — and a gate that cries + wolf on every run is a gate that gets disabled. `tools/` is excluded by default for exactly + this reason; a caller who points `--root` at it means it, and gets the findings. + +**What the scan does not replace.** It cannot see identity that is *described* rather than +*quoted* ("a mid-size Flutter app with a 360 shell"), and it cannot see a value that has been +partially masked on purpose. Neither is a defect to fix by adding patterns: the first is why +this repository's evidence files open with a statement that target identity is absent, and the +second is a judgement call a human makes once and writes down. + +## Failure modes + +| Symptom | Mechanism | What to do | +|---|---|---| +| The scanner reports a hundred hits, none of them a leak | shape-only rules against a document full of legitimate identifiers (versions, hashes, symbol names) | Do not widen the exemption list one hit at a time. Run `--show-exempt`, group the reasons, and fix the *rule* — a context-free regex is the bug | +| The scanner reports nothing and you believe it | a root that excluded the directory the leak was in; a text file with an extension the walker skips; a value inside a fenced block the reader never scrolled to | Read the header line — it prints the file count and the root. Then point `--root` at one directory and confirm the count is plausible | +| A real leak was suppressed | an exemption matched a substring (`` inside something else), or the benign-prefix list is too broad (`com.`-prefixed platform list catching a target that starts with the same two segments) | `--show-exempt` prints the *reason* per suppressed hit. An empty or vague reason is the defect | +| Same hit reported twice on one line | two rules whose scopes overlap (`SECRET_KEY=` is both a token and an appkey) | Expected, and left in deliberately: two independent rules naming one value is corroboration. Exact duplicates (same rule, same value, same line) are collapsed | +| The gate is green in CI and the leak is in the tarball | CI scans the tree; the artifact was built from a work area, or from a branch that was never scanned | Scan the directory the artifact is built from, with the same exclusions the build uses | +| A ``-style placeholder is reported | the placeholder uses a different bracket, or no brackets at all | Add the token to `PLACEHOLDER_TOKENS`. This is the one exemption class that should grow freely — placeholders cannot leak | +| The published report contains the leak it was written about | the report quotes each hit value in order to be concrete, and the report is the published artifact — a report is a copy of what it found | Never quote a hit value in a published document. Keep rule, category, strength, position and count, and describe the value by shape (`.`, `16×[A-Z0-9]`, `24×[0-9a-f]`) or with a placeholder. Scan the file you are about to publish, not the file you started from | + +## The entry-point surface + +A published skill is not only read by an agent; its **entry files are read as instructions**. +`SKILL.md`, a `README.md`, and the front-matter of any reference are the parts a host process +loads *before* it decides anything, and a repository assembled from third-party material — a +borrowed reference file, a contributed script — carries whatever those files say. This +repository's convention, observed in its own history: content reproduced from elsewhere is +summarised as prose with the source named, never quoted verbatim in a way that would land in an +instruction position, and a contributed file is read before it is linked. + +The practical consequence for redaction is the same as for the rest of this file: **the +redaction must not be reversible by asking the file.** A placeholder that is defined next to +its own value, a "sample transcript" that still carries the serial in a comment, or a finding +report pasted into a document that later becomes an entry point all reintroduce the identity +that a scanner would have caught — because the scanner ran before the report was written. Run +the scan on the file you are about to publish, not on the file you started from. + +## Cross-references + +- `references/long-task-discipline.md` §Keep a live record, not a log — the record is written + during the work, so it is where identity lands first, and the file most likely to be published + by accident. +- `references/verification.md` §Reporting — the evidence/report split this file mirrors: what + ships is neutral and mechanical, what identifies the target stays in the analysis. +- `references/precedents/README.md` — the case library; its "write back to the repository" + checklist is where a desensitization finding becomes an index line. +- `references/third-party-builds.md` — the same do-not-anonymize reasoning applied to a sample + you did not produce, where the identity is someone else's. diff --git a/skills/apk-reverse/references/detection-and-anti-analysis.md b/skills/apk-reverse/references/detection-and-anti-analysis.md index c7dd32e..4be9e2e 100644 --- a/skills/apk-reverse/references/detection-and-anti-analysis.md +++ b/skills/apk-reverse/references/detection-and-anti-analysis.md @@ -35,7 +35,7 @@ same app run without it.** Everything else that looks like detection is usually | app refuses to start at all | ABI mismatch, missing native lib for this device, install problem | check the live mapping (`scripts/lib_map.py`) and that a clean unmodified build starts | | app exits on this device only | emulator/root detection, **or** a device-state problem | run preflight; try the same build on a different device class | | app hangs forever | a frozen thread from a bad patch, or a real ANR | `pitfalls.md` P7 and the "never make it not return" rule | -| dynamic tool "cannot attach" | **the tool cannot work on this environment at all** | see Step 3 — this is the one people waste days on | +| dynamic tool "cannot attach" | **the tool cannot work on this environment at all** | see Step 4 — this is the one people waste days on | **Rule: reproduce with the control first.** Instrumentation-free run, then instrumented run, same build. That single comparison separates "the app detects me" from "my tooling is broken", and it takes @@ -66,7 +66,216 @@ detector you were not asked about. change the user asked for. That is the drift this skill treats as most expensive, in its runtime form (`long-task-discipline.md` §the most expensive drift). -## Step 3: when the tool cannot work here at all +## Step 3: locating the check — the order of search + +Step 2 decides *whether* to fight. This step is the part the two steps around it do not provide: +**once you know a check fired, in what order do you look for it?** The order matters because the +cheap probes answer most cases and the expensive ones only pay off after a cheap probe has already +narrowed the field. Every stage below names its input signal, the smallest action that answers it, +the evidence that makes it true rather than plausible, and where to go when it does not. + +Two shaping notes before the stages. + +**Order is the whole content here.** Reading this as a menu and starting at the middle is how a +session becomes an arms race: each probe you add changes the process further, so the later probes are +measuring a target you already modified. Do them in order and stop as soon as one answers. + +- **Stage order is a funnel, not a checklist.** Stage 1 is five minutes and rules out whole classes of + problem; Stage 5 requires the offsets Stage 2 produced. Skipping forward costs more than it saves. +- **Every stage ends in `observed` or in a stated negative.** "No hook fired" is only a fact once you + have shown the hook can fire on this target (`pitfalls.md` P25/P26 in tool form). + +**Strength note.** The stages are a restatement, for judgement purposes, of a six-phase +anti-instrumentation pipeline published in `index-login/MobileRE-Skill` +(`https://github.com/index-login/MobileRE-Skill`, retrieved 2026-09). That project's phase *modules* +are not vendored here and none of its scripts are used; what is taken is the **order** and the +branch conditions, which are the parts that transfer. Stages 2, 3, 5 and 6 are **observed** as +mechanisms on this repository's own device (evidence and exact commands: +`docs/tool-verification/EXTENSION-detection-pipeline.md`). Stage 4's kill path was reached once and +is unstable across runs, so treat its detail there as `observed`-once, not reproducible. +Stage 1 is `observed` and is the one that most often ends the investigation without an escalation. + +### Stage 0: rule out the two non-detection causes first + +Before hooking anything: is the process alive at all, in what state, and does it die the same way +with nothing attached? + +```bash +adb shell 'cat /proc//status | head -4' # state matters: D is uninterruptible sleep +adb shell 'cat /proc//stat | cut -d" " -f3' +``` + +**Observed failure that looks exactly like detection:** a target in **`D` (uninterruptible disk +sleep)** makes an attach hang and then fail, and makes a second attach report *"process not found"* — +while the process is still in `ps`. No detector is involved; a userspace attach needs the process to +run. On the measured run this was the actual cause, and it was read as "the app refuses +instrumentation" for one round. Check state before you blame the target, and check **whether attach +works on any other process on the same device** — that one-line control separates a broken toolchain +from a hostile target in a single command. + +### Stage 1: a benign probe, to establish that your pipeline can fire at all + +**Input signal:** nothing yet. **Action:** load an *observer-only* probe +(`scripts/anti_detect_probe.js`) that patches nothing. **Evidence it worked:** the probe's own +`armed` line and its environment self-report — `TracerPid`, which of your artefacts are visible in +the process's own maps, whether it can read `/proc/net/tcp` at all. **If nothing arrives:** the +problem is plumbing or liveness, not detection. Re-read Stage 0; do not escalate. + +**The self-report is the cheapest half of this file's whole subject, and it is usually skipped.** +Before asking "does it detect frida", ask "is frida visible to this process". Measured on the test +device, a probe spawned into a target read 4 frida-named mappings in the target's own +`/proc/self/maps` before any check ran — so "was it detectable" was never in doubt, and the +remaining question was only which check reads it. + +**Stream, do not batch, on a target that self-destructs.** A probe that reports only at the end of a +window reports *nothing* when the target dies inside the window — precisely the run that mattered. +Measured: three spawn-and-probe arms of one target; one delivered a full event sequence ending in a +self-destruct, the next two lost the script before its first timer fired. Emit a heartbeat and stream +each first hit. The heartbeat's absence at time *T* is itself the datum: it separates "the target was +quiet" from "the script was gone before *T*". + +### Stage 2: name the detector, by caller offset + +**Input signal:** Stage 1's environment events. **Action:** hook the path-access and process-control +surface (`open`/`openat`/`fopen`/`access`/`stat`/`readlink`, `dlopen`/`dlsym`, `pthread_create`, +`kill`/`tgkill`/`exit*`) and report, for each call, the **caller module + offset**. **Evidence:** +`libX.so+0x1cef8` — a named module and an address. **If no caller resolves:** you are on a runtime +that cannot unwind here; record the module and move to Stage 6's static path instead of adding a +third unwinding strategy. + +This stage's product is a **name**, and everything after it is cheaper. Two branch conditions decide +which Stage 4 branch you take, so they are worth reading off explicitly: + +| What Stage 2 shows | Branch | Why | +|---|---|---| +| the check runs on a thread the detector created itself (`pthread_create`/`clone` caller is the detector module) | **block at creation** | the polling loop never exists to fire | +| the check runs inline inside a library constructor, and the calls you can see are only libc wrappers | **Stage 3** | under that ordering a libc hook is the wrong layer | +| the detector is a *shell* library that also decrypts | **do not patch broadly** | the same module carries load-bearing work; scope to named offsets | + +### Stage 3: if it fires before your hooks exist, take the constructor order + +**Input signal:** the death happens before or during first library load, or Stage 2's caller offsets +all sit in a constructor path. **Action:** find the constructor dispatcher and hook it *before* +constructors run. **Evidence:** the module's `init_array` entries are visible and their addresses +map to the checks. **If the device's linker does not expose the symbol:** switch to the static route +(`scripts/svc_scan.py` on that library) rather than guessing an address. + +```bash +# the symbol name is mangled; grep for the substring, not the bare name +readelf -sW | grep call_constructors +``` + +**Measured:** on this repository's device the symbol exists as +`__dl__ZN6soinfo17call_constructorsEv` at `0x4a258` — a **mangled** name. A search for a bare +`call_constructors` export reports "absent" and sends the analyst to an offset-guessing path that +was never necessary. Confirm presence/absence before building either route. + +**The measured cost of getting this stage wrong:** do not clear a module's whole `init_array`. A +shell library's constructor does payload decryption as well as detection; emptying it converts a +detectable app into a non-running one, which is a worse result than the detection was. Scope to +functions you have a reason to suspect — a per-function static scan for the syscall form in Stage 6 +is how you get that reason. + +### Stage 4: keep the process alive — deliberately, and with a stop condition + +**Input signal:** every attempt shows events up to a point and then the process is gone. **Action:** +make the termination observable (record path and caller) and, if necessary, prevent it. **Evidence:** +the death is attributed to a module and an offset, not to "the platform". **If attributing it costs +more than three arms, stop attributing and switch to static.** + +Two measured shapes, and they need opposite treatment: + +- **A libc-visible kill.** Measured on a clean MASTI-style target: a probe saw + `strstr("frida")` at ~300 ms and then the process was gone, with **no** tombstone, no crash and no + ANR record — a clean self-exit. The abort path itself surfaced once as a libc `raise`/signal call; + it did not surface on the later arms that died sooner. That instability is the finding: **on a + fast detector the observable window is sub-second and not reproducible**, so a single successful + attribution is not a pipeline. +- **A libc-invisible kill.** If a module issues `svc #0` itself, no libc-level hook can see it, and + the absence of an event beside a death is the signal to go static (Stage 6). + +**`exit_blocker`-style keepalive has a failure mode that looks like success.** Blocking the exit +turns a clean death into a spinner: the process stays alive, stops responding, and you can no longer +tell "the check is neutralised" from "the app is wedged". Keep the blocked-call count, a timeout, and +one behavioural check — does the UI still reach the state you care about — or the keepalive is +measuring itself. + +### Stage 5: shellcode and runtime-generated code + +**Input signal:** Stage 2's loader stream shows executable mappings appear that were never loaded +from a file. **Action:** watch allocation with execute permission and read-only-executable +transitions; disassemble around the allocation's caller. **Evidence:** an `mmap`/`mprotect` call with +execute permission whose caller offset is in the detector module. **If the region is unreadable:** +stop; the offsets from Stage 2 are all a static session needs. + +A target can make itself harder to look at without doing anything exotic: on one measured run the +recovered caller chain sat inside `jit-cache`, i.e. **Java code the runtime had JIT-compiled**, not a +shipped library. A crash frame in JIT output cannot be mapped to a file offset. Expect it on any +API where the framework compiles reflection at runtime, and treat "the frame has no file" as a fact +about compilation, not as a hidden module. + +### Stage 6: the precise stop, and the static alternative to it + +**Input signal:** a named module + offset from Stage 2, Stage 3 or Stage 5. **Action:** neutralise +that site — or decide not to. **Evidence:** a fresh control run with the site left alone dies where +it died before, and the patched run does not. **If the offset came from a trace you could not +reproduce:** do not patch it; go to the static route below. + +The static route is often strictly better than the dynamic one and needs no live process: + +```bash +python scripts/svc_scan.py /data/local/tmp/libDetect.so +``` + +**Measured discriminator.** A module that contains **no** inline `svc` site for a termination syscall +cannot be bypassing libc for its kill — so a missing exit event is then a finding about your hook, +not about the target. Measured on this device: the ROM's `libc.so` carries exactly 4 termination +sites (`exit`, `exit_group`, `kill`, `tgkill`), and those are libc's *own* exported implementations — +so hooking `exit`/`kill` really does see callers that go through libc. A shell library on the other +hand showed 21 byte-scan "svc sites" that were all **data**, with no call-number load anywhere near +them: a byte scan for `svc` matches inside data, so read the neighbours before believing a count. +That distinction — structural absence vs. observed absence — is what Stage 6 is for. + +### The routing chain, on one page + +``` +Stage 0 process state + does attach work on ANY process? -> D-state / broken toolchain, not detection +Stage 1 observer-only probe + environment self-report -> what the target can see +Stage 2 path/loader/thread/kill hooks with caller offsets -> a module+offset name + |-- caller is a created thread -> block at creation (Stage 4, branch A) + |-- death precedes hooks, or the checks are in a loader ctor -> Stage 3, or static (Stage 6) +Stage 3 constructor order / mangled symbol check -> the check is reachable at all +Stage 4 attribute the kill, then decide whether to block it -> attributed death, or "unreproducible" +Stage 5 executable mappings not backed by a file -> shellcode / JIT, often a dead end +Stage 6 NOP one site, or prove statically that libc is enough -> observed before/after +``` + +## Step 2B: two boundaries this file adopts + +**(a) Observers and interceptors are separate modules and stay separate.** An observation module must +not change behaviour, and an intervention module must not double as a monitor. This is the discipline +the pipeline above is built to survive: Stage 1's value is that its output describes the *original* +process, and the moment the same script also blocks an exit or NOPs a function, every later +observation belongs to a target that no longer exists. The practical version: a probe that watches +and a patch that neutralises are two files, loaded in two runs, and the runs are the experiment. +`scripts/anti_detect_probe.js` is the observer half and patches nothing by contract. + +Be honest about the leak in that contract: **hooking is itself observable.** Attaching changes +timing, and on a hardened target the first event the probe recorded was a `dlopen` of the target's +own code followed within ~50 ms by a check — i.e. the act of being watched is part of the +environment the target is reacting to. Separation makes runs comparable to each other; it does not +make them a description of the untouched process. + +**(b) Name the cost before you add another module, and know when to leave.** Each avoidance module +is another thing the target can notice and another reason a later result is unattributable, and the +detection is not standing still: a check that was a string compare becomes an inline syscall becomes +a native check in a module you have not looked at. Track three numbers, and stop when two of them +are true: how many bypass modules are loaded, whether the run needs a keepalive to stay up at all, +and whether the target only behaves under the full stack of modules. **When they are true, switch to +static** — that is the same conclusion as Step 2B, reached with a threshold instead of a judgement +call, and it is the exit this whole step needs so that Stage 6 does not become a permanent home. + +## Step 4: when the tool cannot work here at all Some environments cannot run a given dynamic tool, and no amount of trying will change that. Recognise it and move on — this is a *finding about the environment*, and recording it prevents the next person @@ -91,12 +300,12 @@ also needed the app to run, use the device-level observables instead: does the U fall silent, does a file appear or stop appearing. Those are properties of the shipped artifact, and they do not care that you could not attach. -## Step 4: what a detection layer means for the deliverable +## Step 5: what a detection layer means for the deliverable Two distinct questions — keep them separate, because conflating them produces either a broken artifact or an unnecessary retreat. -1. **Does it block your analysis?** → handle with Step 2/3. This is your problem, and it is temporary. +1. **Does it block your analysis?** → handle with Step 2/3/4. This is your problem, and it is temporary. 2. **Will it block the patched build on the user's device?** → this is a property of the artifact. A root check that merely requires an unrooted device is usually irrelevant to a repackaged APK. A **self-integrity or signature check** is a different matter entirely, and it belongs to @@ -109,7 +318,7 @@ it?* Only the first kind follows your build into delivery. environment and an illegitimate deliverable when the request was an installable build for a normal phone. Say which one you actually have. -## Step 5: keep a one-line environment fact +## Step 6: keep a one-line environment fact When you conclude that something cannot be done in this environment, write one line in the task record: @@ -123,6 +332,13 @@ and which will mislead the next reader. ## Checklist +- [ ] **Located before escalating:** Stage 0 state check, an attach control on another process, then the + stages in order — and stopped at the first stage that answered +- [ ] **Observer and interceptor are not in the same script**, and the observation run was kept free of + any patch +- [ ] Detector named as **module+offset**, or the negative stated ("no caller resolvable on this runtime") +- [ ] **Cost measured, not felt:** bypass-module count, whether a keepalive is required, whether the + target only runs under the full stack — two of three true means switch to static - [ ] Controlled the comparison: same build with and without instrumentation - [ ] Preflight run, so device state is excluded before blaming a detector - [ ] Classified: detection vs environment vs my own tooling diff --git a/skills/apk-reverse/references/long-task-discipline.md b/skills/apk-reverse/references/long-task-discipline.md index 7c2de0e..4012531 100644 --- a/skills/apk-reverse/references/long-task-discipline.md +++ b/skills/apk-reverse/references/long-task-discipline.md @@ -56,6 +56,32 @@ Most long-task damage comes from hypotheses drifting upward into "facts" simply you catch yourself referring to something as established, check the grade in the record. If it is not observed, go observe it or label it again. +## Every script answers with a token, not with prose + +A long task is a chain of decisions, and each decision reads the *result* of the step before it. When +that result is human-readable prose, the reading step becomes an interpretation — and the +interpretation is where a wrong conclusion enters with no error message attached. This repository has +already paid for it once: a `logcat`-only verdict concluded that a hooking module never ran, and the +module had run nine times. + +So a script in this kit ends with a machine-decidable answer, and the exit code carries the same +meaning: + +- **A final `RESULT=` line**, from a small closed vocabulary (`clean`, `patched`, `success`, + `crash`, `timeout`, `unavailable`, …). The token is the answer; everything before it is evidence for + a human. +- **Exit codes with fixed meaning**: `0` the token names success, `1` a real negative finding, `2` the + script could not do its job (usage, unreadable input, missing dependency). **A `2` is never a + finding about the target** — that distinction is the whole point, and conflating it turns a broken + harness into a property of the app. +- **Counts as tokens too** where a count is the measurement (`PROBE_COUNT=`, `VULN_COUNT=`, + `events=0`), so "nothing happened" is a value you can branch on rather than an absence you have to + notice. + +The rule for the reader is the other half: **decide from the token and the exit code, and only then +read the prose.** When the two disagree, the token is what the script's author defined and the prose +is what they were thinking that day. + ## Single-variable discipline across the whole task The most common source of a wrong *and durable* conclusion is a compound experiment: two changes, one diff --git a/skills/apk-reverse/references/precedents/README.md b/skills/apk-reverse/references/precedents/README.md new file mode 100644 index 0000000..f63b4b7 --- /dev/null +++ b/skills/apk-reverse/references/precedents/README.md @@ -0,0 +1,97 @@ +# Precedent library — what a pass got right, and how it got there + +`pitfalls.md` is the negative catalogue: what went wrong. This directory is the other half — +**cases where the work actually converged**, recorded so the route can be walked again instead +of re-derived. The two do not overlap on purpose: a case here points at `pitfalls.md` by entry +number rather than restating the failure, and a pitfall that already has a case does not need a +second copy in prose. + +Read a case when you are **about to do this kind of work again** — not when you are stuck. A case +answers "what sequence produced a defensible result, and what did it cost"; when you are stuck +the symptom index in `SKILL.md` is the faster route. + +## Why this exists at all + +This repository's most expensive recurring loss is not ignorance. It is **a conclusion that was +correctly retracted and then re-adopted**, because the retraction lived in a conversation while +only the original claim was ever written to a file. Three of the cases below are exactly that +shape: an attribution that was wrong, corrected by measurement, and corrected again. In each one +the *wrong* version is the one that reads as more coherent, which is why it is the one that would +be believed by a reader who only saw the summary. + +A case therefore records the execution chain **including the dead ends** — the wrong turn, and +the measurement that killed it. Without that part, the case is a success story, and success +stories teach nothing that survives the next surprise. + +## The cases + +| Case | Route it establishes | Strongest claim in it | Grade | +|---|---|---|---| +| `flutter-plus-shell-case-1.md` | classify a hardened Flutter sample before trusting any dynamic result: a pid that keeps moving has to be attributed before it is explained | the same drift was attributed to frida, then to memory pressure, then to the sample's own root-environment check — only the third survived | observed (drift and its cause), unverified (that this generalises) | +| `frida-spawn-hangs-zero-events-case-2.md` | a harness can manufacture the very negative result it is looking for: `spawn` leaves the process suspended, and `CONFIG.autoStart` follows before your configuration lands | a zero-event trace was produced by two independent harness defects, not by the target | observed (the mechanism, and the warning that now fires) | +| `logd-broken-module-never-ran-case-3.md` | decide whether a hooking module ran at all, on a device whose log daemon is broken | nine successful injections produced **zero** `logcat` lines; the only evidence was a file under `/data/adb/lspd/log/` | observed (both channels), inferred (the framework-bridge explanation) | +| `l1-equal-length-patch-case-4.md` | the full chain from a two-byte dex edit to a visible change on screen, with a control build | the dialog disappeared **and** the control build failed the same way first, which is what makes the four bytes responsible | observed end to end | +| `manual-grep-finds-what-nobody-grepped-case-5.md` | why a leak scan is a gate and not a document, told from this repository's own near-miss | a shipped pass had to be corrected after the fact for identity a human grep found | observed (the omission and the correction), inferred (that a scanner would have caught it) | + +Each case ends with an action checklist naming the **repository files that should be written +back to** if you hit the same thing. A case that does not produce a file edit has not finished. + +## Template + +Copy this skeleton. Keep the headings — the read-back check depends on them, and the grade +column is the part that stops a hypothesis from drifting upward into a fact. + +```markdown +# — one line naming the conclusion, not the activity + +## Metadata +| Field | Value | +|---|---| +| Context | what was being done, in one line | +| Cost | what it cost before it converged, measured | +| Outcome | what artifact or conclusion exists now | +| Evidence | the evidence files, by path | +| Related | `pitfalls.md` entries, and the neighbouring reference files | + +## Assertions and grade +| # | Assertion | Grade | Evidence | +|---|---|---|---| +| 1 | ... | observed / inferred / unverified | exact command, file, or line | +Reserve `observed` for a claim with a command and its output behind it. An assertion whose +evidence is "the mechanism implies it" is `inferred`, and saying so is not hedging. + +## Execution chain (including the dead ends) +The wrong turn, the measurement that killed it, and only then the route that held. Numbered, +because the order is the information: a reader who starts from the conclusion will pick the +wrong sequence. + +## Pits +| Pit | Cost | What it was mistaken for | +|---|---|---| +| ... | measured number, or a count of re-runs | ... | +Use a cost you measured. "Two round trips" is a measurement; "a while" is not. + +## Reusable pattern +The part that transfers to a different target with no editing. If it needs this target's names, +it belongs in the chain above instead. + +## Write back to the repository +- [ ] `<file>` — <what to add to it, so the next reader does not pay this again> +- [ ] `<file>` — <index line, if a new file was created> +``` + +## Rules + +- **Grade every assertion, and grade it down when in doubt.** `observed` means a command was run + and its output is in `docs/tool-verification/`. "I reasoned it out" is `inferred`. This is the + same standard `references/long-task-discipline.md` §Grade your own conclusions sets for a live + record, and it is the reason a case is worth reading a year later. +- **Record the retraction with the claim.** If an earlier conclusion was wrong, both versions + belong in the case: the wrong one, why it was wrong, and what replaced it. Deleting the mistake + loses the most transferable thing here. +- **Do not restate `pitfalls.md`.** Point at the entry. A duplicated failure mode drifts out of + sync with its original and then contradicts it. +- **One case, one route.** A case that establishes three unrelated things establishes none of + them; split it. +- **A case with no write-back checklist is a diary entry.** The last section is what makes it + knowledge the repository holds rather than knowledge the author had. diff --git a/skills/apk-reverse/references/precedents/flutter-plus-shell-case-1.md b/skills/apk-reverse/references/precedents/flutter-plus-shell-case-1.md new file mode 100644 index 0000000..95eb278 --- /dev/null +++ b/skills/apk-reverse/references/precedents/flutter-plus-shell-case-1.md @@ -0,0 +1,74 @@ +# A pid that keeps moving must be attributed before it is explained + +## Metadata +| Field | Value | +|---|---| +| Context | dynamic analysis of a hardened Android sample (Flutter AOT payload behind a third-party application-stub) on a rooted physical device | +| Cost | three attribution corrections; two of them were written down as findings before being refuted | +| Outcome | the drift is attributed to the sample's own environment check, and the operational rule that falls out of it ("configure root hiding, then re-baseline") | +| Evidence | `docs/tool-verification/EXTENSION-device-run.md` (the pid-drift section and the frida-dexdump section); `docs/tool-verification/EXTENSION-lsposed.md` (the root-hiding layer) | +| Related | `SKILL.md` §Stop conditions (last bullet); `pitfalls.md` P9, P18; `references/detection-and-anti-analysis.md` | + +## Assertions and grade +| # | Assertion | Grade | Evidence | +|---|---|---|---| +| 1 | The sample starts and stays resident with no instrumentation attached | observed | `ps -A \| grep <PKG>` → `u0_a623 22691 … 642488 SyS_epoll_wait 0 S <PKG>`, 12 s after `am start` | +| 2 | During a session its pid changed repeatedly: `22691 → 23452 → 24267 → 27274 → 31284 → 32259 → 1368 → 8161` | observed | the same evidence file's pid-drift transcript | +| 3 | The drift reproduces with `frida-server` **stopped** and no attach | observed | a 90-second observation loop, four pid changes, with the frida process absent | +| 4 | The process dies as the **foreground top activity**, with no crash, no tombstone and no ANR record, and the platform relaunches it on a 7–18 s cycle | observed | `adb logcat \| grep <PKG>` → `Process <PKG> (pid N) has died: fg TOP` followed by `Start proc <pid> for top-activity <PKG>` | +| 5 | No application on the device was hidden from root at the time | observed | `magisk --denylist status` = enforced, `--denylist ls` printed nothing, `/data/adb/shamiko` absent (the root-hiding section of `EXTENSION-lsposed.md`) | +| 6 | The cause is the sample detecting its rooted environment and exiting on purpose | **inferred** — the mechanism fits every observation, but it was not proven by disabling the check | +| 7 | Memory pressure was **not** the cause | observed (refuted) | the drift continued at 300–750 MB free of 11.5 GB, and a reclaim kill leaves an `lmkd` line and a low-memory record — neither present | +| 8 | Instrumentation was **not** the cause | observed (refuted) | the `frida-server`-stopped control run | +| 9 | The device's own hooking framework was already damaged during the window (`lspd` alive but not updating its configuration, `zygote crashed too many times, rolling-back`) | observed | the framework-damage section of `EXTENSION-device-run.md` | +| 10 | A `start timeout` kill line discriminates between these causes | **observed negative** — it reads identically for a slow init, a reclaiming device and a deliberate delay, so it is not evidence either way | the `ActivityManager` transcript quoted in the same evidence file | + +## Execution chain (including the dead ends) +1. The control run looked healthy: start the sample with no instrumentation, wait 12 s, `ps` shows + it resident at 642 MB. +2. Later the pid had changed eight times. The natural reading — *the sample fights instrumentation* + — was written down as the finding. **Dead end 1.** +3. The control that killed it: stop `frida-server` entirely, restart the sample, observe for 90 + seconds. The pid still moved, roughly every 10–20 s. Instrumentation is out. +4. The replacement attribution: memory pressure while a Flutter app saturates the CPU during its + startup window (439 % CPU, `kswapd0` at two thirds of a core were both measured). **Dead end 2.** +5. The measurement that killed it, and the reason it is worth recording: **a different instrument + was pointed at the same question.** `logcat` — not pid sampling — showed + `has died: fg TOP` with no crash record and an immediate relaunch. A reclaiming device does not + do that to a foreground process, and a memory-pressure kill leaves different traces. +6. That left "the sample checks its own environment and quits", and the environment check was then + findable: nothing on the device was hidden from root. A hardened sample that refuses to run + rooted produces exactly this loop, forever. +7. Only after the attribution did the *practical* conclusion exist: re-run the baseline with root + hiding configured, and treat every earlier dynamic result on that device as measured against a + process that was trying to die. + +## Pits +| Pit | Cost | What it was mistaken for | +|---|---|---| +| Sampling the pid instead of reading the platform's lifecycle log | two full attribution rounds | a target-resistance hypothesis, then a host-resource hypothesis — the log settled it in one command | +| A 12-second control window | it looked like a stable baseline | the sample's lifetime was 7–18 s, so a single sample can land inside one life and read as stability | +| A third candidate mechanism discovered mid-pass (a zygote in a crash-and-rollback cycle restarts app processes by itself) | it invalidated single-cause attribution for the whole window | it did not create the drift, but it means "the pid changed" had at least three independent causes on this device at that moment | +| Two refuted attributions already written into an evidence file | deletion cost, not re-measurement cost | each reads as more coherent than the truth, which is why they would have been believed | + +## Reusable pattern +- **Attribute before explaining.** A target that dies, drifts, or restarts gets a *mechanism* from + the platform's own record first (`logcat`, tombstones, `dumpsys`) — sampling tells you that it + happened, never why. +- **Refute the cheapest alternative first, with a control that removes it entirely.** Stopping the + instrumentation is one command; it eliminated the leading hypothesis in a single run. +- **When a measurement disagrees with a recorded conclusion, reopen the conclusion.** That rule + (`SKILL.md` §Stop conditions) was applied twice to this pass's own record, and both times the + record was the thing in the wrong. +- **Record the device's framework state next to any dynamic result.** This device could restart + the target without the target being involved. + +## Write back to the repository +- [ ] `references/detection-and-anti-analysis.md` — a "the target exits on a rooted device, and the + environment is why" subsection: the `fg TOP` + immediate relaunch shape, the two kills that + look like it and are not, and the re-baseline step. +- [ ] `references/precedents/README.md` — already indexed as case 1; update the grade column if a + future pass proves assertion 6 by disabling the check. +- [ ] `references/pitfalls.md` — only if the *sampling-instead-of-logcat* mistake is not already + covered there; if it is, extend that entry with this measured cycle length rather than adding + a second one. diff --git a/skills/apk-reverse/references/precedents/frida-spawn-hangs-zero-events-case-2.md b/skills/apk-reverse/references/precedents/frida-spawn-hangs-zero-events-case-2.md new file mode 100644 index 0000000..4cfb02e --- /dev/null +++ b/skills/apk-reverse/references/precedents/frida-spawn-hangs-zero-events-case-2.md @@ -0,0 +1,76 @@ +# A zero-event trace was manufactured by the harness, not by the target + +## Metadata +| Field | Value | +|---|---| +| Context | instruction-level tracing with Frida Stalker on Android 11 / arm64-v8a, on system processes, to test whether `Stalker.exclude()` fixes the community-reported zero-event and crash behaviour | +| Cost | three arms plus one harness correction; the two defects below both produce a *believable* zero-event trace, so they would have been read as a target finding | +| Outcome | the crash half has a clean cause (`exclude` fixes it); the zero-event half does not, and is now an explicit warning in the script rather than a silent empty log | +| Evidence | `docs/tool-verification/EXTENSION-stalker-exclude.md`; `docs/tool-verification/EXTENSION-native-dbi.md` (device attempts A and B) | +| Related | `references/native-dbi-and-deobfuscation.md` §6; `scripts/stalker_trace.js`, `scripts/stalker_report.py`; `pitfalls.md` P26 | + +## Assertions and grade +| # | Assertion | Grade | Evidence | +|---|---|---|---| +| 1 | A follow with an **empty** exclusion list killed the target process; the same process survived an attach + resume with no follow at all | observed | three-arm run: `baseline alive=True`, `control alive=False err=script has been destroyed`, `treatment alive=True` | +| 2 | Excluding 20 system modules kept the process alive through the follow window | observed | `treatment … DONE\|reason=timeout` with the process still running, `EXCL\|excluded=20/20 […] not-loaded=3` | +| 3 | Exclusion did **not** restore event delivery | observed | `treatment` still reported `blocks=0 blk=0 calls=0 truncated=0` after the full 6 s window | +| 4 | The zero-event warning added to the script fires correctly | observed | `treatment` emitted `WARN\|zero events 1500ms after follow … -- the pipeline is NOT proven`; `control` could not, because its process was already gone | +| 5 | `CONFIG.autoStart` defaults to `true`, so loading the script starts a follow with default options before any runtime configuration can land | observed | the design section of `EXTENSION-stalker-exclude.md`; the harness patches it to `false` and asserts the patch count | +| 6 | `frida` spawns a process **suspended**: without an explicit `device.resume(pid)` the followed thread executes nothing and the run reports zero events | observed | the same design section, and the `DONE … blocks=0` transcript in `EXTENSION-native-dbi.md` | +| 7 | `Process.getMainThreadId()` returned a tid that was **not** the pid on Android (26261 for a pid of 19938), so the script's default thread selection watched the wrong thread | observed | `TRIG following tid=26261 (main-thread follow (pid=19938))` | +| 8 | Re-running with the real main thread `start(19938)` produced the same zero | observed | `DONE reason=timeout blocks=0 blk=0 calls=0` | +| 9 | A hot-function trigger (`malloc` in `libc.so`) produced 201 complete follow cycles with zero blocks in any of them | observed | 609 log lines, 201 follows, 201 `DONE`s, `nonzero_blk = 0` | +| 10 | A per-call follow/unfollow cycle on that hot function crashed a system process | observed once | `systemui` pid `18030 → 26932`, `SIGSEGV` with frame `#00` in an anonymous executable region and the return path into `libart` | +| 11 | The crash is the explanation for the zero-event results | **observed negative** — those runs produced their zeros before any death, on two other processes, and one never touched an export trigger | the separation stated in the same native-DBI evidence file | +| 12 | Why events do not arrive | **unverified** | the measurement says it is not the exclusion list; it does not identify the cause | + +## Execution chain (including the dead ends) +1. The question was framed as a two-claim question, which is why it resolved: does `exclude` stop + the crash, and does it restore events? Three arms, one device, one package, one module, one + follow window, exactly one variable moving between them. +2. The `baseline` arm exists purely to make a death attributable: without it, "following killed it" + and "the app dies under frida anyway" are the same observation. +3. Two harness defects had to be removed **before** the measurement meant anything, and both were + found by noticing that an arm produced the *expected* result for the wrong reason: + - `CONFIG.autoStart: true` followed once at `load()` time with default options, which is neither + the control nor the treatment, and therefore contaminated both. + - spawn leaves the process suspended, so with no explicit `device.resume(pid)` the followed + thread ran nothing at all. +4. Result: the crash half is answered with a cause, and the zero-event half is answered as a + **negative** — exclusion is not its fix, and the community framing that lists "no events" among + the symptoms `exclude` cures is contradicted by this measurement. +5. Independent corroboration that the pipeline itself was healthy: the export-trigger run produced + 201 complete follow cycles. A trigger that never fired would look different; a filter that + dropped everything would look the same, which is exactly why the claim is limited to "events do + not arrive" rather than "Stalker is broken". + +## Pits +| Pit | Cost | What it was mistaken for | +|---|---|---| +| `autoStart` following at `load()` | it invalidated two arms at once | a legitimate treatment result | +| No `device.resume()` after spawn | a whole arm's worth of zero events | the target refusing instrumentation | +| Reading a zero-event log as "the code did not run" | the reason the `WARN\|zero events` line was added to the script this pass | a finding about the target | +| `Process.getMainThreadId()` trusted as "the main thread" | one wasted run | a mis-selected thread | +| A system process died mid-run from the trigger pattern | one attribution round | an unrelated crash that could have been folded into the wrong conclusion | + +## Reusable pattern +- **A negative result needs its own control arm.** `baseline` (attach, resume, never follow) is + what converts "it died" into "the follow killed it". +- **Engineer the harness's own failure modes out before measuring.** Both defects here produced the + exact shape under investigation; a run that reports the expected symptom is not yet a run that + measured anything. +- **Make a dead pipeline announce itself.** A warning line the moment the follow window ends with + zero events turns an ambiguous empty log into a labelled observation — and it is testable, because + the arm whose process died cannot print it. +- **Separate "the follow is unsafe" from "the follow observes nothing".** Two arms, two conclusions, + never one. + +## Write back to the repository +- [ ] `references/native-dbi-and-deobfuscation.md` §6 — keep the two measured boundaries; add the + harness-defect paragraph if absent, because a reader who reproduces a zero event will + otherwise attribute it to the device. +- [ ] `scripts/stalker_trace.js` — already carries the zero-event warning; ensure the `autoStart` + comment says *why* the default is dangerous (it is a trap, not a convenience). +- [ ] `references/precedents/README.md` — indexed as case 2; move assertion 12 to `observed` only if + a future pass identifies the cause. diff --git a/skills/apk-reverse/references/precedents/l1-equal-length-patch-case-4.md b/skills/apk-reverse/references/precedents/l1-equal-length-patch-case-4.md new file mode 100644 index 0000000..d591a4f --- /dev/null +++ b/skills/apk-reverse/references/precedents/l1-equal-length-patch-case-4.md @@ -0,0 +1,80 @@ +# Four bytes of dex, and a control build that still failed the old way + +## Metadata +| Field | Value | +|---|---| +| Context | an equal-length dex patch against a public crackme (`UnCrackable-Level1`), taken all the way to an installed, launched build whose blocking dialog is gone | +| Cost | one full pipeline re-run after the repository changed mid-pass; one false verification caused by a reused remote filename | +| Outcome | the end-to-end chain is measured, with a zero-change control through the **same** pipeline — which is what makes the four bytes, rather than the pipeline, responsible | +| Evidence | `docs/tool-verification/EXTENSION-benchmark-l1-l3.md` (the B1 sections); `tests/benchmark.md` row B1 | +| Related | `references/byte-level-patching.md`, `references/patch-audit.md`, `references/repack-and-sign.md`, `references/verification.md` (the control-build rule) | + +## Assertions and grade +| # | Assertion | Grade | Evidence | +|---|---|---|---| +| 1 | The shipped dex's `signature` field is stale while its `checksum` is correct | observed | `checksum field 7fb7d9fa` / `adler32(d[12:]) 7fb7d9fa` matches, while `sha1(d[32:])` does **not** match the stored signature | +| 2 | The edit site is the first arm of a three-way root `OR`, and the *fall-through* is the "continue" side | observed | `0x9d0 39000e00 if-nez v0 -> 0x9ec` preceded by `invoke-static …a()Z` / `move-result`, and `0x9ec` is `const-string "Root detected!"` | +| 3 | Branch polarity was pinned by the following instruction, not by the method name | observed | `expect_next` required `invoke-static …b()` (the *next* root check) after the branch — `polarity: expect_next satisfied` | +| 4 | The patch owns exactly 4 bytes and the header fields own 24; nothing else in the file moved | observed | whole-file diff offsets `['0x8','0xa','0xb','0xc'…'0x1f', '0x9d0','0x9d2']`, and both header fields recomputed from the written file | +| 5 | The signature was computed **before** the checksum | observed | `zlib.adler32(d[12:])` and `hashlib.sha1(d[32:])` recomputed on the output reproduce both stored fields, which is only possible in that order | +| 6 | `repack.py` reported `resources.arsc STORED and 4-byte aligned (OK)` for control and patched alike | observed | the alignment gate output, identical for both builds | +| 7 | Both builds signed with v1+v2+v3 and the same certificate digest | observed | `apksigner verify --print-certs --verbose --min-sdk-version 21` | +| 8 | The installed artifact is the built artifact | observed | device `base.apk` sha256 equals the local sha256, for control (`e5a9f335…`) and patched (`ad6a51ce…`) | +| 9 | The control build — same pipeline, byte-identical dex — **still shows** the blocking dialog | observed | four window-focus samples, 2 s apart: `Window{… Root detected!}` for original and control, `Window{… <PKG>/…MainActivity}` for patched | +| 10 | The patched build reaches its own UI without a dialog | observed | the screenshots: control/greyed-out input with the modal over it, patched with the input live, caret present, `VERIFY` enabled | +| 11 | A second, independently built patched APK from the one-command pipeline produced the **same** sha256 as the manual route | observed | `ad6a51cef4699b73ff052d4e049ca9efd64e9570900d19509f0b7a7c20f87e56` | +| 12 | The app's functional check compares the user's input against a decrypted plaintext, and the 16-byte AES key is a `const-string` in the dex | observed | `a.b` hook output: key `8d12…73cc`, data decrypts to `I want to believe`, `[a.a] REAL input="" -> false` | +| 13 | The input field cannot be driven by `adb shell input text` on this ROM | observed | 7 of 17 characters survived and every space was dropped, across three independent quoting routes; `uiautomator dump` after each attempt is what distinguished "the app said Nope" from "the input never arrived" | +| 14 | The success branch was reached by forcing the boolean, not by typing the secret | observed, and labelled as such in the evidence file | `evidence/09_success_branch/*.png` shows the app's own `Success! …` dialog | + +## Execution chain (including the dead ends) +1. **Read the branch structure before editing.** The edit target was chosen because its fall-through + continues into the next root check — not because of its name. That is the difference between a + patch that removes one arm of an `OR` and a patch that removes the check. +2. Dry run first: the tool prints the site with the instructions before and after, and the polarity + evidence. Nothing was written until `polarity: expect_next satisfied`. +3. Applied, then proved with a byte-level diff of the whole file rather than the tool's own report: + four payload bytes and the two header fields, and nothing else. +4. Repack. **Dead end:** the documented default signing route failed with + `error: signer jar not found: uber-apk-signer.jar`, and its own hint — "any zipalign+apksigner + based signer works here" — pointed at a code path that did not exist yet. The working route was + the supported `--no-sign` seam plus explicit `zipalign` → `apksigner`, and the repository changed + mid-pass (another writer landed an `apksigner` route), so the pass recorded **both** states and + re-ran the one-command form, which produced byte-identical output. The lesson kept with it: + *a "this tool cannot do X" finding on a shared worktree has an expiry date.* +5. Install through the **root** path (`pm install -r -d`), because this ROM intercepts the streaming + install (see case 5's sibling record for the `[-99]` shape). +6. Verify, in the order that matters: hash the installed `base.apk` against the local build; `am + start -W`; then window focus sampled four times; then **look at the screenshots**. +7. Build a zero-change control through the identical pipeline and repeat steps 5–6. It still fails + the old way, which is what closes the attribution. + +## Pits +| Pit | Cost | What it was mistaken for | +|---|---|---| +| Scanning the method for a plausible name instead of reading the branch structure | would have patched an `OR` arm that changes nothing | a successful patch | +| Relying on the tool's own "self-verify: checksum_ok=True signature_ok=True" without an independent diff | the stale-signature starting state could have hidden a wrong header order | a correct header | +| Reusing the `--tag` value as the remote filename across runs | the final verification showed the **control** build's dialog, from the control run's leftover remote path | a patch that did not work | +| `adb shell input text` on a field that silently drops characters | two attempts read as "wrong secret" | a wrong secret | +| Trusting a screenshot alone | it cannot tell which build is installed | a verified patch | + +## Reusable pattern +- **Equal-length byte patches are the cheap route; choose the site by branch semantics.** Nothing + moves, so no offset, `try` block or debug pointer can be invalidated, and the diff is auditable + down to the byte. +- **A patch needs a same-pipeline control that fails the old way.** "It installs and runs" and "the + behaviour changed" are different claims, and only the second one is the deliverable. +- **Hash the artifact on the device, at the moment you observe it.** This is the guard that caught + the reused-tag mistake, and it is one command. +- **Look at the screen, and look at more than one sample.** A modal that appears and is gone between + samples is invisible to a single capture. +- **Re-derive the header yourself once.** Recomputing both fields from the written bytes is what + proves the *order*, and the tool's own success message cannot. + +## Write back to the repository +- [ ] `references/repack-and-sign.md` — if the `signer jar not found` message and its apksigner + route are not already reconciled, that is a documented contradiction worth closing. +- [ ] `references/verification.md` — the control-build rule is the load-bearing part of this case; + add the "hash the installed APK before observing" step if it is only implicit there. +- [ ] `references/precedents/README.md` — indexed as case 4; the row's grade stays `observed` end to + end. diff --git a/skills/apk-reverse/references/precedents/logd-broken-module-never-ran-case-3.md b/skills/apk-reverse/references/precedents/logd-broken-module-never-ran-case-3.md new file mode 100644 index 0000000..a586997 --- /dev/null +++ b/skills/apk-reverse/references/precedents/logd-broken-module-never-ran-case-3.md @@ -0,0 +1,70 @@ +# "The module never ran" was an artefact of a broken log channel + +## Metadata +| Field | Value | +|---|---| +| Context | verifying that an LSPosed hook module was actually injected into a hardened target, on a rooted Android 11 device | +| Cost | the verification criterion itself had to be corrected; `logcat` was empty across nine successful injections | +| Outcome | the injection is proven from `/data/adb/lspd/log/modules_<timestamp>.log`, and the read rule is now "both channels, always" | +| Evidence | `docs/tool-verification/EXTENSION-lsposed.md` (the injection-line, log-surfaces and scope sections) | +| Related | `docs/tool-verification/EXTENSION-device-run.md` (the framework-damage section); `references/lsposed-and-modules.md`; `references/environment.md` | + +## Assertions and grade +| # | Assertion | Grade | Evidence | +|---|---|---|---| +| 1 | The module was injected nine consecutive times, each run reaching the target's `MainActivity.onCreate`, the WebView process and an ad SDK activity | observed | nine full injection blocks; nine pids in ~3 minutes (`8111 → 9392 → 10557 → 11453 → 12345 → 13236 → 14155 → 15061 → 15934`) | +| 2 | `logcat -s <TAG>` was **empty**, as was `logcat -s LSPosed-Bridge` and a full-buffer `grep -i <TAG>` | observed | the injection-line section of the LSPosed evidence file, stated as the correction to the pass's own earlier draft | +| 3 | The module's output existed only in `/data/adb/lspd/log/modules_<timestamp>.log` | observed | the evidence file for the pass is 201 lines, of which 126 are `<TAG>` | +| 4 | The same file earlier appeared to contain nothing but `Logd maybe crashed (err=Socket operation on non-socket), retrying in 1s...` | observed | the log-surfaces section of the same evidence file quotes the entire content of an older `modules_*.log` | +| 5 | The ROM's `logd` route is broken, so LSPosed's logcat channel delivers nothing while its file channel works normally | **inferred** — the two observations above are consistent with exactly this, and the mechanism was not probed further | +| 6 | A reader debugging a module with `logcat` alone would conclude the module never ran, while looking at nine successful injections | observed (as a statement about the evidence), inferred (as a statement about other readers) | +| 7 | The target's double-`Application` packer swap is visible in the hook order, and hooks installed at `handleLoadPackage` time survive it | observed | `Application.attach` fires twice — the real `BaseApplication`, then the stub — and by `Activity.onCreate` the reported classes are the **unpacked** ones | +| 8 | Scope is per app, not per process | observed | the module also injected into the target's `com.google.android.webview` process and reported an ad SDK activity | +| 9 | The pre-reboot self-restart cycle (10–20 s) can be compared with the post-reboot cycle (22–23 s) | **inferred at best, and not a controlled result** — a reboot moves several variables at once | the injection-line section of the same evidence file | +| 10 | `verbose_*.log` is a full logcat snapshot, useful for correlating a run with system state and useless as a module log | observed | the log-surfaces section of the same evidence file | + +## Execution chain (including the dead ends) +1. The module was built, installed, PM-enabled, and scoped through LSPosed Manager's UI — read from + `uiautomator dump` coordinates, never tapped by guess. The scope change was confirmed in the + framework's own SQLite database (`scope` gained `(2, '<PKG>', 0)`, `modules.enabled` flipped to + `1`, and the database file grew from 156,376 to 201,696 bytes), which is what proves the + framework — not merely its daemon process — came back. +2. The pass then read `logcat -s <TAG>` and found nothing. **Dead end:** the criterion in the + reference file, in the pass's own earlier draft, and in the scaffold's generated README all said + to watch `logcat`. +3. Rather than conclude "injection failed", the other channel was read: LSPosed's file log. Nine + full injection blocks. +4. The correct criterion was written down, and it is the transferable part: read **both** channels. + `logcat -s <TAG>` where the platform is healthy; `/data/adb/lspd/log/modules_*.log` always. +5. The reason the two disagree on this device is a broken `logd` (`Socket operation on non-socket`), + which is also visible in unrelated incident lines from the same window (`system server died`, + `am is dead`, `no response from bridge, retry in 1s`, `Magisk: zygote crashed too many times, + rolling-back`). + +## Pits +| Pit | Cost | What it was mistaken for | +|---|---|---| +| Trusting one log channel on a ROM whose log daemon is damaged | it would have produced a false negative on a working feature | "the module never loaded" | +| `logcat` silence read as absence of evidence *for* the target's behaviour | the verification criterion itself was wrong, not just the reading | a statement about the module | +| The same silent-failure shape appears twice in the pass | once for the module channel, once for the framework's own state | two different "not installed"-looking states, one of which was real | + +## Reusable pattern +- **Before accepting a silence as a finding, ask whether the instrument could have spoken.** This is + `pitfalls.md` P26's rule pointed at a *log channel* rather than at a script: check that the channel + has ever delivered for a healthy case. +- **Prefer a channel that is a file over a channel that is a daemon.** `logcat` depends on `logd`; + a file in the module's own directory depends on the module. +- **Verify the verifier.** The scope change was confirmed in the framework's own database rather + than in the manager's UI reflecting it back — a UI can render a cached state, and the file is the + state. +- **On a packed target, a Java-layer hook is reachable without touching the APK**: the double + `Application` swap is observable, and hooks installed before it survive it. That is a *capability* + result, separate from the log-channel lesson, and it is the reason this route is worth the setup. + +## Write back to the repository +- [ ] `references/lsposed-and-modules.md` — the verification section must name both channels and the + exact file path pattern; a single-channel instruction is what this case disproves. +- [ ] `scripts/lsposed_scaffold.py` — the generated README's verification line should carry the same + two-channel instruction, because it is the artifact a user reads first. +- [ ] `references/precedents/README.md` — indexed as case 3; assertion 5 stays `inferred` until the + logd mechanism is probed directly. diff --git a/skills/apk-reverse/references/precedents/manual-grep-finds-what-nobody-grepped-case-5.md b/skills/apk-reverse/references/precedents/manual-grep-finds-what-nobody-grepped-case-5.md new file mode 100644 index 0000000..1fc1f58 --- /dev/null +++ b/skills/apk-reverse/references/precedents/manual-grep-finds-what-nobody-grepped-case-5.md @@ -0,0 +1,81 @@ +# The leak nobody grepped for, found by hand and only after the fact + +## Metadata +| Field | Value | +|---|---| +| Context | this repository's own history: a complete verification pass was recorded, reviewed and merged, and the target identity in its evidence files was found afterwards | +| Cost | a manual sweep, and the omission survived at least one pass that produced evidence files | +| Outcome | a desensitization convention (`<PKG>` / `<DEVICE>` placeholders, target identity absent by policy) **and** an automated scan (`scripts/scan_leaks.py`), added in the same pass that recorded this case | +| Evidence | `docs/tool-verification/EXTENSION-desensitization.md`; `docs/tool-verification/EXTENSION-lsposed.md` (its identifier-convention and injection-line sections); `docs/tool-verification/EXTENSION-device-run.md` (its environment section) | +| Related | `references/desensitization-and-leak-scans.md`; `references/long-task-discipline.md` §Keep a live record, not a log; `references/verification.md` §Reporting | + +## Assertions and grade +| # | Assertion | Grade | Evidence | +|---|---|---|---| +| 1 | Identity was found by a human grep after the material had already been written, and the files had to be corrected | observed as this repository's stated history; the correction itself is visible in the convention every evidence file now opens with | the environment section of `EXTENSION-device-run.md`: *"Target identity is deliberately absent. The sample is referred to as the sample, and its package identifier appears as `<PKG>`"* | +| 2 | The convention was enforced per file, by hand, at write time | observed | the identifier-convention section of `EXTENSION-lsposed.md` normalises every occurrence of the package identifier and its sub-package prefixes, and states that the transcripts are otherwise verbatim | +| 3 | Verbatim transcripts were kept, and that is deliberate | observed | the same section keeps the hook-reported class names, "those are what make the evidence checkable" | +| 4 | No structural gate could have caught it | observed | `check_repo.py` validates layout/frontmatter/`--help`; `check_refs.py` validates section anchors; neither reads content for identity. Both passed on the material that had to be corrected | +| 5 | An automated scan would have caught it | **inferred** — the scanner was built and measured against a planted corpus, but it was never run against the historical, pre-correction tree | `docs/tool-verification/EXTENSION-desensitization.md`, the planted-corpus and repository-scan sections | +| 6 | The scan finds the four classes that actually occur here — bundle ids in `pm`/`ps`/`manifest` contexts, 16-character serial-shaped tokens, inline key assignments, host user paths | observed | planted corpus: 30 findings across 6 categories, every rule firing | +| 7 | The scan's first run against the real tree was overwhelmingly false positives, and each one was a shape that legitimately appears in this documentation | observed | the repository-scan section of `docs/tool-verification/EXTENSION-desensitization.md`: 40 findings when the file above was still being written, of which 26 were `strong`/`certain` — and every one of those 26 was inside that pass's own evidence file. After the correction: 0 strong, 4 weak (RFC 5737 usage examples) across 129 files | +| 8 | Over-redaction is the failure on the other side, and it is not hypothetical here | observed | the do-not-anonymize list in the reference file is derived from identifiers that a shape-only scanner flagged in this tree: tool names, SDK packages, dex constant identifiers, public crackme names | +| 9 | The ratio, not the count, decides whether a gate survives | observed from this case | the pass that produced 27-of-28 false positives also had to fix 26 real ones in its own report on the same day. Both numbers moved the design: exemptions for the shapes that legitimately appear, and a split exit code so the remaining documentation-range hits do not make the gate permanently red | + +## Execution chain (including the dead ends) +1. The convention existed and was written down per evidence file: identity absent, placeholders for + the package, the device serial and the host paths, verbatim transcripts otherwise. +2. It was enforced by hand, at write time, by whoever wrote the file. **This is the dead end**: a + convention enforced by attention has no failure signal. The omission was invisible until someone + went looking, and by then the material was already in the tree. +3. The correction happened after the fact, which is why this case exists at all. A search for the + shape — not for the value — is the only route that scales, because the value is exactly what you + do not yet know. +4. The scan was then built, and the interesting result was **not** "it finds leaks" (that was the + easy part). It was that a first run over the real tree produced 28 findings of which 27 were + legitimate documentation: a JDK version that looks like an IPv4 literal, tombstone hex tokens + that look like serials, a public crackme's package name, a `probe.synthetic.*` fixture, a + property lookup. A scanner that shipped in that state would have been ignored within a week, and + the leak it exists to catch would have gone with it. +5. The final state separates the two decisions: the scanner reports shapes and admits known-benign + explanations **with a printed reason** (`--show-exempt`), and a human makes the identity call + from the context window the report already contains. + +## Pits +| Pit | Cost | What it was mistaken for | +|---|---|---| +| A convention with no automated check | one manual sweep after the fact | adequate coverage | +| Searching for the *shape* after the fact, rather than during the write | it only works if you already suspect | a finished pass | +| A shape-only scanner | 27 false positives in 28 findings on this tree alone | a working gate | +| Suppressing the false positives by blocking whole address ranges | it would also have hidden the one real weak hit (`scripts/tls_check.py`'s RFC 5737 example) | noise reduction | + +## Reusable pattern +- **Desensitize at write time, and verify by shape, not by value.** The value you have to remove is + the one you cannot enumerate in advance; the shape is the thing you can write a rule for. +- **Decide in code what is exempt, and let the code print why.** An exemption list that cannot be + audited is indistinguishable from an exemption list that is too broad, and only one of those is + visible in review. +- **Keep the transcripts verbatim and replace the identity with a placeholder.** Redacting the + *evidence* destroys the thing that makes the record checkable; redacting the *identifier* costs + nothing. +- **Measure the scanner against your own tree before trusting it.** The false-positive ratio, not + the true-positive count, decides whether anyone will keep running it. + +## Write back to the repository +- [ ] `references/desensitization-and-leak-scans.md` — the reasoning, the split rule, and the + failure-mode table; this case is its motivating evidence. +- [ ] `docs/tool-verification/README.md` — the index entry for `EXTENSION-desensitization.md`. +- [ ] `references/precedents/README.md` — indexed as case 5. +- [ ] **Closed in this pass**: `scripts/scan_leaks.py` is wired into the repository workflow (run it + over the tree before a commit) and the `RESULT=` convention is documented in + `references/long-task-discipline.md` §Every script answers with a token, not with prose. +- [ ] **The scanner caught its own first pass.** The tool written to prevent this class of leak found + 26 `strong`/`certain` hits on the day it was written, all of them inside its own evidence file — + the fixture's literal match values, quoted to prove the rules fire. The fix was to keep the + location, rule, category and grade for all 30 findings and replace every literal with a shape + description. Boundary that makes it a rule rather than a scare: **the corpus may contain values; + the published surface may not.** Evidence and transcripts: the "self-leak this file carries" and + "inherited versus post-fix" sections of `docs/tool-verification/EXTENSION-desensitization.md`. + This is the strongest available argument for the gate — the author of the rule needed it on the + same day, and did not see it by hand. + diff --git a/skills/apk-reverse/references/toolchain.md b/skills/apk-reverse/references/toolchain.md index 616c460..e14432b 100644 --- a/skills/apk-reverse/references/toolchain.md +++ b/skills/apk-reverse/references/toolchain.md @@ -204,6 +204,7 @@ not that the method is empty. | `radare2` / `rizin` | `r2 -q -c '<cmds>' file` | scriptable disassembly/analysis; the practical choice when no GUI is available | | `Ghidra` (headless) | `analyzeHeadless <proj> <name> -import <file> -postScript <s>` | decompiler without a GUI; slower to script but far better output than a raw disassembler | | `capstone` (python) | library | decoding in your own scripts — see the silent-stop trap below | +| **`svc_scan.py`** | `python scripts/svc_scan.py libfoo.so [--context 3]` | **names the syscall behind an inline `svc`** and reports which PT_LOAD segment each hit is in. Use it *before* planning a libc-level hook: a module with its own `exit`/`kill` `svc` sites is not observable through libc, and one without them means a missing event is a finding about your hook. Read the neighbours (`--context`) — a byte scan for `svc` also matches inside data. Measured on the test device: device `libc.so` = 4 termination sites (its own exports), a shell library's 21 hits = all data | | `keystone` (python) | library | assembling a short patch when you are editing bytes by hand | | `pyelftools` (python) | library | **section-based — unreliable on hardened targets.** Prefer hand-walking `PT_LOAD`/`PT_DYNAMIC`, as `scripts/elf_plt.py` does | | IDA Pro | GUI, **or headless via `idalib-mcp`** | the strongest decompiler output. Do not assume it needs a human in front of it — a headless MCP server exists (see *MCP tool servers* below). Ask for a human only when that setup is genuinely unavailable | @@ -219,6 +220,15 @@ completeness matters (`native-tamper-and-suicide.md` §Scanner traps). 4-byte window into *some* instruction, so a wrong start offset yields plausible-looking garbage. Bound your window with a known entry point or a known call site. +**A decoder is a decoder, not an oracle.** Two independent decoders agreeing is the signal worth +having (measured: a hand-written word scan for the `svc` encoding and the `capstone`-based scan in +`svc_scan.py` returned the identical 214-site set on a device `linker64`, set difference empty); one +decoder's disagreement with itself is not. **`dexdump`** (from build-tools, e.g. +`E:\tools\android-14\dexdump.exe`) is the cheap independent reader for the dex half of this: use it +to count a dump's class and method records rather than trusting a self-written parser, the same way +`advanced-unpacking.md` §Establish "the bodies do not decode" with a decoder you did not write +requires. + ### Ghidra — the decompiler for a target whose decisive layer is aarch64 The recurring shape: the app is fine, the dex is readable, and the layer that actually decides @@ -293,6 +303,32 @@ Rules that follow: between "install a decompiler" and "the decompiler cannot help here", and they look identical from the outside. +## Where to get a tool we do not ship — a sourced gap list, not a link dump + +The list below exists because "install the tool" needs a *source*, and searching for one on a target's +clock is how a ten-minute install becomes an hour. **Two grades are used here and they are different +claims:** `URL verified` means the repository was resolved on the date given; `tool unverified` means +nobody has run it here. Most rows are `URL verified / tool unverified` — treat them as leads with a +date on them, not as recommended tools, and check the row's own prerequisites when you install it +(§Closing a capability gap — installing the tool IS the task). Rows marked `measured` have been used here. + +| Gap | Source | Grade / what is actually known | +|---|---|---| +| Decompiler for dex/Java | `https://github.com/skylot/jadx` | URL verified 2026-09-21; tool unverified here (not installed). The kit's route is `droidasc`/`ddc` for queries and jadx only to *read* a class already located | +| Repacking beyond `scripts/repack.py` | `https://github.com/iBotPeaches/Apktool` | URL verified 2026-09-21; tool unverified. Not installed on this host, and the `apktool.bat` in `E:\tools\bin` is a dead link — do not trust a `.bat` to mean the tool works | +| smali round-trip outside the bundled dexlib2 kit | `https://github.com/JesusFreke/smali` | URL verified 2026-09-21, **last upstream push 2024-01-17** — maintenance status is a fact to check before adopting it for a new dex version | +| Anti-detection on the frida side (patched server plus a script surface aimed at RASP) | `https://github.com/CrackerCat/strongR-frida-android` | URL verified 2026-09-21; tool unverified. A patched `frida-server` is an **environment** change: record it in the task record, because results obtained under it are not comparable to a run without it | +| A dex dumper that does not use `ptrace` | `https://github.com/index-login/MobileRE-Skill` (`.kilo/skill/rev-dex-dumper/`) — `panda-dex-dumper`, `mem-dex-dumper` + C source | **Partly measured**: both ELF images were inspected here and neither imports a `ptrace` symbol (`panda` = aarch64 `ET_DYN`, 48 symbols; `mem` = aarch64 static, stripped), so the *claim* is consistent at the symbol layer; the **dump behaviour was not run here**. Our own `/proc/<pid>/mem` read is `advanced-unpacking.md`'s root-side route and is measured | +| eBPF-based dex extraction | `https://github.com/LLeavesG/eBPFDexDumper` | URL verified 2026-09-21; tool unverified, **and unusable on `<DEVICE>` anyway** — kernel 4.14 against a 5.10+ requirement (`kernel-and-environment-hardening.md`). Listed so the next reader does not re-derive the version gate from scratch | +| Decompilation/deobfuscation/unpacking without a JVM | `https://github.com/adam-040/Enigma`, `https://github.com/1-3-7/disrobe` | URLs verified 2026-09-21; tools unverified. Both are interesting precisely because they remove the JVM/IDE dependency a Ghidra or IDA route carries — evaluate prerequisites before adopting | +| Java2C generation to *build* a fixture | `https://github.com/amimo/dcc` | **Partly measured here**: `dcc` itself was run in a previous pass, but the C compile needs an NDK this host does not have, so no Java2C artifact was ever produced (`java2c-and-jni-sinking.md`) | +| Building an LSPosed module without gradle, or comparing against a maintained template | `https://github.com/Jordan231111/lsposed-universal-template` | URL verified 2026-09-21; tool unverified. Its layout is gradle-based, which this host cannot build — useful as a **manifest/scope reference** only. (`mabbcoll13/xposed-module-kit` no longer resolves: HTTP 404 on 2026-09-21, which is why this row's date is worth writing down) | +| Dart AOT snapshot front end | see the §Closing a capability gap table above | measured here: `aotopsy` runs with no toolchain; `blutter` measured ≈78 s end to end on this host | + +Two habits that keep this list from rotting: **write the access date next to a URL** (a dead link with +no date reads as a current recommendation — one row above is already 404), and **promote a row only +when someone runs the tool**, so `URL verified` never silently becomes "we use this". + ## MCP tool servers — an external dependency, not a tool on the shelf An MCP server is neither a CLI tool nor one of the scripts here. It is a **separate installation plus diff --git a/skills/apk-reverse/scripts/anti_detect_probe.js b/skills/apk-reverse/scripts/anti_detect_probe.js new file mode 100644 index 0000000..7a50c64 --- /dev/null +++ b/skills/apk-reverse/scripts/anti_detect_probe.js @@ -0,0 +1,353 @@ +/* + * anti_detect_probe.js -- an OBSERVER. It changes nothing. + * + * Purpose: answer "if a detector is running, what is it looking at?" without + * answering it with a patch. That division is deliberate and is the point of this + * file: an observation module must not also intervene (no exit_blocking, no + * Interceptor.replace of a detection function, no NOPing). A probe that patches + * while it watches produces a process whose behaviour you can no longer use as + * evidence about the original -- the death you prevented is exactly the signal you + * were there to date. + * + * What it reports, as one `RESULT=` block: + * - the mapped libraries, so a detection .so can be named rather than guessed + * - every open/openat/fopen/access/stat on a path, with the *caller module+offset* + * for the ones that look like environment probing (/proc, /sys, magisk, su, + * frida, xposed). The caller offset is the deliverable: it is what turns + * "something checks root" into "0x1cef8 in libX.so checks root". + * - kills: kill/tgkill/exit/exit_group, with caller, so a self-destruct is + * attributed to a module instead of to the platform + * - thread creation (pthread_create/clone) with the real entry point, because a + * polling detector thread is a different shape from an in-line check + * - dlopen/dlsym, so a runtime-resolved symbol is visible + * - the process's own view of "am I instrumented": TracerPid, frida-named maps, + * frida-named threads, and TCP listeners (a detector often reads these first) + * + * Load it with spawn (-f) to see the early phase, attach (-p) to see the steady + * state. On this repository's test ROM attach-by-name fails, so use --pid. + * + * Usage (from the kit's own driver): + * python scripts/run_probe.py scripts/anti_detect_probe.js <pid> --via usb --log probe.log + * Usage (raw frida, explicit pid because -n does not resolve on this ROM): + * frida -H 127.0.0.1:27099 -p <pid> -l scripts/anti_detect_probe.js + * + * Config comes from CONFIG below or from RPC: + * script.exports.config({ watchMs: 8000, pathFilter: '/proc' }) + */ + +var CONFIG = { + // How long to keep collecting before printing the RESULT block. 0 = print only + // on the stop() RPC, for an interactive session. + watchMs: 12000, + // Stream a heartbeat every N ms. This exists because of a measured failure mode: + // a detector that kills the process inside the watch window means a probe that + // only reports at the end reports nothing at all -- the one run that mattered + // produced no data. The heartbeat says "the script was still alive at T", which is + // what separates "the target died" from "my script never armed". + heartbeatMs: 1000, + // Stream a line the moment a category fires for the first time, so the last + // events before a death are on the wire already. + streamFirstHit: true, + // Substrings that mark a path access as environment probing. Everything else is + // still counted, but not individually reported (it would drown the interesting + // lines -- a busy app opens thousands of files). + pathFilter: ['/proc', '/sys', 'magisk', 'supersu', 'frida', 'xposed', 'gum-js', + '/data/local/tmp', 'busybox', 're.frida', 'linjector'], + // Report process-wide events for every thread (false) or only the main thread. + allThreads: true, + // Hard cap on reported lines per category, so a chatty target cannot flood the log. + maxLinesPerCategory: 60, +}; + +if (typeof CONFIG_OVERRIDE !== 'undefined' && CONFIG_OVERRIDE.anti_detect_probe) { + Object.assign(CONFIG, CONFIG_OVERRIDE.anti_detect_probe); +} + +var counts = {}; +var lines = {}; +var started = Date.now(); + +function bump(cat, msg) { + counts[cat] = (counts[cat] || 0) + 1; + if (!lines[cat]) lines[cat] = []; + if (lines[cat].length < CONFIG.maxLinesPerCategory) lines[cat].push(msg); + if (CONFIG.streamFirstHit && counts[cat] === 1) { + // On the wire immediately: if the target dies a moment later, this is the only + // trace of the event that existed. + send({ type: 'LIVE', cat: cat, msg: msg, t: Date.now() - started }); + } +} + +function backtraceOwners(ctx) { + // Module+offset for every return address inside a mapped module. The offset is + // what a static tool needs; the module name is what tells a shell library from + // libc. The link register is read first because on arm64 it is where the caller + // of the hooked export lives, and it survives even when Thread.backtrace cannot + // unwind (Frida's unwinder needs the target's frames to be intact, which is not + // guaranteed in a constructor or a signal path). + try { + var owners = []; + var lr = null; + try { lr = ctx.lr; } catch (e) { lr = null; } + if (lr) { owners.push(describe(lr, 'lr')); } + var bt = Thread.backtrace(ctx, Backtracer.ACCURATE); + for (var i = 0; i < bt.length && owners.length < 5; i++) { + owners.push(describe(bt[i], 'bt' + i)); + } + return owners.length ? owners : ['no-unwind']; + } catch (e) { + return ['bt-failed:' + e.message]; + } +} + +function describe(addr, tag) { + try { + var d = DebugSymbol.fromAddress(addr); + if (d && d.moduleName) { + var base = Module.findBaseAddress(d.moduleName); + var off = base ? addr.sub(base) : null; + return d.moduleName + (off ? '+0x' + off.toString(16) : '+?') + '(' + tag + ')'; + } + if (d && d.name) return d.name + '(' + tag + ')'; + return 'anon:' + addr + '(' + tag + ')'; + } catch (e) { + return 'addr?' + addr; + } +} + +function ownerString(ctx) { + // Takes the context explicitly. `this` inside a Frida Interceptor callback is not + // reliably set when the callback is an anonymous function expression, and the + // measured symptom of getting this wrong is a log full of + // "cannot read property 'context' of undefined" -- which reads like a hooking + // failure and is actually a JavaScript binding mistake. Callers pass `this.context`. + if (!ctx) return 'no-ctx'; + return backtraceOwners(ctx).join(' | '); +} + +function looksInteresting(p) { + for (var i = 0; i < CONFIG.pathFilter.length; i++) { + if (p.indexOf(CONFIG.pathFilter[i]) >= 0) return true; + } + return false; +} + +function hookExport(modName, exp, cat, argIndex) { + var addr = Module.findExportByName(modName, exp); + if (addr === null) return false; + try { + Interceptor.attach(addr, { + onEnter: function (args) { + try { + var p = null; + if (argIndex >= 0) p = args[argIndex].readUtf8String(); + var who = ownerString(this.context); + if (p === null) { bump(cat, exp + '() <- ' + who); return; } + if (looksInteresting(p)) { + bump(cat, exp + '("' + p + '") <- ' + who); + } else { + counts[cat + '_quiet'] = (counts[cat + '_quiet'] || 0) + 1; + } + } catch (e) { /* unreadable pointer: count it, never throw out of a hook */ } + } + }); + return true; + } catch (e) { + bump('hook_errors', modName + '!' + exp + ': ' + e.message); + return false; + } +} + +function hookTerminators() { + // A detector's self-destruct is often a direct libc call; the caller offset is + // what identifies it. Note: a module that issues `svc #0` itself will NOT show up + // here -- the absence of a report plus a death is itself the signal that a libc + // hook cannot see it (scripts/svc_scan.py decides that statically). + ['exit', 'exit_group', '_exit', 'abort', 'kill', 'tgkill', 'raise', 'pthread_kill'] + .forEach(function (fn) { + var a = Module.findExportByName('libc.so', fn); + if (a === null) return; + try { + Interceptor.attach(a, { + onEnter: function (args) { + bump('termination', fn + '(' + args.join(',') + ') <- ' + + ownerString(this.context)); + } + }); + } catch (e) { bump('hook_errors', fn + ': ' + e.message); } + }); +} + +function hookThreadCreation() { + var pc = Module.findExportByName('libc.so', 'pthread_create'); + if (pc !== null) { + try { + Interceptor.attach(pc, { + onEnter: function (args) { + bump('thread_create', 'pthread_create entry=' + args[2] + ' <- ' + + ownerString(this.context)); + } + }); + } catch (e) { bump('hook_errors', 'pthread_create: ' + e.message); } + } + // clone() is how a detector bypasses the pthread wrapper; args[3] points at the + // argument block whose fifth word holds the real entry point on arm64. + var cl = Module.findExportByName('libc.so', 'clone'); + if (cl !== null) { + try { + Interceptor.attach(cl, { + onEnter: function (args) { + var extra = ''; + try { extra = ' realEntry=' + args[3].add(96).readPointer(); } catch (e) {} + bump('thread_create', 'clone()' + extra + ' <- ' + ownerString(this.context)); + } + }); + } catch (e) { bump('hook_errors', 'clone: ' + e.message); } + } +} + +function hookLoader() { + ['dlopen', 'android_dlopen_ext', 'dlsym'].forEach(function (fn) { + var a = Module.findExportByName('libc.so', fn); + if (a === null) return; + try { + Interceptor.attach(a, { + onEnter: function (args) { + var s = null; + try { s = args[0].readUtf8String(); } catch (e) {} + bump('loader', fn + '("' + s + '") <- ' + ownerString(this.context)); + } + }); + } catch (e) { bump('hook_errors', fn + ': ' + e.message); } + }); +} + +function selfView() { + // What the process can see about itself right now. This is evidence about the + // *environment*, not about the detector: it says what is available to be found, + // which is the other half of attributing a death. + var out = {}; + var files = ['/proc/self/status', '/proc/self/maps', '/proc/self/task', + '/proc/net/tcp', '/proc/net/tcp6', '/proc/self/cmdline']; + out.files = {}; + files.forEach(function (f) { + try { + var content = File.readAllText(f); + if (f === '/proc/self/status') { + var m = content.match(/TracerPid:\s*(\d+)/); + out.tracerpid = m ? m[1] : 'absent'; + // A detector may match any of these names; report which are present so a + // "frida is visible here" claim is not made from memory. + out.status_flags = (content.match(/^(Name|State|Seccomp):.*$/gm) || []).join(' ; '); + } else if (f === '/proc/net/tcp' || f === '/proc/net/tcp6') { + var listening = (content.match(/^[^\n]*\s0A\s/gm) || []).length; + out[f] = 'lines=' + content.split('\n').length + ' listening=' + listening; + } else if (f === '/proc/self/maps') { + var hits = (content.match(/frida|gum|gadget|linjector/gi) || []); + out.frida_named_maps = hits.length; + out.map_sample = hits.slice(0, 5); + } else { + out[f] = 'readable (' + content.length + ' B)'; + } + } catch (e) { + out[f] = 'unreadable: ' + e.message; + } + }); + try { + out.current_thread = Process.getCurrentThreadId(); + out.threads = Process.enumerateThreads().map(function (t) { return t.id; }).slice(0, 24); + } catch (e) { out.threads = 'enumerate failed: ' + e.message; } + try { + out.modules_named = Process.enumerateModules() + .filter(function (m) { return /frida|gum|gadget|linjector/i.test(m.name); }) + .map(function (m) { return m.name + '@' + m.base; }); + } catch (e) { out.modules_named = []; } + return out; +} + +function report() { + var out = { + kind: 'anti_detect_probe', + mode: 'observer-only (this script never changes the target)', + ranMs: Date.now() - started, + counts: counts, + details: lines, + selfView: selfView(), + }; + send({ type: 'RESULT', payload: out }); + console.log('\n=== RESULT (probe) ==='); + console.log('ranMs=' + out.ranMs); + Object.keys(counts).sort().forEach(function (k) { + console.log(' ' + k + ' = ' + counts[k]); + }); + var sv = out.selfView; + console.log(' tracerpid=' + sv.tracerpid + ' frida_named_maps=' + sv.frida_named_maps + + ' frida_modules=' + JSON.stringify(sv.modules_named)); + var interesting = ['termination', 'thread_create', 'env_probe', 'loader']; + interesting.forEach(function (cat) { + if (!lines[cat]) return; + console.log('\n=== ' + cat + ' ==='); + lines[cat].forEach(function (l) { console.log(' ' + l); }); + }); + console.log('\n=== how to read this ==='); + console.log(' a hook that reports nothing is a finding about the hook (is the export'); + console.log(' really there? did the code run?) -- establish the pipeline on a control'); + console.log(' target before concluding "no detection".'); +} + +function start() { + hookExport('libc.so', 'open', 'env_probe', 0); + hookExport('libc.so', 'openat', 'env_probe', 1); + hookExport('libc.so', 'fopen', 'env_probe', 0); + hookExport('libc.so', 'fopen64', 'env_probe', 0); + hookExport('libc.so', 'access', 'env_probe', 0); + hookExport('libc.so', 'stat', 'env_probe', 0); + hookExport('libc.so', 'stat64', 'env_probe', 0); + hookExport('libc.so', 'lstat', 'env_probe', 0); + hookExport('libc.so', 'readlink', 'env_probe', 0); + hookExport('libc.so', 'opendir', 'env_probe', 0); + hookExport('libc.so', 'scandir', 'env_probe', 0); + hookExport('libc.so', 'strstr', 'strstr_probe', 1); + hookExport('libc.so', 'strcmp', 'strcmp_probe', 1); + hookTerminators(); + hookThreadCreation(); + hookLoader(); + console.log('=== anti_detect_probe armed (observer only) ==='); + var sv0 = selfView(); + console.log('selfView at arm time: ' + JSON.stringify(sv0)); + send({ type: 'LIVE', cat: 'armed', t: 0, msg: 'script armed', selfView: sv0 }); + if (CONFIG.heartbeatMs > 0) { + // The heartbeat is the liveness proof. Its absence at time T means the script + // (or the whole process) was gone before T, which is a different finding from + // "the target was quiet". + setInterval(function () { + send({ type: 'LIVE', cat: 'heartbeat', t: Date.now() - started, + counts: JSON.parse(JSON.stringify(counts)) }); + }, CONFIG.heartbeatMs); + } + if (CONFIG.watchMs > 0) { + var t = setTimeout(report, CONFIG.watchMs); + setTimeout(function () { clearTimeout(t); }, CONFIG.watchMs + 1); + } +} + +rpc.exports = { + config: function (patch) { Object.assign(CONFIG, patch); return CONFIG; }, + status: function () { return { counts: counts, ranMs: Date.now() - started }; }, + report: function () { report(); return true; }, + // Kept for interface parity with the kit's other probes; nothing is patched. + patch: function () { return 'observed-only: nothing patched'; }, +}; + +if (Java.available) { + // Only to record what the Java side is, not to hook it: an observer that hooks + // Java would change the timing it is there to measure. + try { + Java.perform(function () { + console.log('java classloader ready, app=' + + (Java.androidVersion ? 'Android ' + Java.androidVersion : 'unknown')); + }); + } catch (e) { /* Java optional */ } +} + +setImmediate(start); diff --git a/skills/apk-reverse/scripts/scan_leaks.py b/skills/apk-reverse/scripts/scan_leaks.py new file mode 100644 index 0000000..0468d46 --- /dev/null +++ b/skills/apk-reverse/scripts/scan_leaks.py @@ -0,0 +1,757 @@ +#!/usr/bin/env python3 +"""Leak scan for a skills repository: find target identity that should not be shipped. + +Why this exists: a skill repository is published, and the material feeding it is real work +notes -- device transcripts, packet captures, package listings. Target identity leaks into +those notes one line at a time (a bundle id in a `pm path` line, a device serial echoed +before a `dumpsys`, a token pasted while debugging a download), and it is invisible to +every other gate: `check_repo.py` verifies structure, `check_refs.py` verifies anchors, +neither looks at *content*. Human grep finds it only after someone thinks to look. + +Two principles, both borrowed from published anonymization practice, shape the design: + + 1. **A "do not anonymize" list.** Tool names, library names, function names, protocol + field names, CVE ids, hardening-product names, public crackme/benchmark names and + URLs, and placeholders like `<PKG>`/`<DEVICE>` are *reusable* and must never be + reported. A scanner that flags them trains its readers to ignore it. + 2. **Context retention.** A finding is reported with the surrounding characters of its + line, not a bare line number, so the fixer does not have to reopen the file to know + what to delete. + +Exit codes are fixed so this can gate a pipeline: + + 0 clean nothing found outside the exemptions + 0 leaks_found_strong_only findings exist, but every one of them is `weak` + 1 leaks_found at least one strong/certain finding, or any finding + at all under --fail-on any + 2 error usage or read error + +and the last line of stdout is always one of + + RESULT=clean | leaks_found_strong_only | leaks_found | error + +`--fail-on strong` (the default) decides which findings take the exit code off zero. It is the +maintenance-habit setting, and it exists because a repository's own documentation legitimately +contains RFC 5737 addresses, which the endpoint rules report on purpose; `--fail-on any` is the +strict pass to run before publishing. Neither setting changes what is printed. + +Everything is a heuristic. A finding means "a human should look at this line", never +"this is definitely a secret". Strength labels (`certain` / `strong` / `weak`) say how +much of the matching was context rather than shape -- see references for the table. + +Usage +----- + python scan_leaks.py # scan the repo it lives in + python scan_leaks.py --root . # explicit root + python scan_leaks.py --root . --format json # machine-readable + python scan_leaks.py --root . --only pat,appkey # one or more categories + python scan_leaks.py --root . --quiet # token line only + python scan_leaks.py --root . --fail-on any # strict: weak hits fail too + python scan_leaks.py --root . --list-rules + +Only text files are read. Directories named in --exclude, VCS metadata, caches and +`tools/` (a git-ignored work area by convention) are skipped; a root supplied explicitly +is still scanned even if it sits under an excluded name. + +Pure standard library, python3, no third-party imports. POSIX and Windows. +""" +from __future__ import annotations + +import argparse +import json +import os +import re +import sys + +# -------------------------------------------------------------------------------------- +# Exemption data -- the "do not anonymize" list, expressed as code so it can be audited. +# -------------------------------------------------------------------------------------- + +# Reverse-domain prefixes that are never a target's own identity: platforms, frameworks, +# SDK vendors, JDK/ART packages, documentation examples and this task's own fixtures. +# Matching is on the value itself and on its first two segments, lowercased. +BENIGN_PACKAGE_PREFIXES = ( + # platforms / runtimes / language packages + "android", "androidx", "java", "javax", "jdk", "kotlin", "kotlinx", "dalvik", "org.w3c", + "org.xml", "org.json", "org.apache", "org.jetbrains", "org.lsposed", "org.reactivestreams", + "com.android", "com.google", "com.sun", "com.oracle", "com.squareup", "com.github", + "io.reactivex", "io.github", "dalvik.system", + # ad / analytics SDKs and their activities (these are inventory, not identity) + "com.bytedance", "com.qq.e", "com.tencent", "com.kwad", "com.kuaishou", "com.baidu", + "com.umeng", "com.sigmob", "com.bdx", "com.anythink", "com.mbridge", "com.chinatowercom", + "cn.connor", "bytedance", + # hardening products, by their own naming (a product name, not a target) + "com.stub", "com.secneo", "com.qihoo", "com.nqshield", "com.tencent.StubShell", + "com.wrapper", "com.eg.android.AlipayGphone", + # documentation examples / fixtures used by this repository itself + "com.example", "org.example", "net.example", "io.example", "probe.synthetic", + # probe modules built by this repository's own verification passes + "com.t1", "com.revprobe", "com.lsphook", +) + +# Whole-value exemptions for the `bundle` rule: placeholders and documented stand-ins. +PLACEHOLDER_TOKENS = ( + "<pkg>", "<device>", "<serial>", "<app>", "<sample>", "<target>", "<host>", "<token>", + "<label>", "<hash>", "<work>", "<out>", "<path>", "<user>", "<name>", "<id>", "<n>", + "<pid>", "<activity>", "<module>", "<key>", "<value>", "<abi>", "<file>", "<dir>", + "${", "{{", "%s", "%(", "$env:", "$(", "xn--", +) + +# IPs that are loopback, unspecified, emulator-host or link-local. Note that the RFC 5737 +# documentation ranges (192.0.2.0/24, 198.51.100.0/24, 203.0.113.0/24) are deliberately NOT +# exempt: they are the *correct* way to write a synthetic address in a document, so a hit +# there is a signal rather than noise, and suppressing them would hide a real one. +# +# That decision costs the gate its pass condition -- this repository's own documents and its +# own usage example keep producing `endpoint/weak` hits -- so the two are separated instead: +# they stay reported, and `--fail-on strong` (the default) does not fail on them. See the +# reference file for the reasoning: a gate that cannot go green is a gate people learn to +# ignore, and `--fail-on any` is there for the publish-time strict pass. +BENIGN_IP_PREFIXES = ( + "127.", "0.0.0.0", "255.255.255.255", "169.254.", "224.", "239.", "10.0.2.", "10.0.3.", +) + +# Deterministic-value words that make a package token a component name rather than a bundle +# id. A camelCase segment or one of these words at the end of a reverse-domain string is a +# *thing*, not an app -- `com.android.pathclassloader`-shaped text in a log line, or a +# library identifier quoted in a reference file. +CODE_IDENTIFIER_HINTS = ( + "pathclassloader", "dexclassloader", "bootclassloader", "inmemorydexclassloader", + "loadedapk", "activitythread", "instrumentation", "application", "activity", + "service", "provider", "receiver", "helper", "manager", "factory", "impl", "utils", + "constants", "buildconfig", "r8", "internal", "runtime", "validator", "inspector", + "converter", "adapter", "listener", "callback", "module", "trampoline", "gen", + # string/collection methods: `args.package.split(...)` is code, not a bundle id + "split", "substring", "indexof", "lastindexof", "tostring", "tolowercase", + "touppercase", "equals", "hashcode", "getname", "getvalue", "length", "format", +) + +# Public crackme / benchmark families whose *reverse-engineered* package names appear in +# write-ups. These are public targets, and naming them is expected reuse rather than a leak. +PUBLIC_TARGET_PREFIXES = ( + "sg.vantagepoint", "owasp.mstg", "jakhar.aseem", "com.revo", "com.example.uncrackable", +) + + +def _is_placeholder(value: str) -> bool: + low = value.lower() + return any(tok in low for tok in PLACEHOLDER_TOKENS) + + +def _slug_segments(value: str) -> list: + return [seg for seg in re.split(r"[.\-_/]", value.lower()) if seg] + + +def _has_camel_mid_segment(value: str) -> bool: + # `com.foo.BarBaz` in prose: a segment carrying an interior capital is a class name. + for seg in re.split(r"[.\\/]", value): + if len(seg) > 3 and re.search(r"[a-z][A-Z]", seg): + return True + return False + + +def _looks_like_code_identifier(value: str) -> bool: + segs = _slug_segments(value) + if not segs: + return False + if segs[-1] in CODE_IDENTIFIER_HINTS: + return True + if _has_camel_mid_segment(value): + return True + # `com.example.target.Helper.method` style: three or more segments whose tail is short + # and lowercase with no digits is still prose, so only flag when a known hint appears. + return False + + +def _benign_package(value: str) -> str: + """Return an exemption reason for a reverse-domain token, or '' if it is reportable.""" + low = value.lower().strip("`'\"()[],;:") + if not low: + return "empty" + if _is_placeholder(low): + return "placeholder" + if re.search(r"^[a-z0-9_]+(\.[a-z0-9_]+)+$", low) is None: + return "not a reverse-domain token" + if low.startswith(("ca.", "de.", "jp.", "uk.", "fr.", "ru.", "cn.", "top.", "xyz.")) \ + and low.count(".") >= 3: + return "not a reverse-domain token" + for prefix in BENIGN_PACKAGE_PREFIXES: + if low == prefix.lower() or low.startswith(prefix.lower() + "."): + return "benign prefix: " + prefix + for prefix in PUBLIC_TARGET_PREFIXES: + if low == prefix.lower() or low.startswith(prefix.lower() + "."): + return "public crackme/benchmark package: " + prefix + two = ".".join(low.split(".")[:2]) + for prefix in BENIGN_PACKAGE_PREFIXES: + if two == prefix.lower(): + return "benign prefix (2-segment): " + prefix + if _looks_like_code_identifier(low): + return "code identifier, not a bundle id" + if low.count(".") < 2: + return "single-label pair, too generic to be a bundle id" + return "" + + +def _benign_ip(value: str) -> str: + ip = value.split(":")[0] + for prefix in BENIGN_IP_PREFIXES: + if ip == prefix.rstrip(".") or ip.startswith(prefix): + return "loopback/unspecified/emulator address" + return "" + + +def _benign_path(value: str) -> str: + low = value.lower() + if _is_placeholder(low): + return "placeholder user" + user = re.split(r"[\\/]", low.rstrip("\\/"))[-1] + if user in ("users", "home", "local", "tmp", "data", "appdata", "temp", "private"): + return "no user segment" + return "" + + +# A four-part dotted run of digits is an IPv4 address only when nothing nearby says +# "version". `JDK 17.0.4.1`, `build-tools 34.0.0` and `frida 16.7.19` are the shapes that +# actually occur in this repository's evidence files, and all of them are legitimate. +VERSION_CONTEXT_RE = re.compile( + r'(?:jdk|jre|java|python|frida|node|npm|gradle|build-tools|ndk|apktool|version|ver|v)\W*$', + re.IGNORECASE) + +# An SDK's own version directory: `/5.6.10.1/`, `/1.2.3.4/`. Package-manager cache paths look +# exactly like an IPv4 literal and are inside an APK's private storage, not an endpoint. +VERSION_PATH_RE = re.compile(r'[/\\]$') + +# Code member access: `args.package.split(...)`, `self.cfg.value`. A reverse-domain **shape** +# with a program's own variable in front of it is a property lookup, never a bundle id. +CODE_MEMBER_ACCESS_RE = re.compile(r'(?:\b(?:args|argv|self|this|opts|options|cfg|config)\.)') + + +def _benign_ip_literal(m, line: str) -> str: + """Return an exemption reason for a bare IPv4-shaped literal, or ''.""" + reason = _benign_ip(m.group(1)) + if reason: + return reason + head = line[max(0, m.start() - 30):m.start()] + if VERSION_CONTEXT_RE.search(head): + return "version string, not an address" + if VERSION_PATH_RE.search(head) and line[m.end():m.end() + 1] in ("/", "\\"): + return "version directory in a package cache path, not an address" + return "" + + +def _package_exempt(m, line: str, group: int = 1) -> str: + """Line-aware bundle-id exemption: code member access first, then the benign list. + + The window starts at the capture itself, not at the whole match: an unbounded prefix in + some rules means the match can begin well before the value, and a window that stops at + `m.start()` would then never see the `args.` that makes it a property lookup. + """ + vstart = m.start(group) + if CODE_MEMBER_ACCESS_RE.search(line[max(0, vstart - 24):vstart + 12]): + return "code member access, not a bundle id" + return _benign_package(m.group(group)) + + +# -------------------------------------------------------------------------------------- +# Rules +# -------------------------------------------------------------------------------------- +# Each rule: id, category, description, strength, pattern, exemption function (optional). +# `strength` describes how much of the decision came from context rather than shape: +# certain -- shape alone is conclusive (a provider-specific token format) +# strong -- shape plus a nearby context word agrees +# weak -- pattern only; expect false positives, read the context + +RULE_SPECS = [ + ( + "bundle_pkg_attr", "package", "a `package=` bundle id in a manifest or transcript", + "strong", r'\bpackage\s*=\s*"([A-Za-z_][A-Za-z0-9_.]*)"', lambda m: _benign_package(m.group(1)), + ), + ( + "bundle_pkg_decl", "package", "a bundle id in a `<manifest>` line", + "strong", r'<manifest[^>\n]*?\spackage="([^"]+)"', lambda m: _benign_package(m.group(1)), + ), + ( + "bundle_pm_path", "package", "a bundle id in a `pm`/`am`/`monkey` invocation", + "strong", r'\b(?:pm|am|monkey|cmd package)\s+[^\n]{0,40}?\b([a-z][a-z0-9_]*\.[a-z0-9_]+(?:\.[a-z0-9_]+)+)', + lambda m: _benign_package(m.group(1)), + ), + ( + "bundle_process_line", "package", "a bundle id in a `ps`/`pidof`/`top` capture", + "strong", + r'\b(?:pidof|ps -A|ps -e|top)\b[^\n]*?(?<![\w.])([a-z][a-z0-9_]*\.[a-z0-9_]+(?:\.[a-z0-9_]+)+)(?!\s*\()', + lambda m: _benign_package(m.group(1)), + ), + ( + "bundle_component", "package", "a bundle id in a `component=`/`-n` activity reference", + "strong", r'(?:component\s*=|/\.|-\s*n\s+)(?<![\w.])([a-z][a-z0-9_]*\.[a-z0-9_]+(?:\.[a-z0-9_]+)+)(?=/)', + lambda m: _benign_package(m.group(1)), + ), + ( + "device_serial", "device", "an `adb devices`-shaped device serial", + "strong", r'(?<![\w.-])(?=[A-Z0-9]{16}(?![\w.-]))(?=[A-Z0-9]{0,15}\d)([A-Z0-9]{16})(?![\w.-])', + # A bare run of 16 hex digits is usually a stack address, a hash fragment or a + # protocol field -- the three cases this repository actually produces -- so an + # all-digits token is suppressed. A real platform serial carries at least one letter. + lambda m: "all-digit token, not a device serial" if m.group(1).isdigit() else "", + ), + ( + "device_serial_context", "device", "a serial next to a device/install/`su` context word", + "strong", + r'(?:serial|device|adb|install|flash|\bsu\b)[^\n]{0,40}?\b([A-Z0-9]{16})\b', + None, + ), + ( + "device_serial_tabular", "device", "a bare 16-char token in a device-listing table", + "strong", + r'\b([A-Z0-9]{16})\t+(?:device|unauthorized|offline|no permissions)\b', + None, + ), + ( + "token_github_pat", "token", "a GitHub fine-grained personal access token", + "certain", r'github_pat_[A-Za-z0-9_]{20,}', None, + ), + ( + "token_github_classic", "token", "a GitHub classic token", + "certain", r'\bgh[pousr]_[A-Za-z0-9]{20,}', None, + ), + ( + "token_env_assignment", "token", "an inline API-key/token/secret assignment", + "strong", + r'\b(?:API_?KEY|APIKEY|ACCESS_?TOKEN|AUTH_?TOKEN|SECRET_?KEY|SECRET|PASSWORD|PASSWD|BEARER_?TOKEN)\s*[=:]\s*["\']?([A-Za-z0-9_\-./+]{12,})', + None, + ), + ( + "token_authorization_header", "token", "a literal Authorization header value", + "strong", r'(?:Authorization|X-Auth-Token)["\']?\s*:\s*["\']?(?:Bearer\s+|Basic\s+)?([A-Za-z0-9_\-./+=]{16,})', + None, + ), + ( + "appkey_assignment", "appkey", "an SDK appkey/appsecret assignment with a literal value", + "strong", + r'\b(?:APPKEY|APP_KEY|appSecretKey|AppSecret|APP_SECRET|SECRET_KEY|UMENG_APPKEY|com\.tencent\.map\.api\.KEY)\b\s*[=:]\s*["\']?([A-Za-z0-9_\-]{8,})', + None, + ), + ( + "plain_addr", "endpoint", "a non-loopback literal IP:port", + "weak", r'(?<![\w.])(\d{1,3}(?:\.\d{1,3}){3}:\d{1,5})(?![\w.])', None, + ), + ( + "plain_addr_ipliteral", "endpoint", "a non-loopback bare IPv4 literal", + "weak", + # Four dot-separated 1-3 digit groups, not bounded by further dots (which is what a + # version string looks like: `17.0.4.1`, `5.6.10.1`) and not bounded by word + # characters (which is what a file name looks like). + r'(?<![\w.])(\d{1,3}(?:\.\d{1,3}){3})(?![\w.])', None, + ), + ( + "user_path_posix", "path", "an absolute POSIX user-home path", + "strong", r'(?:/home/|/Users/)([A-Za-z][A-Za-z0-9._-]{1,31})/', lambda m: _benign_path(m.group(0)), + ), + ( + "user_path_windows", "path", "an absolute Windows user-profile path", + "strong", r'[A-Za-z]:\\Users\\([^\\/\s`"\']+)', lambda m: _benign_path(m.group(0)), + ), +] + +# A user path whose user segment *is* a placeholder (`C:\Users\<user>\...`) must not report; +# handled by the exemption functions above plus this post-filter. +PLACEHOLDER_PATH_RE = re.compile(r'(?:/home/|/Users/|[A-Za-z]:\\Users\\)[<{[$%]') + + +def build_rules(): + rules = [] + for rid, cat, desc, strength, pattern, exempt in RULE_SPECS: + rules.append({ + "id": rid, + "category": cat, + "description": desc, + "strength": strength, + "regex": re.compile(pattern), + "exempt": exempt, + }) + return rules + +# -------------------------------------------------------------------------------------- +# Walking +# -------------------------------------------------------------------------------------- + +TEXT_EXTENSIONS = (".md", ".py", ".js", ".json", ".txt", ".yml", ".yaml", ".sh", ".ps1", + ".toml", ".cfg", ".ini", ".xml", ".java", ".smali", ".html", ".csv") + +SKIP_DIRS = { + ".git", ".hg", ".svn", "__pycache__", "node_modules", ".venv", "venv", ".mypy_cache", + ".pytest_cache", ".idea", ".vscode", ".tox", "site-packages", "dist", "build", +} + +DEFAULT_EXCLUDES = ("tools", "存在问题和例子", "新的建议和思路", ".venv", "node_modules") + +DEFAULT_ROOT_FILES = ("README.md",) +DEFAULT_ROOT_DIRS = ("skills", "docs") +ROOT_GLOBS = ("check_*.py", "build_scripts.py") + + +def _norm(path: str) -> str: + return os.path.normpath(os.path.abspath(path)).replace("\\", "/").lower() + + +def iter_files(root: str, excludes, skip_self: bool, from_list=None): + """Yield candidate text files. + + Layout-sensitive on purpose: given a repository root (one holding `skills/` or the + maintenance scripts) it scans exactly the shipped surface -- README, skills/, docs/ and the + root check scripts -- so that a git-ignored work area is not reported as if it were + publishable. Given any other directory (a fixture, a subtree) it scans that directory + recursively, because a caller who names a path means it. + + `from_list` overrides both behaviours: it is an explicit list of paths (relative to `root`, or + absolute) and nothing else is read. The maintenance gate passes the tracked-file list here. + """ + root_abs = os.path.abspath(root) + # An explicit list is an explicit instruction: the *default* directory exemptions (a local + # `tools/` work area, a sample folder) are about what a tree walk would otherwise pick up, and + # must not silently drop a path the caller named. Only `--exclude` narrows an explicit list. + if from_list is not None: + excluded = [_norm(os.path.join(root_abs, e)) for e in (excludes or []) + if e not in DEFAULT_EXCLUDES] + else: + excluded = [_norm(os.path.join(root_abs, e)) for e in excludes] + self_path = _norm(__file__) if skip_self else None + + def is_excluded(path: str) -> bool: + norm = _norm(path) + if self_path and norm == self_path: + return True + for ex in excluded: + if norm == ex or norm.startswith(ex + "/"): + return True + return False + + candidates = [] + if from_list is not None: + for entry in from_list: + full = entry if os.path.isabs(entry) else os.path.join(root_abs, entry) + if os.path.isfile(full): + candidates.append(full) + elif os.path.isfile(root_abs): + candidates.append(root_abs) + else: + looks_like_repo = (os.path.isdir(os.path.join(root_abs, "skills")) + or os.path.isfile(os.path.join(root_abs, "check_repo.py"))) + if looks_like_repo: + for name in DEFAULT_ROOT_FILES: + p = os.path.join(root_abs, name) + if os.path.isfile(p): + candidates.append(p) + for name in DEFAULT_ROOT_DIRS: + p = os.path.join(root_abs, name) + if os.path.isdir(p): + candidates.append(p) + for entry in sorted(os.listdir(root_abs)): + full = os.path.join(root_abs, entry) + if os.path.isfile(full): + for glob_pat in ROOT_GLOBS: + if re.fullmatch(glob_pat.replace("*", ".*"), entry): + candidates.append(full) + break + else: + candidates.append(root_abs) + + seen = set() + for cand in candidates: + if is_excluded(cand): + continue + if os.path.isfile(cand): + key = _norm(cand) + if key not in seen: + seen.add(key) + yield cand + continue + for dirpath, dirnames, filenames in os.walk(cand): + dirnames[:] = sorted(d for d in dirnames + if d not in SKIP_DIRS and not is_excluded(os.path.join(dirpath, d))) + for fn in sorted(filenames): + full = os.path.join(dirpath, fn) + if not fn.lower().endswith(TEXT_EXTENSIONS): + continue + if is_excluded(full): + continue + key = _norm(full) + if key in seen: + continue + seen.add(key) + yield full + + +def read_text(path: str): + try: + with open(path, "r", encoding="utf-8", errors="strict") as fh: + return fh.read(), None + except UnicodeDecodeError: + try: + with open(path, "r", encoding="utf-8", errors="replace") as fh: + return fh.read(), "decoded with replacement characters" + except OSError as exc: + return None, "read failed: %s" % exc + except OSError as exc: + return None, "read failed: %s" % exc + + +def context_of(line: str, start: int, end: int, before: int = 48, after: int = 48) -> str: + left = max(0, start - before) + right = min(len(line), end + after) + head = "..." if left > 0 else "" + tail = "..." if right < len(line) else "" + return head + line[left:right].strip() + tail + + +def scan_text(path: str, rel: str, text: str, rules, only): + findings = [] + for lineno, line in enumerate(text.splitlines(), 1): + if len(line) > 4000: + # a minified line has no useful context; still scan it, clipped + pass + for rule in rules: + if only and rule["category"] not in only: + continue + for m in rule["regex"].finditer(line): + value = m.group(1) if m.groups() else m.group(0) + if not value: + continue + if rule["id"] in ("plain_addr", "plain_addr_ipliteral"): + reason = _benign_ip(value) + if not reason and rule["id"] == "plain_addr_ipliteral": + reason = _benign_ip_literal(m, line) + elif rule["id"] in ("user_path_windows", "user_path_posix"): + reason = _benign_path(value) + if not reason and PLACEHOLDER_PATH_RE.search(m.group(0)) is None: + # `<user>` style is already covered by _benign_path; re-check the + # *whole* match for a placeholder segment before reporting. + if _is_placeholder(m.group(0)): + reason = "placeholder path segment" + elif rule["category"] == "package": + reason = _package_exempt(m, line, rule.get("group", 1)) + else: + reason = "" + if rule["exempt"] is not None: + reason = rule["exempt"](m) + if reason: + findings.append({ + "file": rel, "line": lineno, "column": m.start() + 1, + "rule": rule["id"], "category": rule["category"], + "strength": rule["strength"], "match": value, + "context": context_of(line, m.start(), m.end()), + "exempted": True, "exempt_reason": reason, + }) + continue + findings.append({ + "file": rel, "line": lineno, "column": m.start() + 1, + "rule": rule["id"], "category": rule["category"], + "strength": rule["strength"], "match": value, + "context": context_of(line, m.start(), m.end()), + "exempted": False, "exempt_reason": "", + }) + return findings + + +# -------------------------------------------------------------------------------------- +# Reporting +# -------------------------------------------------------------------------------------- + +def rel_of(path: str, root: str) -> str: + try: + return os.path.relpath(path, os.path.abspath(root)).replace("\\", "/") + except ValueError: + return path.replace("\\", "/") + + +def main(argv=None): + parser = argparse.ArgumentParser( + prog="scan_leaks.py", + description="Scan a skills repository for target identity that should not be published " + "(bundle ids, device serials, tokens, SDK keys, literal endpoints, user paths). " + "Exit 0 clean / 1 leaks_found / 2 error; prints RESULT=<token> last. " + "`--fail-on strong` (default) keeps the gate green when the only hits are " + "documentation-range addresses.", + formatter_class=argparse.RawDescriptionHelpFormatter, + epilog="Categories: package, device, token, appkey, endpoint, path\n" + "Exemptions (tool names, library names, CVE ids, hardening products, public\n" + "crackme names, <PKG>-style placeholders, loopback/emulator addresses) are built\n" + "in and audited with --show-exempt.\n") + default_root = os.path.dirname(os.path.dirname(os.path.dirname(os.path.dirname( + os.path.abspath(__file__))))) + parser.add_argument("--root", default=default_root, + help="repository root to scan (default: the repo this script lives in)") + parser.add_argument("--only", default="", + help="comma-separated categories to restrict to, e.g. pat,appkey") + parser.add_argument("--exclude", action="append", default=[], + help="extra directory name to skip (repeatable)") + parser.add_argument("--format", choices=("text", "json"), default="text") + parser.add_argument("--show-exempt", action="store_true", + help="also print findings that were suppressed by an exemption") + parser.add_argument("--list-rules", action="store_true", help="print the rule table and exit") + parser.add_argument("--max", type=int, default=0, + help="stop after N findings (0 = no limit)") + parser.add_argument("--quiet", action="store_true", help="print only the RESULT token") + parser.add_argument("--fail-on", choices=("strong", "any"), default="strong", + help="which findings make the exit code non-zero: `strong` (default) " + "fails only on strong/certain findings, `any` also fails on `weak` " + "ones such as an address from the RFC 5737 documentation range") + parser.add_argument("--no-color", action="store_true", help="accepted for compatibility") + parser.add_argument("--files-from", default="", + help="scan exactly the paths listed in this file (one per line, relative " + "to --root or absolute) instead of walking the tree -- the " + "maintenance gate passes the tracked-file list here, so a git-ignored " + "work area cannot produce findings") + args = parser.parse_args(argv) + + rules = build_rules() + + if args.list_rules: + print("== rules ==") + for r in rules: + print(" %-26s %-9s %-8s %s" % (r["id"], r["category"], r["strength"], r["description"])) + print("RESULT=clean") + return 0 + + only = {c.strip() for c in args.only.split(",") if c.strip()} + known = {r["category"] for r in rules} + unknown = only - known + if unknown: + print("error: unknown category (or categories): %s" % ", ".join(sorted(unknown)), + file=sys.stderr) + print("RESULT=error") + return 2 + + root = os.path.abspath(args.root) + if not os.path.isdir(root) and not os.path.isfile(root): + print("error: root does not exist: %s" % root, file=sys.stderr) + print("RESULT=error") + return 2 + + excludes = list(DEFAULT_EXCLUDES) + list(args.exclude) + kept, exempted, errors = [], [], [] + files_scanned = 0 + + from_list = None + if args.files_from: + # A list of exactly what a caller wants scanned (paths relative to --root, or absolute). + # This is how the maintenance gate restricts the scan to the *committed* surface: a + # git-ignored work area may legitimately hold real identifiers, and scanning it would turn + # a leak gate into a false-positive generator. + try: + with open(args.files_from, "r", encoding="utf-8") as fh: + from_list = [ln.strip().strip('"') for ln in fh + if ln.strip() and not ln.startswith("#")] + except OSError as exc: + print("error: cannot read --files-from %s: %s" % (args.files_from, exc), + file=sys.stderr) + print("RESULT=error") + return 2 + + for path in iter_files(root, excludes, skip_self=True, from_list=from_list): + text, err = read_text(path) + if text is None: + errors.append("%s: %s" % (rel_of(path, root), err)) + continue + files_scanned += 1 + rel = rel_of(path, root) + for f in scan_text(path, rel, text, rules, only): + if f["exempted"]: + exempted.append(f) + else: + kept.append(f) + if args.max and len(kept) >= args.max: + break + if args.max and len(kept) >= args.max: + break + + kept.sort(key=lambda f: (f["file"], f["line"], f["column"], f["rule"])) + exempted.sort(key=lambda f: (f["file"], f["line"], f["column"], f["rule"])) + + # Collapse a rule firing more than once on the same value in the same line (labels such + # as "device serial:" match a context rule and the shape rule at two offsets). Cross-rule + # agreement on one line is kept: two independent rules naming the same value is stronger + # evidence, not repetition. + deduped, seen_keys = [], set() + for f in kept: + key = (f["file"], f["line"], f["rule"], f["match"]) + if key in seen_keys: + continue + seen_keys.add(key) + deduped.append(f) + kept = deduped + + result = "leaks_found" if kept else "clean" + if kept and args.fail_on == "strong": + # Weak findings are shapes a document may legitimately contain (an address from the + # RFC 5737 documentation range in a usage example). They are still printed; they just + # do not fail the gate. Strong/certain findings always do. + result = "leaks_found_strong_only" if all( + f["strength"] == "weak" for f in kept) else "leaks_found" + + if args.format == "json": + print(json.dumps({ + "root": root, + "files_scanned": files_scanned, + "findings": kept, + "exempted": exempted if args.show_exempt else [], + "errors": errors, + "result": result, + "fail_on": args.fail_on, + }, indent=2, ensure_ascii=False)) + print("RESULT=%s" % result) + return (2 if errors else 0) if result in ("clean", "leaks_found_strong_only") else 1 + + if not args.quiet: + print("== leak scan: %d file(s) scanned under %s ==" % (files_scanned, root)) + if kept: + counts = {} + for f in kept: + counts[f["category"]] = counts.get(f["category"], 0) + 1 + print(" findings by category: " + ", ".join( + "%s=%d" % (k, counts[k]) for k in sorted(counts))) + if errors: + for e in errors: + print(" READ-ERROR %s" % e) + + for f in kept: + print("") + print(" %s:%d:%d [%s/%s] %s" + % (f["file"], f["line"], f["column"], f["category"], f["strength"], f["rule"])) + print(" match: %s" % f["match"]) + print(" context: %s" % f["context"]) + + if args.show_exempt and exempted: + print("") + print("== suppressed by exemption: %d ==" % len(exempted)) + for f in exempted: + print(" %s:%d [%s] %s -> %s" + % (f["file"], f["line"], f["rule"], f["match"], f["exempt_reason"])) + + print("") + if kept: + print("== result: %d finding(s) -- each one is a line to look at, not a verdict ==" + % len(kept)) + if result == "leaks_found_strong_only": + print(" all are `weak` (the documented address ranges); the gate is green under" + " the default --fail-on strong, and `--fail-on any` is the strict pass") + else: + print("== result: clean (nothing outside the exemption list) ==") + if errors: + print("== read errors: %d ==" % len(errors)) + + print("RESULT=%s" % result) + if errors and not kept: + return 2 + if result in ("clean", "leaks_found_strong_only"): + return 0 + return 1 + + +if __name__ == "__main__": + if hasattr(sys.stdout, "reconfigure"): + try: + sys.stdout.reconfigure(encoding="utf-8") + except (ValueError, OSError): + pass + sys.exit(main()) diff --git a/skills/apk-reverse/scripts/svc_scan.py b/skills/apk-reverse/scripts/svc_scan.py new file mode 100644 index 0000000..9676fef --- /dev/null +++ b/skills/apk-reverse/scripts/svc_scan.py @@ -0,0 +1,467 @@ +#!/usr/bin/env python3 +"""Scan an ELF or a raw memory capture for inline `svc` instructions and name the syscall. + +This is the tool behind one question: **is a libc-level hook even capable of seeing +this call?** Code that issues `svc #0` directly -- or through its own inline +`exit`/`kill`/`mprotect` table -- never enters libc, so `Interceptor.replace` on +`exit_group`/`kill`/`mprotect` cannot observe it and a userspace "block the exit" +strategy silently does nothing. That is the discriminator in +`references/detection-and-anti-analysis.md` between "the exit came from libc" and +"the exit bypassed libc": if the library that dies carries its own svc sites for +`exit`/`kill`, a libc hook is the wrong instrument. + +It is also the sanity check for the *other* direction: a library with **no** svc +sites for the termination syscalls is evidence that a libc-level hook can work, so a +missing event is a finding about the hook rather than about the target. + +How the syscall number is recovered, and why it is a heuristic: + +- arm64 loads the number in `x8`, almost always as `movz x8, #imm` (or `mov` to x8, + which assembles to the same), immediately before the `svc`. The scan looks back a + short BOUNDED window for the nearest register-writing instruction that targets x8, + and resolves the common immediate forms (`movz`, `mov` alias, `orr x8, xzr, #imm`). + A number computed at runtime, or loaded from a table, will not resolve -- those + sites are reported as `nr=?` with the raw instruction that wrote x8, because + "I cannot read this one" is a different statement from "this is not a syscall". +- arm (32-bit Thumb/A32) uses `r7` and a `svc` with an 8-bit immediate; the same + bounded lookback applies. + +This script **finds and names**; it does not decide intent. A library full of +`svc` sites that are all `futex`/`read` is ordinary libc-free code, not evasion. +Read the histogram, not the total. + +Usage: + python svc_scan.py libfoo.so + python svc_scan.py /data/local/tmp/libjiagu_a64.so --json + python svc_scan.py linker64 --arch arm64 --follow-exec + python svc_scan.py region.bin --raw --arch arm64 --base 0x7a12340000 + +Exit codes: 0 = scan completed (hits or not), 1 = the input could not be read or the +architecture could not be determined. A scan that finds zero svc sites exits 0: that +is a result, not an error. +""" +import argparse +import json +import os +import struct +import sys + +try: + from capstone import Cs, CS_ARCH_ARM64, CS_ARCH_ARM, CS_MODE_ARM, CS_MODE_THUMB + from capstone.arm64 import ARM64_OP_IMM, ARM64_OP_REG + from capstone.arm import ARM_OP_IMM, ARM_OP_REG +except ImportError: # pragma: no cover - dependency message is the point + sys.stderr.write( + "capstone is required: python -m pip install capstone\n" + "(this repository's other native scripts use it as well)\n") + raise SystemExit(1) + +# arm64 syscall numbers (linux, asm-generic). Only the ones worth naming: a scan +# report that names `futex` is more useful than one that prints 98. +ARM64_SYSCALLS = { + 56: "openat", 57: "close", 61: "getdents64", 62: "lseek", 63: "read", + 64: "write", 66: "writev", 78: "readlinkat", 79: "newfstatat", 80: "fstat", + 93: "exit", 94: "exit_group", 98: "futex", 99: "set_robust_list", + 101: "nanosleep", 113: "clock_gettime", 117: "ptrace", 129: "kill", + 130: "tkill", 131: "tgkill", 134: "rt_sigaction", 135: "rt_sigprocmask", + 160: "uname", 167: "prctl", 169: "gettimeofday", 172: "getpid", + 173: "getppid", 174: "getuid", 175: "geteuid", 178: "gettid", 214: "brk", + 215: "munmap", 216: "mremap", 220: "clone", 221: "execve", 222: "mmap", + 226: "mprotect", 260: "wait4", 273: "set_robust_list", 276: "renameat", + 278: "getrandom", 280: "utimensat", 291: "statx", +} + +# The three groups that decide whether a userspace strategy is even applicable. +TERMINATION = {93, 94, 129, 130, 131} # exit / exit_group / kill / tkill / tgkill +EVASION = {117, 167, 226, 222, 220} # ptrace / prctl / mprotect / mmap / clone +DISCOVERY = {56, 57, 61, 62, 63, 64, 78, 79} # openat / close / getdents64 / read... + +# arm (32-bit) numbers, for the armeabi-v7a half of a dual-ABI APK. +ARM_SYSCALLS = { + 1: "exit", 3: "read", 4: "write", 5: "open", 6: "close", 20: "getpid", + 37: "kill", 38: "rename", 45: "brk", 54: "ioctl", 91: "munmap", + 120: "clone", 122: "uname", 125: "mprotect", 192: "mmap2", 199: "getuid", + 224: "gettid", 238: "tkill", 240: "futex", 248: "exit_group", 268: "statfs64", +} + +# How far back to look for the instruction that loads the syscall number. Real +# compilers put it 1-3 instructions before the svc (sometimes with an intermediate +# `orr`/`and`); 8 words is generous without letting an unrelated x8 write leak in. +LOOKBACK = 8 + + +def parse_elf_segments(data): + """Return [(file_offset, vaddr, filesz, exec_flag)] for PT_LOAD, or None if not ELF. + + Hand-walked rather than delegated to pyelftools on purpose: this kit's other + native scripts do the same, because a target's section header table is exactly + the thing a hardened binary forges (see `references/native-tamper-and-suicide.md`). + + Note the pf_x handling: it is *reported*, not used as a filter. Measured on a + real device, the ROM's own `libc.so` and `linker64` both carry `pf_x == 0` on + every PT_LOAD — so filtering to executable segments silently drops every hit on + this platform. The flag is printed per segment instead, so a run that mixes code + and data is visible rather than wrong. + """ + if len(data) < 64 or data[:4] != b"\x7fELF": + return None + is64 = data[4] == 2 + little = data[5] == 1 + if not little: + return None # no big-endian Android target in this kit's scope + if is64: + e_phoff, = struct.unpack_from("<Q", data, 0x20) + e_phentsize, e_phnum = struct.unpack_from("<HH", data, 0x36) + segs = [] + for i in range(e_phnum): + off = e_phoff + i * e_phentsize + if off + 56 > len(data): + break + # Elf64_Phdr: p_type u32, p_flags u32, p_offset u64 ... + p_type, p_flags = struct.unpack_from("<II", data, off) + p_offset, p_vaddr, _p_paddr, p_filesz = struct.unpack_from("<QQQQ", data, off + 8) + if p_type == 1: # PT_LOAD + segs.append((p_offset, p_vaddr, p_filesz, p_flags)) + return segs + e_phoff, = struct.unpack_from("<I", data, 0x1C) + e_phentsize, e_phnum = struct.unpack_from("<HH", data, 0x2A) + segs = [] + for i in range(e_phnum): + off = e_phoff + i * e_phentsize + if off + 32 > len(data): + break + p_type = struct.unpack_from("<I", data, off)[0] + p_offset, p_vaddr, _p_paddr, p_filesz, p_flags = struct.unpack_from("<IIIII", data, off + 4) + if p_type == 1: + segs.append((p_offset, p_vaddr, p_filesz, p_flags)) + return segs + + +def guess_arch(data): + """'arm64' | 'arm' from the ELF header, else None.""" + if len(data) >= 20 and data[:4] == b"\x7fELF": + machine = struct.unpack_from("<H", data, 18)[0] + return {0xB7: "arm64", 0x28: "arm"}.get(machine) + return None + + +def make_disassembler(arch, thumb): + if arch == "arm64": + md = Cs(CS_ARCH_ARM64, 0) # little-endian is the default + else: + mode = CS_MODE_THUMB if thumb else CS_MODE_ARM + md = Cs(CS_ARCH_ARM, mode) + # operand types (register vs immediate) are only populated in detail mode; + # without it capstone raises CS_ERR_DETAIL on `insn.operands`. + md.detail = True + return md + + +def load_immediate_into(md, arch, code, svc_off, reg_name): + """Walk back from an svc and return (value, mnemonic, off) for the write to reg_name. + + Best-effort by design: a number assembled at runtime returns (None, last_write, off) + so the site is still reported, with the instruction that produced the unknown. + """ + start = max(0, svc_off - LOOKBACK * 4) + window = code[start:svc_off] + last_write = None + for insn in md.disasm(window, start): + ops = insn.operands + if not ops: + continue + op0 = ops[0] + if op0.type != (ARM64_OP_REG if arch == "arm64" else ARM_OP_REG): + continue + try: + written = insn.reg_name(op0.reg) + except Exception: # pragma: no cover - capstone version drift + continue + if written != reg_name: + continue + last_write = (insn.mnemonic, insn.op_str, insn.address) + # `movz x8, #93` / `mov x8, #93` / `orr x8, xzr, #93` + for op in ops[1:]: + if op.type == (ARM64_OP_IMM if arch == "arm64" else ARM_OP_IMM): + return op.imm, insn.mnemonic, insn.address + return None, last_write, None + + +def disasm_context(data, md, arch, off, before=4, after=2): + """Disassemble a window around a site so a false positive can be told from a real one. + + The reason this exists: a byte scan for `svc` matches inside data. Measured on a + real shell library, 21 "svc sites" turned out to sit in the middle of `scvtf` / + `orr v25.4s` / `udf` byte soup with no call sequence anywhere near them — data, + not a syscall instruction. A site whose neighbours are `udf` or NEON arithmetic + is a byte-scan artefact; a real one sits at the end of a short sequence that + loads a value into the syscall-number register. + """ + lo = max(0, off - before * 4) + hi = min(len(data), off + 4 + after * 4) + out = [] + for insn in md.disasm(data[lo:hi], lo): + out.append("%s0x%x: %s %s" % ("-> " if insn.address == off else " ", + insn.address, insn.mnemonic, insn.op_str)) + return out + + +def looks_like_data(md, data, arch, off, before=6): + """True when the bytes just before the site are not a plausible instruction stream. + + Heuristic, and deliberately conservative: it only fires on the shapes that were + actually observed around byte-scan hits inside data (`udf`, `scvtf` with a wide + shift, NEON `mul v..s[..]`, an `orr w.., w.., #0xfffff...` mask). It never removes + a site -- it labels it, because "this is a real svc I do not understand" and + "this is not an instruction" need different follow-ups. + """ + if off < 16: + return False + window = data[max(0, off - before * 4):off] + for insn in md.disasm(window, max(0, off - before * 4)): + m = insn.mnemonic + if m == "udf": + return True + if m in ("scvtf", "ucvtf") and "#0x1" in insn.op_str: + return True + if m in ("mul", "mla") and "s[" in insn.op_str: + return True + if m == "orr" and "#0xfffff" in insn.op_str: + return True + return False + + +def scan_blob(data, arch, base, thumb=False, segs=None, code_only=False, context=0): + """Return (sites, per_segment_counts). `segs` limits the scan to PT_LOAD regions.""" + md = make_disassembler(arch, thumb) + reg = "x8" if arch == "arm64" else "r7" + table = ARM64_SYSCALLS if arch == "arm64" else ARM_SYSCALLS + + ranges = [] + if segs: + for idx, (p_offset, p_vaddr, p_filesz, p_flags) in enumerate(segs): + if code_only and not (p_flags & 1): + continue + end = min(len(data), p_offset + p_filesz) + if end > p_offset: + ranges.append((p_offset, end, p_vaddr - p_offset, idx, p_flags)) + else: + ranges.append((0, len(data), base, 0, None)) + + sites = [] + per_segment = {} + for lo, hi, vaddr_bias, idx, p_flags in ranges: + # Word-aligned walk only: an unaligned resync over the whole file produces + # garbage sites. Real svc sites sit on an instruction boundary, and the walk + # is done per 4 bytes so a straddling pattern cannot hide one. + n_here = 0 + off = lo + ((4 - lo % 4) % 4) + while off + 4 <= hi: + word = struct.unpack_from("<I", data, off)[0] + is_svc = (word & 0xFFE0001F) == 0xD4000001 if arch == "arm64" else ( + (word & 0x0F000000) == 0x0F000000) + if is_svc: + imm = word & 0xFFFF if arch == "arm64" else word & 0xFF + nr, mnemonic, woff = load_immediate_into(md, arch, data, off, reg) + site = { + "offset": off, + "vaddr": vaddr_bias + off, + "svc_imm": imm, + "nr": nr, + "name": table.get(nr, "nr=%s" % nr) if nr is not None else None, + "writer": mnemonic, + "writer_off": woff, + "segment": idx, + "segment_exec": None if p_flags is None else bool(p_flags & 1), + "likely_data": looks_like_data(md, data, arch, off), + } + if context: + site["context"] = disasm_context(data, md, arch, off, + before=context, after=2) + sites.append(site) + n_here += 1 + if arch == "arm64": + off += 4 + continue + off += 4 + per_segment[idx] = (n_here, p_flags, lo, hi) + return sites, per_segment + + +def group_of(site, arch): + nr = site.get("nr") + if nr is None: + return "unresolved" + table = TERMINATION if arch == "arm64" else {1, 37, 238, 248} + evasion = EVASION if arch == "arm64" else {117, 125, 120, 167} + discovery = DISCOVERY if arch == "arm64" else {3, 4, 5, 6, 78, 79} + if nr in table: + return "termination" + if nr in evasion: + return "evasion" + if nr in discovery: + return "discovery" + return "other" + + +def main(argv=None): + ap = argparse.ArgumentParser( + description="Scan an ELF (or raw capture) for inline `svc` sites and name the " + "syscall each one issues. Use it to decide whether a libc-level hook " + "can see a call at all: code that issues `svc #0` itself never enters " + "libc, so `Interceptor.replace` on exit/kill/mprotect cannot observe it.") + ap.add_argument("path", help="ELF file (scanned via PT_LOAD when possible) or raw capture") + ap.add_argument("--arch", choices=["arm64", "arm", "auto"], default="auto", + help="instruction set; 'auto' reads the ELF header (default), and is " + "required to be explicit for a raw capture") + ap.add_argument("--thumb", action="store_true", + help="arm only: decode as Thumb rather than A32") + ap.add_argument("--raw", action="store_true", + help="treat the input as an opaque memory capture even if it has ELF magic") + ap.add_argument("--base", default="0x0", + help="load address for a raw capture, so --json reports real vaddrs") + ap.add_argument("--json", action="store_true", help="emit the site list as JSON") + ap.add_argument("--code-only", action="store_true", + help="scan only PT_LOAD segments whose pf_x is set. Off by default: " + "measured device libc/linker64 carry pf_x=0 on every segment, so " + "this flag can drop every hit on a real ROM. Use it to shrink a " + "noisy run, and check the per-segment counts it reports.") + ap.add_argument("--context", type=int, default=0, metavar="N", + help="disassemble N instructions before each site and print them; the " + "way to tell a real svc from a byte-scan hit inside data") + ap.add_argument("--max-sites", type=int, default=0, metavar="N", + help="stop after N sites (0 = no limit); for a hostile blob where the " + "point is the histogram, not every offset") + args = ap.parse_args(argv) + + try: + with open(args.path, "rb") as fh: + data = fh.read() + except OSError as exc: + sys.stderr.write("cannot read %s: %s\n" % (args.path, exc)) + return 1 + + segs = None + arch = args.arch + if not args.raw: + if data[:4] != b"\x7fELF": + sys.stderr.write( + "%s is not an ELF; pass --raw --arch <arm64|arm> to scan it as a capture\n" + % args.path) + return 1 + detected = guess_arch(data) + if args.arch == "auto": + if detected is None: + sys.stderr.write("cannot determine the architecture from the ELF header; " + "pass --arch explicitly\n") + return 1 + arch = detected + segs = parse_elf_segments(data) + elif args.arch == "auto": + sys.stderr.write("--raw needs an explicit --arch (there is no header to read)\n") + return 1 + + base = int(args.base, 0) + sites, per_segment = scan_blob(data, arch, base, thumb=args.thumb, segs=segs, + code_only=args.code_only, context=args.context) + truncated = False + if args.max_sites and len(sites) > args.max_sites: + sites = sites[:args.max_sites] + truncated = True + + by_group = {} + by_name = {} + n_data = 0 + n_noload = 0 + for s in sites: + g = group_of(s, arch) + by_group[g] = by_group.get(g, 0) + 1 + key = s["name"] or "unresolved" + by_name[key] = by_name.get(key, 0) + 1 + if s.get("likely_data"): + n_data += 1 + if s["writer"] is None: + n_noload += 1 + + if args.json: + print(json.dumps({ + "path": os.path.abspath(args.path), + "arch": arch, + "code_only": args.code_only, + "segments": [ + {"index": i, "exec": None if fl is None else bool(fl & 1), + "file_range": [lo, hi], "svc_sites": c} + for i, (c, fl, lo, hi) in sorted(per_segment.items()) + ], + "sites": sites, + "by_group": by_group, + "by_name": by_name, + "likely_data_sites": n_data, + "no_load_sites": n_noload, + "truncated": truncated, + }, indent=2)) + return 0 + + print("== svc scan: %s (%s) ==" % (os.path.basename(args.path), arch)) + if per_segment and segs: + print("segments (pf_x is reported from the program header, not used as a filter):") + for i, (c, fl, lo, hi) in sorted(per_segment.items()): + print(" seg%-2d off=0x%-8x size=0x%-8x pf_x=%d svc=%d" + % (i, lo, hi - lo, (fl & 1) if fl is not None else -1, c)) + skipped = len(segs) - len(per_segment) + if skipped: + print(" (%d PT_LOAD segment(s) skipped by --code-only)" % skipped) + print("sites: %d%s" % (len(sites), " (truncated by --max-sites)" if truncated else "")) + if not sites: + print("no inline `svc` instruction found in the scanned ranges.") + print("reading: this file does not issue syscalls directly in the scanned bytes -- " + "for the termination group that means a libc-level hook is not structurally " + "excluded (see references/detection-and-anti-analysis.md).") + return 0 + + fmt = " %-10s %-10s %-6s %-12s %-5s %s" + print(fmt % ("offset", "vaddr", "svc#", "syscall", "nr", "note")) + for s in sites: + note = [] + if s.get("likely_data"): + note.append("neighbours are udf/NEON soup") + if s["writer"] is None: + note.append("no syscall-number load nearby") + print(fmt % ("0x%x" % s["offset"], "0x%x" % s["vaddr"], s["svc_imm"], + s["name"] or "?", s["nr"] if s["nr"] is not None else "?", + "; ".join(note))) + for line in s.get("context", ()): + print(" %s" % line) + + print("\nhistogram by group: %s" % ", ".join( + "%s=%d" % (k, by_group[k]) for k in sorted(by_group))) + print("histogram by syscall: %s" % ", ".join( + "%s=%d" % (k, by_name[k]) for k in sorted(by_name, key=lambda x: -by_name[x])[:12])) + + term = by_group.get("termination", 0) + unresolved = by_group.get("unresolved", 0) + print("\nreading:") + if n_data or n_noload: + print(" %d site(s) carry a data-artefact flag (%d udf/NEON neighbours, %d with no " + "syscall-number load nearby). Both mean `likely not an instruction`: verify " + "with --context before treating any of them as a call, because a byte scan " + "for `svc` matches inside data." % (max(n_data, n_noload), n_data, n_noload)) + if term: + print(" %d termination site(s) (exit/exit_group/kill/tkill/tgkill): a libc-level " + "hook CANNOT observe these. Prefer a static NOP at the site over an " + "Interceptor.replace on libc, and check the site's own writer instruction " + "before editing it." % term) + else: + print(" no termination site: your libc-level hook is not structurally excluded, so " + "a missing exit event is a finding about the hook, not about the target.") + if unresolved: + print(" %d site(s) whose number is not a literal: either computed at runtime " + "(`mov x8, x0`, `sxtw x8, w1`) or hidden behind a table. Report them as " + "unknown -- do not assume a syscall number. A module whose svc sites are " + "*all* unresolved is consistent with a deliberate indirection layer, but " + "this scan cannot prove that; say `unresolved` in the record." % unresolved) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tests/benchmark.md b/tests/benchmark.md index 1d8688d..5b6886b 100644 --- a/tests/benchmark.md +++ b/tests/benchmark.md @@ -36,6 +36,8 @@ reserves `observed` for a claim with an exact command and its output behind it): | B9 | Split APK / App Bundle handling | two real split sets pulled from a device (8 members = base + 7 density splits each carrying its own `resources.arsc`; 9 members = base + ABI split with only `lib/` + 7 density splits with 40-byte placeholder tables) | `repack.py` (split mode), `split-apk.md` | Both branches of the decision ran on real sets: after `resign` every member shares one certificate with v1+v2+v3 all true; `merge` **correctly refused** the set whose members carry their own `resources.arsc` (exit 1, before writing anything) and **succeeded** on the code/native-only set, folding 2 `.so` files into `base`. The single-APK legacy path still passes its regression (`--dexdir` replacement and a zero-change round trip). Signing has no `uber-apk-signer` on this host; the measured call is `zipalign -p -f 4` then `apksigner.jar --v1 --v2 --v3`, and the order cannot be reversed (`[-124]`). `adb install-multiple` and launch confirmation were **not** executed | observed (analyze/resign/merge) / unverified (install, launch) | `EXTENSION-split-apk.md` | | B10 | Kernel-side syscall answer forging | KernelSU/APatch module template, host toolchain, device kernel | `kernel-and-environment-hardening.md`, `kernelsu_syscall_mask.py` | The generator produced an 11-file userspace module skeleton plus KPM/LKM/eBPF templates and `verify` reports 0 problems — but **no kernel-side artefact was compiled or loaded**: the `<DEVICE>` kernel is 4.14.186 (eBPF needs 5.10+), the host has no `aarch64-none-elf-gcc` / `make` / `ndk-build`, and there is no kernel source. Documentation correction with teeth: **an ordinary KernelSU module is a userspace module and structurally cannot change a syscall return value** | observed (generation) / unverified (kernel side) | `EXTENSION-kernel-weapons.md` | | B11 | JNI dynamic registration discovery (`RegisterNatives` with no `Java_*` symbol) | a real system library + CyReverse `plugin/` | `java2c-and-jni-sinking.md`, `java2c_probe.py` | A real dynamically-registering library contains **zero `RegisterNatives` symbols** (the C++ `jni.h` inlines the call), so symbol-based discovery fails structurally. The combination that did fire on a real library is "**exports `JNI_OnLoad` and zero `Java_*`**" | observed | `EXTENSION-java2c.md` | +| B12 | Dart AOT string-table format across ABIs (the premise any string patch rests on) | a locally supplied hardened Flutter APK (360 shell + AOT payload, both ABIs; uncommitted, `tools/_work/dart/`) | `dart-aot.md` §7/§10, `scripts/dart_pool_strings.py` | **Two claims tested, both were wrong before this row.** (a) arm64 packed `[0x80\|(len<<1)][payload]` is confirmed — `dart_pool_strings.py` keeps 4,980 chained entries and the tag byte equals `0x80\|(len<<1)` at every literal checked (`AdmobBanner` 11 chars → `0x96`, `showSplashAd` and `AdPgl_Splash` 12 → `0x98`). (b) The armv7 form "`[len*2 u32le][UTF-16LE]`" is **refuted**: the same extractor keeps **0** entries on the 32-bit snapshot, and the byte layer shows why — each 4-byte-aligned record reads `[header u32][byte-count u32le][UTF-8 payload]`, with the two adjacent literals `SystemChrome.setEnabledSystemUIMode` (36) + `SystemChrome.setSystemUIOverlayStyle` (34) carrying lengths `0x46` (70) and `0x48` (72) that exactly equal their byte lengths. The extractor's zero is a format mismatch, not an empty string table. A third claim, "the engine banner gives `x.y.z (stable)`", is also refuted on this target: the banner is 66 bytes of `0.0.1` with a blank build id. Full detail and the raw probes: `EXTENSION-dart-aot-formats.md` | observed (both formats, banner) / unverified (an end-to-end device patch — no repack+install was run) | `EXTENSION-dart-aot-formats.md` | +| B13 | Naming the check that fires, and the environment it reads (anti-instrumentation triage) | MASTG `owasp.mstg.uncrackable3` (public target) for the three arms; device `libc.so` and a hardened shell library for `svc_scan.py` | `anti_detect_probe.js`, `detection-and-anti-analysis.md` Step 3, `svc_scan.py` | **The check was named, and the asymmetry is the finding.** Three arms on one target with a single variable: control (`spawn` + `resume`, no script) ends `alive_at_end=True`; the arm that also loads the observer probe ends `alive_at_end=False`, with the stream showing `strstr("frida")` at t=303 ms and, in an earlier same-shape arm, the kill itself at t=352 ms — process gone with **no tombstone, no crash and no ANR record**. The probe's own self-report on the same arm: `TracerPid=0` while the target's `/proc/self/maps` carries **4 frida-named mappings**, so a `TracerPid`-only detector passes and a maps/thread-name detector does not. Reproducibility is honestly bounded: 3 same-shape arms, 1 delivered a full sequence (a ≈300 ms self-destruct leaves a sub-second window). Separately, `svc_scan.py`'s two independent decoders — a hand-written word scan and capstone — produced an **identical 214-site set** (set difference empty), and a structural criterion fell out: the device `libc.so`'s 4 termination sites are libc's own exports (so a libc-level hook is structurally able to see them), while the hardened shell library's 21 `svc` sites are **all data** (neighbours are `udf`/NEON filler) with no termination site | observed (the arms, the named check, decode agreement) / unverified (dual-dumper byte agreement — no second producer existed; a dump from the hardened sample) | `EXTENSION-detection-pipeline.md` §1, §4, §5 | ## How to read a row