forkless/strix
1
0
Fork 0
mirror of https://github.com/usestrix/strix.git synced 2026-08-16 09:26:39 +02:00
Code Issues Packages Projects Releases Wiki Activity

docs(skills): add research-backed katana output-reduction flags

Browse Source 
Per projectdiscovery katana docs, add the flags that actually bound
crawl output size and a reduce-then-delete workflow:
- -mdp (max-domain-pages; default is unlimited), -fsu (filter-similar),
  -fs scope, -f url (URL-only), -or/-ob (omit raw/body), -mrs.
- Baseline now includes -mdp 2000 -fsu; new 'Keeping output small'
  section: bound scope/volume, shrink records, distil then delete raw
  crawls.
This commit is contained in:
Ahmed Allam
2026-07-16 04:47:56 -07:00
committed by Ahmed Allam
parent 98990bae45
commit 89ee7b9e5e
1 changed files with 15 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files
strix/skills/tooling/katana.md
+15 -6
View File
@@ -24,7 +24,15 @@ High-signal flags:
- `-p, -parallelism <n>` concurrent input targets - `-p, -parallelism <n>` concurrent input targets
- `-rl, -rate-limit <n>` request rate limit - `-rl, -rate-limit <n>` request rate limit
- `-timeout <seconds>` request timeout - `-timeout <seconds>` request timeout
- `-ct, -crawl-duration <s|m|h|d>` maximum time to crawl the target
- `-retry <n>` retry count - `-retry <n>` retry count
- `-mdp, -max-domain-pages <n>` cap pages crawled per domain (default: unlimited)
- `-fsu, -filter-similar` collapse similar URLs (e.g. /users/123 and /users/456)
- `-fs, -field-scope <dn|rdn|fqdn|regex>` crawl scope (default `rdn` = root domain + ALL subdomains)
- `-f, -field <url|path|...>` emit only one field (e.g. `-f url` for a plain URL list)
- `-or, -omit-raw` omit raw request/response from JSONL output
- `-ob, -omit-body` omit response body from JSONL output
- `-mrs, -max-response-size <bytes>` cap per-response bytes read (default 4194304)
- `-ef, -extension-filter <list>` extension exclusions - `-ef, -extension-filter <list>` extension exclusions
- `-tlsi, -tls-impersonate` experimental JA3/TLS impersonation - `-tlsi, -tls-impersonate` experimental JA3/TLS impersonation
- `-hl, -headless` enable hybrid headless crawling - `-hl, -headless` enable hybrid headless crawling
@@ -37,7 +45,7 @@ High-signal flags:
- `-silent`, `-j, -jsonl`, `-o <file>` output controls - `-silent`, `-j, -jsonl`, `-o <file>` output controls
Agent-safe baseline for automation: Agent-safe baseline for automation:
`mkdir -p crawl && katana -u https://target.tld -d 3 -ct 10m -jc -kf robotstxt -c 10 -p 10 -rl 50 -timeout 10 -retry 1 -ef png,jpg,jpeg,gif,svg,css,woff,woff2,ttf,eot,map -silent -j -o crawl/katana.jsonl` `mkdir -p crawl && katana -u https://target.tld -d 3 -ct 10m -mdp 2000 -fsu -jc -kf robotstxt -c 10 -p 10 -rl 50 -timeout 10 -retry 1 -ef png,jpg,jpeg,gif,svg,css,woff,woff2,ttf,eot,map -silent -j -o crawl/katana.jsonl`
Common patterns: Common patterns:
- Fast crawl baseline: - Fast crawl baseline:
@@ -59,11 +67,12 @@ Critical correctness rules:
- For `-kf`, keep depth at least `-d 3` so known files are fully covered. - For `-kf`, keep depth at least `-d 3` so known files are fully covered.
- If writing to a file, ensure parent directory exists before `-o`. - If writing to a file, ensure parent directory exists before `-o`.
Keeping output manageable: Keeping output small (katana has NO default page cap, so plan for volume):
- katana does not cap its own output size, so bound crawls with `-ct` (crawl-duration) and `-d` (depth); on large sites an unbounded `-jsl` / `-kf all` deep crawl can grow very large. - Bound scope and volume: `-fs fqdn` (or `-cs`/`-cos` regex) so the crawl doesn't wander across every subdomain, `-mdp <n>` to cap pages per domain, `-fsu` to collapse near-identical URLs, and `-ct`/`-d` to bound time and depth.
- Reserve deep JS crawling (`-jsl`, `-kf all`, higher `-d`) for a specific narrowed target rather than broad scopes. - Shrink each record: default JSONL is verbose. If you only need endpoints, emit a plain URL list with `-f url` instead of `-j`. If you need JSONL, drop the heavy parts with `-or` (omit raw) and `-ob` (omit body), and lower `-mrs` to cap per-response bytes.
- After a crawl, glance at output size (`du -sh <out>`); if it looks outsized for the scope, tighten `-d`/`-ct`/`-ef` or split per host. - Reserve `-jsl` / `-kf all` / higher `-d` for a specific narrowed target — they multiply output fast on large sites.
- For JSONL output, extract and dedupe URLs before use (`jq -r '.url' <out.jsonl> | sort -u > urls.txt`), then remove the raw `.jsonl` once you've preserved any metadata you still need. - Reduce, then delete: once the crawl finishes, extract just what you need (e.g. `katana ... -f url -o urls.txt` or `sort -u` a URL list, or a short note of interesting paths) and remove the raw crawl file/dir. Don't keep large raw crawls around after you've distilled them.
- Sanity-check size (`du -sh <out>`); if it's outsized for the scope, tighten `-fs`/`-mdp`/`-fsu`/`-d`/`-ct` and re-run rather than keeping it.
Usage rules: Usage rules:
- Keep `-d`, `-c`, `-p`, and `-rl` explicit for reproducible runs. - Keep `-d`, `-c`, `-p`, and `-rl` explicit for reproducible runs.