Skip to content

Prometheus/OpenMetrics endpoint #1415

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
DL6ER wants to merge 1 commit into
base: master
Choose a base branch
from
Draft

Prometheus/OpenMetrics endpoint #1415

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
258 changes: 258 additions & 0 deletions docs/api/prometheus.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,258 @@
Pi-hole can expose its statistics in the [Prometheus text exposition format](https://prometheus.io/docs/instrumenting/exposition_formats/) at `GET /api/metrics`. This allows Prometheus (or any other OpenMetrics-compatible collector) to scrape your Pi-hole directly, without running a separate exporter that translates the JSON API into metrics.

The endpoint is **disabled by default** and only becomes available after you have generated a scrape token (see below).

## How authentication works here

The metrics endpoint does *not* use the session-based authentication described in [Authentication](auth.md). Instead, it uses a dedicated bearer token. There are two reasons for this:

1. A scraper runs permanently and would otherwise occupy one of the [limited number of concurrent sessions](auth.md#limited-number-of-concurrent-sessions) - or would have to log in every few seconds.
2. The endpoint stays protected even on installations without a web interface password, where the remaining API is accessible without authentication.

FTL never stores the token itself, only its SHA-256 hash in `webserver.api.prometheus.token`. The raw token is returned exactly once, when you generate it.

<!-- markdownlint-disable code-block-style -->
!!! info "Why SHA-256 and not the balloon hash used for passwords?"
The token is a 256-bit random value generated by Pi-hole itself, so it cannot be guessed or brute-forced. A deliberately slow hash would only burn CPU cycles on every single scrape while a fast digest already stores a preimage-resistant value at rest.
<!-- markdownlint-enable code-block-style -->

## Enabling the endpoint

The quickest way is the command line:

```bash
sudo pihole-FTL --prometheus-token
```

```text
New Prometheus scrape token (shown only once):

7ua0rHPZixX6B3bTa5o4Dv08iyEIm/2q9qgLrF9MNVw=

Stored hash in webserver.api.prometheus.token
```

The token is active immediately, a running `pihole-FTL` picks up the change without a restart.

<!-- markdownlint-disable code-block-style -->
??? info "Using the API instead"

If you prefer to enable the endpoint remotely, e.g., when automating the setup of several Pi-holes, generate the token with `POST /api/auth/prometheus`. This request uses the regular API authentication, see [Authentication](auth.md): a valid session ID (`SID`), unless your Pi-hole has no web interface password, in which case no authentication is needed.

=== "bash / cURL"

```bash
curl -k -X POST "https://pi.hole/api/auth/prometheus" -H "X-FTL-SID: vFA+EP4MQ5JJvJg+3Q2Jnw="
```

=== "Python 3"

```python
import requests

url = "https://pi.hole/api/auth/prometheus"
headers = {"X-FTL-SID": "vFA+EP4MQ5JJvJg+3Q2Jnw="}

response = requests.request("POST", url, headers=headers, verify=False)

print(response.text)
```

The reply repeats the token exactly once:

```json
{
"prometheus": {
"token": "7ua0rHPZixX6B3bTa5o4Dv08iyEIm/2q9qgLrF9MNVw=",
"hash": "9e99d5834ce965262b7eefd9694aad95f0dd24ca0214be659bc06570008d2096"
},
"took": 0.003
}
```

`token` is the raw token you configure in your scraper. `hash` is the value FTL has just stored in `webserver.api.prometheus.token` and is only returned for reference.
<!-- markdownlint-enable code-block-style -->

<!-- markdownlint-disable code-block-style -->
!!! warning "The token is shown only once"
FTL stores only the hash, so the raw token cannot be retrieved later. Store it in a safe place right away. Generating a new token replaces the stored hash and thereby immediately invalidates the previous token.
<!-- markdownlint-enable code-block-style -->

Both ways write the new hash to `/etc/pihole/pihole.toml` automatically. If your configuration is in read-only mode (`misc.readOnly = true`), the CLI refuses the change altogether, while the API still returns a working token that is, however, silently lost on the next restart of `pihole-FTL`.

## Scraping the endpoint

The scraper sends the raw token in an `Authorization: Bearer <token>` header:

<!-- markdownlint-disable code-block-style -->
???+ example "Scrape the metrics endpoint"

=== "bash / cURL"

```bash
curl -k "https://pi.hole/api/metrics" -H "Authorization: Bearer 7ua0rHPZixX6B3bTa5o4Dv08iyEIm/2q9qgLrF9MNVw="
```

=== "Python 3"

```python
import requests

url = "https://pi.hole/api/metrics"
headers = {"Authorization": "Bearer 7ua0rHPZixX6B3bTa5o4Dv08iyEIm/2q9qgLrF9MNVw="}

response = requests.request("GET", url, headers=headers, verify=False)

print(response.text)
```

???+ success "Success response"

Response code: `HTTP/1.1 200 OK`

Content type: `text/plain; version=0.0.4; charset=utf-8`

```text
# HELP pihole_queries Number of DNS queries in FTL's history window (webserver.api.maxHistory)
# TYPE pihole_queries gauge
pihole_queries 8412
# HELP pihole_queries_blocked Number of blocked DNS queries in FTL's history window
# TYPE pihole_queries_blocked gauge
pihole_queries_blocked 1337
...
```
<!-- markdownlint-enable code-block-style -->

The token is accepted **only** in this header. It is deliberately not accepted as a query string parameter (where it would end up in access logs and proxy logs) and not as a cookie (which would reintroduce the CSRF attack surface the session cookie is protected against).

Response code | Meaning
--------------|------------------------------------------------------------
`200` | Success, the body contains the metrics
`401` | The `Authorization` header is missing or the token is wrong
`404` | The endpoint is disabled because no token is configured

The response is identical for a missing and for an incorrect token, and tokens are compared in constant time. As the token is a 256-bit random value, no rate limiting is applied to this endpoint.

## Disabling the endpoint

Revoking the token disables the endpoint again - `GET /api/metrics` then returns `404 Not Found`, just as on an installation that never enabled it:

```bash
sudo pihole-FTL --prometheus-token revoke
```

This is the same as setting `webserver.api.prometheus.token` to an empty string, which you can also do through the API, the web interface, or by editing `/etc/pihole/pihole.toml` directly:

<!-- markdownlint-disable code-block-style -->
??? example "Disable the metrics endpoint through the API"

```bash
curl -k -X PATCH "https://pi.hole/api/config/webserver/api/prometheus/token" \
-H "X-FTL-SID: vFA+EP4MQ5JJvJg+3Q2Jnw=" \
--data '{"config":{"webserver":{"api":{"prometheus":{"token":""}}}}}'
```
<!-- markdownlint-enable code-block-style -->

Generating a new token invalidates the old one, so there is no need to revoke a token before replacing it.

## Configuration options

Setting | Default | Description
--------------------------------------------|---------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
`webserver.api.prometheus.token` | `""` | SHA-256 hash of the scrape token. Empty disables the endpoint. Set by `pihole-FTL --prometheus-token` or `POST /api/auth/prometheus`, see above. Reading it back through the API or the CLI yields `********`, as the value is write-only.
`webserver.api.prometheus.perEntityMetrics` | `false` | Whether per-domain and per-client time series are exported in addition to the aggregate metrics.
`webserver.api.prometheus.topN` | `100` | Upper limit for the number of top domains and top clients exported when `perEntityMetrics` is enabled (`0` - `1000`). `0` disables the per-entity series.

## Exported metrics

### Query statistics

Metric | Type | Description
---------------------------|-------|-------------------------------------------------------------------------------------
`pihole_queries` | gauge | Number of DNS queries in FTL's history window
`pihole_queries_blocked` | gauge | Number of blocked DNS queries in FTL's history window
`pihole_queries_forwarded` | gauge | Number of forwarded DNS queries in FTL's history window
`pihole_queries_cached` | gauge | Number of cached DNS queries in FTL's history window
`pihole_query_frequency` | gauge | Queries per second (rolling average)
`pihole_queries_by_type` | gauge | Queries by record type, label `type` (e.g., `A`, `AAAA`, `HTTPS`)
`pihole_queries_by_status` | gauge | Queries by processing status, label `status` (e.g., `GRAVITY`, `FORWARDED`, `CACHE`)
`pihole_queries_by_reply` | gauge | Queries by reply type, label `reply` (e.g., `NODATA`, `NXDOMAIN`, `IP`)

<!-- markdownlint-disable code-block-style -->
!!! info "Why are the query totals gauges?"
These values describe the queries currently held in FTL's in-memory history, which spans the last `webserver.api.maxHistory` seconds (24 hours by default). Garbage collection decrements them as queries age out, so they can go down as well as up and are *not* monotonically increasing counters. Applying `rate()` or `increase()` to them yields meaningless results - use the `counter` metrics from the [DNS](#dns) and [DHCP](#dhcp) sections below for that.
<!-- markdownlint-enable code-block-style -->

### Clients, domains and gravity

Metric | Type | Description
-----------------------------------------------|-------|--------------------------------------------------
`pihole_clients_total` | gauge | Number of known clients
`pihole_clients_active` | gauge | Number of clients active within the last 24 hours
`pihole_domains_total` | gauge | Number of unique domains seen
`pihole_upstreams_total` | gauge | Number of known upstream destinations
`pihole_gravity_domains` | gauge | Number of domains on the gravity (block) list
`pihole_gravity_last_update_timestamp_seconds` | gauge | Unix time of the last gravity update

### DNS

Metric | Type | Description
----------------------------------|---------|------------------------------------------------------------------------------------------------
`pihole_dns_cache_size` | gauge | Number of entries in the DNS cache
`pihole_dns_cache_inserted_total` | counter | Number of entries inserted into the DNS cache
`pihole_dns_cache_evicted_total` | counter | Number of live cache entries evicted before their TTL
`pihole_dns_cache_expired_total` | counter | Number of expired DNS cache entries
`pihole_dns_cache_immortal` | gauge | Number of immortal DNS cache entries
`pihole_dns_replies_total` | counter | DNS replies by source, label `source` (`local`, `forwarded`, `optimized`, `unanswered`, `auth`)

### DHCP

Metric | Type | Description
-----------------------------|---------|-----------------------------------------------------------------------------------------------------------------------------------------------
`pihole_dhcp_messages_total` | counter | DHCP messages by type, label `type` (`ack`, `decline`, `discover`, `inform`, `nak`, `offer`, `release`, `request`, `noanswer`, `bootp`, `pxe`)
`pihole_dhcp_leases` | gauge | DHCP leases, labels `family` (`ipv4`, `ipv6`) and `operation` (`allocated`, `pruned`)

These counters are also exported when the DHCP server is disabled - they simply stay at zero.

### Per-domain and per-client metrics (optional)

When `webserver.api.prometheus.perEntityMetrics` is enabled, two additional metrics are exported:

Metric | Type | Description
----------------------------|-------|--------------------------------------------------------------
`pihole_top_domain_queries` | gauge | Permitted queries for the most active domains, label `domain`
`pihole_top_client_queries` | gauge | Queries of the most active clients, labels `ip` and `name`

The client IP is used as the primary label because it uniquely identifies a client - two clients sharing a hostname would otherwise produce duplicate time series, which makes Prometheus reject the entire scrape. The hostname is added as a secondary label for readability.

<!-- markdownlint-disable code-block-style -->
!!! warning "Privacy and cardinality"
These series reveal which domains are queried in your network and which clients are active, which is why they are disabled by default. They honor your configured [privacy level](../ftldns/privacylevels.md) as well as the `webserver.api.excludeDomains` and `webserver.api.excludeClients` filters, exactly like the corresponding JSON API endpoints do.

Keep in mind that every domain and client is a separate time series in your Prometheus database. `webserver.api.prometheus.topN` limits how many of them are exported, but even the default of 100 domains plus 100 clients adds up over time as the top lists change.
<!-- markdownlint-enable code-block-style -->

## Prometheus configuration example

<!-- markdownlint-disable code-block-style -->
???+ example "`prometheus.yml`"

```yaml
scrape_configs:
- job_name: 'pi-hole'
metrics_path: '/api/metrics'
scheme: 'https'
authorization:
type: 'Bearer'
credentials: '7ua0rHPZixX6B3bTa5o4Dv08iyEIm/2q9qgLrF9MNVw='
Comment thread
DL6ER marked this conversation as resolved.
tls_config:
# Not needed if your Pi-hole uses a certificate your Prometheus trusts
insecure_skip_verify: true
static_configs:
- targets: ['pi.hole']
```
<!-- markdownlint-enable code-block-style -->

We recommend scraping over HTTPS so the token is not sent in the clear. Pi-hole's self-signed certificate is not trusted by default, so either add the certificate authority to the trust store of the machine running Prometheus (see [TLS/SSL](tls.md)) or skip the verification as shown above. Prometheus can also read the token from a file instead (`credentials_file`), which keeps it out of your configuration file.

{!abbreviations.md!}
1 change: 1 addition & 0 deletions mkdocs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -132,6 +132,7 @@ nav:
- 'Pi-hole API':
- 'Pi-hole API': api/index.md
- 'Authentication': api/auth.md
- 'Metrics': api/prometheus.md
- 'TLS/SSL': api/tls.md
- 'FTLDNS':
- 'FTLDNS': ftldns/index.md
Expand Down