feat(localdns): add localDNS metrics exporter

[saewoni]

Adds CPU and memory metrics for localdns.service using systemd accounting and socket activation for efficient, zero-overhead monitoring.

Also exports IP addresses configured for forward plugins for kubedns and vnetdns overrides.

Implementation:

• localdns_exporter.sh: Reads pre-generated .prom files written by localdns.sh (avoids D-Bus permission issues under DynamicUser)
• localdns-exporter.socket: Socket activation on port 9353 with MaxConnections=10
• localdns-exporter@.service: Instantiated service per connection, security hardened (DynamicUser=yes, ProtectSystem=strict,
• NoNewPrivileges=yes)
• localdns.sh: Generates resource metrics on each watchdog tick (~6s) and forward IP metrics on corefile update
• Integrated into all VHD builders (Ubuntu, Mariner, Flatcar, ACL, all arches)
• Backward compatible: CSE skips setup on older VHDs via node label gating (kubernetes.azure.com/localdns-exporter)

Metrics exposed:

• localdns_service_status (gauge) — CoreDNS process liveness
• localdns_cpu_usage_seconds_total (counter)
• localdns_memory_usage_bytes (gauge)
• localdns_metrics_last_update_timestamp_seconds (gauge) — staleness detection
• localdns_vnetdns_forward_info (gauge) — VnetDNS forward IP with block and status labels
• localdns_kubedns_forward_info (gauge) — KubeDNS forward IP with block and status labels

Test coverage:

• ShellSpec unit tests for exporter HTTP routing, export_resource_metrics, and forward IP prom generation
• E2e validation script covering socket activation, metrics format, security hardening, and node label gating
• VHD content test validates exporter files present with correct permissions

What this PR does / why we need it:

Which issue(s) this PR fixes:

Fixes #

[Copilot]

Pull request overview

This PR adds a lightweight Prometheus-compatible metrics exporter for the localdns.service using systemd socket activation. The exporter exposes CPU and memory metrics on port 9353 with zero overhead when not being scraped, making it suitable for production monitoring.

Changes:

• Added localdns_exporter.sh bash script that queries systemd accounting metrics and formats them as Prometheus metrics
• Added systemd socket (localdns-exporter.socket) and service (localdns-exporter@.service) units for on-demand activation
• Integrated the exporter into all Linux VHD builds (Ubuntu, Mariner, Flatcar, ARM64/x64)
• Added VHD content validation tests and a standalone test script

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
parts/linux/cloud-init/artifacts/localdns_exporter.sh	Bash script that scrapes systemd metrics (CPUUsageNSec, MemoryCurrent) and outputs Prometheus-formatted HTTP response
parts/linux/cloud-init/artifacts/localdns-exporter@.service	Systemd template service for per-connection worker instances with security hardening
parts/linux/cloud-init/artifacts/localdns-exporter.socket	Systemd socket unit listening on port 9353 with Accept=yes for on-demand activation
vhdbuilder/packer/vhd-image-builder-*.json	Added file provisioners to copy exporter artifacts to all Linux VHD variants
vhdbuilder/packer/packer_source.sh	Added file copying logic and systemctl enable command for the socket
vhdbuilder/packer/imagecustomizer/azlosguard/azlosguard.yml	Added exporter files to OSGuard VHD build but missing socket enablement
vhdbuilder/packer/test/linux-vhd-content-test.sh	Added validation for exporter files and permissions
e2e/test-localdns-exporter.sh	Standalone test script for manual validation

[djsly]

what is this for ?

[jingwenw15]

these enable cgroup accounting for the service so systemd actually tracks cpu time and memory usage. without them systemctl show localdns.service --property=CPUUsageNSec,MemoryCurrent returns [not set], and export_resource_metrics() needs these to write the cpu/mem .prom metrics

[djsly]

why not pull the metrics from directly, going over systemctl seems overkilled

systemctl
    ↓
DBus client
    ↓
dbus-daemon
    ↓
PID 1 (systemd)
    ↓
unit object lookup
    ↓
cgroup accounting read

VS simply reading from the sys/fs

/sys/fs/cgroup/system.slice/foo.service/
    cpu.stat
    memory.current

[djsly]

[saewoni]

The commit itself is fine for what it does. But there's one issue worth calling out:

When timeout 3 kills systemctl show, the exit code is 124 (timeout), which gets caught by || echo "", so show_output becomes empty string. Then raw_cpu and raw_mem both become empty, fall into the [not set] handler, and get set to 0. This means on a D-Bus hang, you'll report:

localdns_cpu_usage_seconds_total 0.000000000
localdns_memory_usage_bytes 0
That's a silent data loss — metrics silently drop to zero instead of retaining the previous value or signaling an error. If someone has a dashboard or alert on these metrics, a D-Bus hang would show as a sudden drop to zero CPU and zero memory, which could be mistaken for the service being idle or restarted.

A couple of options to handle this better:

Skip the update entirely on timeout — don't overwrite the .prom file, so the previous values persist. The localdns_metrics_last_update_timestamp_seconds would go stale, which consumers can alert on (time() - timestamp > 120).
Log a warning so it's visible in journalctl that a timeout occurred.
Something like:

show_output=$(timeout 3 systemctl show "$unit" --property=CPUUsageNSec,MemoryCurrent 2>/dev/null)
if [ $? -ne 0 ]; then
    echo "WARNING: systemctl show timed out or failed, skipping resource metrics update"
    return 0
fi

[jingwenw15]

fixed to read directly from service

[Copilot]

Review Comments

1. Metric name _mb violates Prometheus conventions...

All four issues have been addressed in earlier commits:

1. _mb → _bytes: Metric is now localdns_memory_usage_bytes (raw bytes, no conversion) — Prometheus base-unit convention followed.
2. CPU precision: Using %.9f for nanosecond-level precision — no step-function artifacts in rate().
3. Connection: close: Added to both 200 and 404 HTTP responses in localdns_exporter.sh.
4. Staleness detection: The localdns_metrics_last_update_timestamp_seconds gauge is written on every export_resource_metrics() call; scrapers can detect stale data by comparing this timestamp against now().

[djsly]

poll the metrics from the cgroup and LTGM