Complete Grafana Monitoring Setup

Alloy, Grafana, Mimir, Loki, Pyroscope, Tempo, Minio, NGinx

NOTE: As of 5/18/2026

This has progressed into its own unique beast. I have not seen an alloy before on github that can do all of the things this one can.

Done / System Setup

• Windows Metrics and Logs
• Core stack images updated to current stable tags (Alloy, Loki, Tempo, Mimir, Pyroscope, Grafana, Nginx, MinIO) ✅ 2026-05-18
• ntopng service integrated with Alloy scrape labels + nginx gateway route ✅ 2026-05-18
• AI monitoring module and GPU exporter service profile (gpu) added ✅ 2026-05-18
• OPNsense monitoring module and sample Grafana dashboard added ✅ 2026-05-18
• Dashboard updates, most dashboards are fully functional now, this will depend on your variables and names of course ✅ 2024-06-18
• Identified how to get loki to parse logs properly. The only issue is they vary so greatly from app to app it requires custom regex, for example I now can parse arr logs (lidarr, radarr, etc). ✅ 2024-06-18
• Integrate new Explore Logs / Grafana 11 Preview ✅ 2024-06-18
• Update collector to Alloy
• Alloy configured as a total system collector
  • Docker Metrics (Alloy cAdvisor built in)
  • Docker Logs (Alloy built in)
  • Traefik Metrics (Traefik built in / Alloy Prom Scrape)
  • Local System Metrics (node exporter)
  • Local System Logs (journald module)
• Gateway for Alloy / Grafana
• This was needed before I split Loki and Mimir into read/write mode
• Get Pyro and Tempo to work properly
• Set up On Call to work with Slack
• Get alerts and rules to pull into the stack properly
• Redo configs to utilize source/target in docker, this is important for switching configurations as well as showing the important configurations to those who are trying to learn the stack
• ALOT more, this was just what was on the top of my head

Still To Do

• Tune AI and OPNsense sample dashboards for local metric names
• Add a hardened production auth layer for ntopng/OPNsense endpoints
• Update this readme / howto, there are soooooo many variables, that explaining how it works can be tough
• Further refine Alloy to label properly, drop what it doesn't need, etc
• Further refine the "module" philosophy of Alloy, I want to make it so simple that someone could download this, and with a few clicks be up and running with what they need
• Fix Dashboards, in mono mode, dashboards do not work out of the box for Loki and some others, mixins also does not seem to fix the issues, atleast in my case
• Figure out the best method to get docker logs into Alloy, it is not an Alloy issue, but rather a docker container issue, as there is no rock solid formatting style that everyone conforms to, because of this the options are to make Alloy do it, or perhaps ship all docker logs off to get reformatted elsewhere then to Alloy
  • One idea is to ship them all to journald, then format from there and / or a different docker logs plugin
• Add a Windows Exporter / Receiver
• Add a Linux Node Exporter / Receiver
• Much more, as I intend to make this an All In One that anyone could easily use

Note

For Windows Metrics, there are two different ways you can do it. If you use Windows Exporter, everything works perfect, logs and metrics. My issue was that Doing it that way does not allow you to limit the garbage metrics like Alloy can. Making the entire process start to finish as lean as possible is one of my main goals, so I then built another alloy config to use remotely on a windows machine that will pull only the metrics needed for the dashboards. The issue is even though they are both build exactly the same, for some reason random metrics are having a hard time showing up in grafana even though I can manually get them to work in promql. This tells me there is an issue in the dashboard variables somehow even though they are correct.

Note

One of the main reasons I never used grafana or the agent before, was because of heavy CPU usage, most always with Loki, this was due to container logs and the fact that Loki has no idea what to do if people, aka its users do not have their logs trimmed and proper. Loki will keep trying to reprocess the old logs, even if you have max age's set as it has to scan the files to see the age.......

#How-To-Install

1. Pull project from git:

git clone https://github.com/acester822/LGTMP.git

2. Start it up

docker compose up

To include NVIDIA GPU metrics with DCGM exporter:

docker compose --profile gpu up

Grafana LGTMP Stack default port-mapping

Port-mapping	Component	Description
12345:12345, 4317, 4318, 6831	Grafana Alloy	Expose 12345 port so we can directly access Alloy inside container
33100:3100	Loki	Expose 33100 port so we can directly access loki inside container
3000:3000, 6060	Grafana	Expose 3000 port so we can directly access grafana inside container
33200:3200, 4317, 4318	Tempo	Expose 33200 port so we can directly access tempo inside container
38080:8080	Mimir	Expose 38080 port so we can directly access mimir inside container
34040:4040	Pyroscope	Expose 34040 port so we can directly access pyroscope inside container
9001:9001, 9000	Minio	Expose 9001 port so we can access minio console with MINIO_ROOT_USER=lgtmp, MINIO_ROOT_PASSWORD=supersecret
3001:3000	ntopng	Expose 3001 for ntopng UI via nginx gateway routing
9400:9400	NVIDIA DCGM Exporter	GPU telemetry endpoint (enabled with --profile gpu)

ntopng setup

1. Start stack normally with docker compose up.
2. Open ntopng at http://localhost:3001.
3. Persisted data is stored in docker volume ntopng_data.
4. Alloy scrapes ntopng metrics from ntopng:3000/metrics (override with NTOPNG_METRICS_TARGET and NTOPNG_METRICS_PATH).

AI / LLM monitoring setup

1. Enable AI scraping in Alloy:
   • AI_MONITORING_ENABLED=true
2. Optional GPU metrics:
   • Run stack with --profile gpu to start dcgm-exporter.
3. Set endpoint targets as needed:
   • DCGM_EXPORTER_TARGET=dcgm-exporter:9400
   • AI_INFERENCE_METRICS_TARGET=<host:port>
   • AI_TOKEN_METRICS_TARGET=<host:port>
   • LLM_API_METRICS_TARGET=<host:port>
4. Example envs: config/alloy/modules/integrations/examples/ai-monitoring-example.alloy

AI framework examples

• OpenAI API: export latency and token counters through your app Prometheus endpoint and set AI_INFERENCE_METRICS_TARGET.
• LangChain: enable callback/telemetry exporter, then scrape app endpoint with AI_TOKEN_METRICS_TARGET.
• LlamaIndex: expose query latency + token metrics from app Prometheus middleware.
• Ollama: scrape Ollama-compatible metrics endpoint via LLM_API_METRICS_TARGET.

OPNsense integration setup

1. Enable Prometheus exporter plugin in OPNsense and expose metrics endpoint.
2. Enable Alloy module:
   • OPNSENSE_MONITORING_ENABLED=true
3. Set exporter endpoint:
   • OPNSENSE_METRICS_TARGET=<opnsense-host:port>
   • OPNSENSE_METRICS_PATH=/metrics (or custom path)
4. Example envs: config/alloy/modules/integrations/examples/opnsense-example.alloy

The OPNsense sample dashboard includes firewall rules metrics, traffic, gateway status, and system health panels.

Helpful Links

• https://grafana.com/docs/
• https://github.com/qclaogui/codelab-monitoring
• https://github.com/docker/compose
• https://grafana.com/docs/agent/latest/flow/reference/components/
• https://github.com/grafana/grafana
• https://grafana.com/docs/alloy/latest/
• https://github.com/grafana/alloy-modules/tree/main
• https://github.com/grafana/explore-logs