"If you want to have a Prometheus server, running outside of the Docker swarm, to scrape the metrics of your service, the easiest way is to just let it call the metrics endpoint of the published service and everything is fine, right? Well, if your service runs in replicated mode with multiple instances, you won’t get the values you expect. A call to the service actually ends up in the Docker network load balancer, which forwards the scrape request to one (!) of the running instances. So, the data you get are the metrics of one of the service instances (and you don’t know which one). As Prometheus scrapes the service metrics periodically, and every scrape request is routed independently from the previous ones, chances are that the next scrape request is routed to and answered by a different service instance returning the metrics of this instance, and so on. So, worst case is that Prometheus gets a different set of metrics on every scrape request. The resulting data does not give you any coherent picture of your service."
Use amongst the following guides:
1: https://www.innoq.com/en/blog/2020/04/scraping-docker-swarm-service-instances-with-prometheus/
2: https://prometheus.io/docs/guides/dockerswarm/
3: https://blog.okami101.io/2022/02/setup-a-docker-swarm-cluster-part-v-monitoring/
4. https://prometheus.io/docs/prometheus/latest/configuration/configuration/#dockerswarm_sd_config
"If you want to have a Prometheus server, running outside of the Docker swarm, to scrape the metrics of your service, the easiest way is to just let it call the metrics endpoint of the published service and everything is fine, right? Well, if your service runs in replicated mode with multiple instances, you won’t get the values you expect. A call to the service actually ends up in the Docker network load balancer, which forwards the scrape request to one (!) of the running instances. So, the data you get are the metrics of one of the service instances (and you don’t know which one). As Prometheus scrapes the service metrics periodically, and every scrape request is routed independently from the previous ones, chances are that the next scrape request is routed to and answered by a different service instance returning the metrics of this instance, and so on. So, worst case is that Prometheus gets a different set of metrics on every scrape request. The resulting data does not give you any coherent picture of your service."
Use amongst the following guides:
1: https://www.innoq.com/en/blog/2020/04/scraping-docker-swarm-service-instances-with-prometheus/
2: https://prometheus.io/docs/guides/dockerswarm/
3: https://blog.okami101.io/2022/02/setup-a-docker-swarm-cluster-part-v-monitoring/
4. https://prometheus.io/docs/prometheus/latest/configuration/configuration/#dockerswarm_sd_config