k8s-project

A small Kubernetes demo on Go: Envoy Gateway accepts traffic, gateway calls worker, worker does CPU-bound work, and Prometheus with Grafana show what's happening under load.

No databases, queues, or extra glue code. The point of the project is different: spin up a cluster quickly, fire off requests, watch the metrics, and see how HPA starts adjusting the replica count.

What's Inside

• apps/gateway - HTTP facade that accepts GET /api/work and proxies the request to worker.
• apps/worker - service with a heavy GET /work, deliberately designed to load the CPU.
• scripts/loadtest - a simple load generator for the demo stand.
• pkg - shared code for configuration, health checks, and Prometheus metrics.
• charts - Helm chart to deploy the entire stand to Kubernetes.

Architecture

Interactive version: presets/architecture.html - open locally in a browser.

Request flow:

• an external request arrives at Envoy Gateway;
• HTTPRoute forwards it to gateway-service;
• gateway calls worker-service;
• worker does the work and returns a response with the pod name;
• Prometheus scrapes metrics, Grafana draws charts, HPA watches CPU and scales the worker.

What Each Service Does

Gateway

gateway is a plain HTTP server on :8080. It exposes:

• /healthz
• /readyz
• /metrics
• /api/work

The idea is simple: gateway doesn't do heavy work itself - it just forwards the request and appends its own pod name to the response.

req, err := http.NewRequestWithContext(reqCtx, http.MethodGet, cfg.WorkerURL+"/work", nil)
if err != nil {
    http.Error(w, "Failed to build worker request", http.StatusInternalServerError)
    return
}

Worker

worker also starts an HTTP server and exposes:

• /healthz
• /readyz
• /metrics
• /work

The most important endpoint is /work. It intentionally loads the CPU, then waits briefly and returns the pod name:

for i := 0; i < 5_000_000; i++ {
    if i%100_000 == 0 {
        select {
        case <-ctx.Done():
            http.Error(w, "request canceled", http.StatusRequestTimeout)
            return
        default:
        }
    }

    _ = i * i
}

timer := time.NewTimer(20 * time.Millisecond)
defer timer.Stop()

This is exactly what's needed for the demo:

• as concurrency increases, CPU starts rising;
• latency climbs as well;
• HPA sees the load and adds replicas;
• responses show which pod actually handled the request.

Load Test

loadtest is a small separate container that hammers the target URL and logs progress.

It's useful when you need to check:

• how worker behaves under parallel requests;
• how metrics react in Prometheus;
• how quickly HPA kicks in.

Metrics and Alerts

pkg contains a shared metrics layer. It registers:

• http_requests_total
• http_request_duration_seconds
• standard process/go collectors

This makes it convenient to use in Grafana and alerts without extra code in each service.

Useful PromQL queries to explore in Grafana:

rate(process_cpu_seconds_total{namespace="default"}[5m])
sum(rate(http_requests_total{namespace="default", app="worker"}[5m])) by (status)
histogram_quantile(
  0.95,
  sum(rate(http_request_duration_seconds_bucket{namespace="default", app="worker"}[5m])) by (le)
)

The chart already includes basic alerts:

• high CPU;
• high memory;
• high error rate;
• high latency;
• pod not ready.

What You See Under Load

flowchart LR
    Load[Load test] --> W[Worker]
    W -->|http_requests_total| Prom[Prometheus]
    W -->|http_request_duration_seconds| Prom
    Prom --> Graf[Grafana]
    Prom --> HPA[HPA]
    HPA --> W

Loading

In this project the load is not decorative:

• worker runs millions of operations in a loop;
• loadtest fires parallel requests;
• HPA scales worker by CPU with a 50% target, from 2 to 8 replicas;
• Grafana shows rising CPU, RPS, and p95 latency.

If you watch it as a live demo, the picture looks like this:

1. Start loadtest.
2. CPU on worker rises almost immediately.
3. Latency starts growing along with the request queue.
4. HPA adds new replicas.
5. curl responses show different pod names.

Quick Start

The easiest way to bring everything up is via Taskfile:

task up

What task up does:

• builds gateway and worker images;
• loads them into k3d;
• deploys Envoy Gateway;
• installs kube-prometheus-stack;
• deploys the chart;
• shows pod and service status.

If you prefer to do it manually, the order is:

task cluster
task init:envoy
task init:monitoring
task build
task load
task deploy

Useful Commands

task status
task grafana
task ui
task down

• task status - shows pods and services;
• task grafana - opens a port-forward to Grafana;
• task ui - opens k9s in the default namespace;
• task down - removes the Helm release and the k3d cluster.

Manual Verification

After deployment the cluster listens on localhost:8080, so you can hit the gateway without any extra port-forwarding:

curl http://localhost:8080/api/work
curl http://localhost:8080/metrics | head

A typical response looks like:

Gateway [Pod: gateway-7c4d8f5b9c-xyz12] got work: worker-6d7f9c8b4f-abcd3

This is handy when you want to see at a glance that requests are already being spread across different pods.

Load Test

The repository includes a ready-made task load-test, but it's more of a short smoke test. If you need visible CPU growth and autoscaling, let the test run for minutes, not seconds.

Direct run against worker:

kubectl run go-worker-loadtest --rm -i --restart=Never \
  --image loadtest:v1 \
  --env TARGET_URL=http://worker-service.default.svc.cluster.local:8080/work \
  --env DURATION=3m \
  --env CONCURRENCY=100 \
  --env REQUEST_TIMEOUT=30s \
  --env REPORT_EVERY=10s

Run through gateway:

kubectl run go-gateway-loadtest --rm -i --restart=Never \
  --image loadtest:v1 \
  --env TARGET_URL=http://gateway-service.default.svc.cluster.local:8080/api/work \
  --env DURATION=3m \
  --env CONCURRENCY=100 \
  --env REQUEST_TIMEOUT=30s \
  --env REPORT_EVERY=10s

For a clean HPA test, hit worker directly. To verify the full request path, run through gateway.

Things worth watching during the run:

• kubectl get hpa;
• Grafana application dashboard;
• Grafana cluster dashboard;
• Prometheus targets and alerts.

Configuration

Gateway

• PORT
• WORKER_URL
• REQUEST_TIMEOUT
• READ_TIMEOUT
• WRITE_TIMEOUT
• IDLE_TIMEOUT
• SHUTDOWN_TIMEOUT

Worker

• PORT
• READ_TIMEOUT
• WRITE_TIMEOUT
• IDLE_TIMEOUT
• SHUTDOWN_TIMEOUT

Loadtest

• TARGET_URL
• DURATION
• CONCURRENCY
• REQUEST_TIMEOUT
• REPORT_EVERY

Building Images

Images are built separately because gateway and worker use the shared pkg as an additional build context.

docker build -t gateway:v1 --build-context pkg=pkg ./apps/gateway
docker build -t worker:v1 --build-context pkg=pkg ./apps/worker
docker build -t loadtest:v1 ./scripts/loadtest

charts/values.yaml defaults to:

• gateway:v1
• worker:v1
• loadtest:v1

imagePullPolicy: Never is set, so images must be loaded into the cluster before deploying to k3d.

Repository Structure

apps/
  gateway/
  worker/
pkg/
scripts/
  loadtest/
charts/
Taskfile.yaml

Notes

• readyz is currently a stub and always returns 200 OK; in a real project you would typically add a dependency check here.
• HPA watches CPU, not RPS, so CPU load is what matters for the autoscaling demo.
• The chart uses Envoy Gateway, not a regular Ingress.