A real-time deployment monitoring dashboard for Kubernetes. It gives PMs and developers a single pane of glass for every service running on a cluster: health, replica counts, deployed git SHAs, pod-level diagnostics, and ReplicaSet rollout history — all read live from the Kubernetes API on every page load (no sync worker, nothing cached).
Sign-in is PennKey SSO through Penn Labs' Platform, and access is currently limited to team leads — see Authentication.
- Django + DRF backend that wraps the Kubernetes Python client. Stateless — every request hits the K8s API directly, so you always see real-time state.
- React + TypeScript frontend with React Query for auto-refreshing data (every 10–15 s).
- Multi-stage Docker builds for both services. The frontend ships as nginx + 25 MB of static assets; the backend ships a resolved virtualenv with no package-manager caches or build tools.
- CDK8s / Kittyhawk manifests describing how the dashboard would deploy to a real cluster, including the least-privilege ClusterRole the backend needs to read the cluster.
- Sample fake deployments so anyone can run the dashboard locally on
kindwithout needing access to a real cloud cluster. - Penn Labs SSO via
django-labs-accounts, with authorization derived from the product-admin permissions Platform already tracks — so "team leads only" needs no separate list of pennkeys to maintain. - Commit context on each deployment: messages and authors from the GitHub API, and GitHub compare links between revisions, including from the rollback dialog. See Commit details.
- Unit tests that mock the K8s client and verify both the API logic and the access-control rules in isolation.
┌──────────────┐ ┌────────────────┐ ┌─────────────────┐
│ React SPA │ ──HTTP─▶│ Django + DRF │ ──HTTPS▶│ Kubernetes API │
│ (port 3000) │◀────────│ (port 8000) │◀────────│ (live cluster)│
└──────────────┘ └───────┬────────┘ └─────────────────┘
browser │ EKS / kind / minikube
│ OAuth2 + introspection
▼
┌──────────────────────┐
│ platform.pennlabs.org│
│ (PennKey identity) │
└──────────────────────┘
No deployment data is ever persisted. Every API call from the frontend
translates into a fresh request to the Kubernetes API. The label
app.kubernetes.io/managed-by=kittyhawk is used as the discovery filter so
the dashboard only surfaces Penn Labs–managed resources, not system pods.
There is a small SQLite database, but only for authentication — Django's user/session tables plus the OAuth token tables. See Authentication.
| Concept | Where it lives |
|---|---|
| Containerization | backend/Dockerfile, frontend/Dockerfile (both multi-stage) |
| Orchestration | k8s/main.ts — Kubernetes Deployments + Services + Ingress |
| Infrastructure as Code | CDK8s in TypeScript synthesizes raw K8s YAML |
| RBAC / least privilege | k8s/main.ts — narrowly-scoped ClusterRole + bound ServiceAccount |
| SSO / authorization | Penn Labs OAuth2 via django-labs-accounts; access limited to team leads (deployments/backends.py, deployments/permissions.py) |
| Reproducibility | uv.lock pins every transitive dep; pinned base images; docker compose up produces the same env everywhere |
| Portability | Same image runs locally (kind), in EKS, or any K8s cluster |
| Testability | backend/deployments/tests.py with mocked K8s client |
| Observability | The dashboard itself is an observability tool |
This is the recommended path for graders or anyone without Penn Labs cloud access. Brings up a real Kubernetes cluster inside a Docker container, loads fake-but-realistic Deployments, and runs the dashboard against them.
- Docker Desktop (running)
kubectl,kind, Node.js 18+,uvbrew install kubectl kind node uv
uv manages the backend's Python dependencies. You don't need to install
Python yourself — uv sync reads requires-python from
backend/pyproject.toml and fetches a matching interpreter if your system
doesn't already have one.
git clone <this-repo>
cd dashboard
# 1. Spin up a local Kubernetes cluster
kind create cluster --config k8s/kind-cluster.yaml
# 2. Load four fake deployments (3 healthy, 1 deliberately broken)
kubectl apply -f k8s/sample-deployments.yaml
# 3. Backend
cd backend
uv sync # creates .venv from uv.lock
uv run python manage.py migrate # creates the auth database
uv run python manage.py runserver &
cd ..
# 4. Frontend
cd frontend
npm install
npm startOr just ./dev.sh, which does all of the above (including migrate) and
prints which cluster it's about to read.
Open http://localhost:3000 — always port 3000, never 8000 directly, because
the dev server proxies /api through to Django to keep the OAuth flow
same-origin. Sign in with your PennKey, and you should see three product cards:
penn-clubs (green), ohq (green), penn-courses (red — the broken one,
demonstrating the degraded-health UI). Click in to see pods and ReplicaSet
history.
If you get "You don't have access yet", your Penn Labs account isn't a team lead — see Authentication.
One command instead of separate backend/frontend processes. Mounts your host
kubeconfig read-only into the backend container so it talks to whatever
cluster kubectl on your laptop is currently pointed at.
docker compose up --build
# frontend: http://localhost:3000
# backend: http://localhost:8000cd backend
uv run python manage.py test
# Ran 87 tests in 0.09s — OKuv run syncs the environment against uv.lock before executing, so you
never run tests against a stale set of packages.
The tests mock the Kubernetes client and the GitHub API entirely, so they don't need a cluster or a network. They verify view logic (label-based product grouping, dict shaping for the JSON API, 404 handling, rollback input validation) and access control: that anonymous users and signed-in non-leads are refused, that a rollback is rejected before the cluster is touched, and that the mapping from Platform permissions to lead status grants and revokes correctly.
Sign-in is Penn Labs SSO, via django-labs-accounts (DLA) — the same
client library Penn Clubs, OHQ and Penn Mobile use. Access is currently
limited to team leads, plus any pennkey named in DASHBOARD_ALLOWED_USERS.
Signing in is the first thing a visitor sees. Hitting dashboard.pennlabs.org
with no session redirects straight to Platform rather than showing a "sign in"
button, which is what Platform's own frontend does — withAuth in its
frontend/utils/auth.tsx redirects to /accounts/login/?next=… the moment
/accounts/me/ comes back non-ok. Platform does that server-side in
getServerSideProps; this app is a static bundle behind nginx, so AuthGate.tsx
does the equivalent as soon as /api/me/ answers. The one case that does not
auto-redirect is a failed sign-in (?auth_error=…), which would otherwise
loop — that renders an explanation and a manual "Try again".
browser this app platform.pennlabs.org
│ │ │
│ GET /api/accounts/login/ │ │
├─────────────────────────▶│ │
│◀──── 302 ────────────────┤ /accounts/authorize/ │
├──────────────────────────────────────────────────────────────▶│
│ │ (PennKey / Shibboleth) │
│◀──── 302 back to /api/accounts/callback/?code=… ─────────────┤
├─────────────────────────▶│ exchange code at /accounts/token/│
│ ├──────────────────────────────────▶│
│ │ POST /accounts/introspect/ │
│ ├──────────────────────────────────▶│
│ │◀─ user: pennkey, name, email, │
│ │ groups, user_permissions ───┤
│◀──── 302 to / (session cookie set) ──────────────────────────┤
DLA's views are mounted at /api/accounts/ rather than /accounts/. That's
deliberate: the production ingress already routes /api to Django (see
k8s/main.ts), so no extra ingress rule is needed, and /api/accounts/callback/
is the exact path Penn Labs' shared development OAuth client is registered
against.
We don't keep our own list of pennkeys. Penn Labs already tracks this, and we read it:
pennlabs/infrastructure/docker/team-syncruns on a cron. Every GitHub team in the org whose slug ends in-leadsis a lead team, and<x>-leadsmaps to the permission codename<x>_admin(dashes become underscores) — sopenn-clubs-leadsyieldspenn_clubs_admin. Members of theDirectorsteam receive every such codename.- team-sync POSTs
{pennkey: [codenames]}to Platform's/accounts/productadmin/, which materializes them as Django permissions. - Platform reports them back to us in the
user_permissionsfield of the introspection response.
Because step 1 only ever issues *_admin codenames to lead teams and to
Directors, "holds at least one *_admin permission" is exactly equivalent to
"is a lead of some team, or a Director" (deployments/backends.py). It needs no
new GitHub team and no change to Platform. Lead status is re-evaluated on
every login, so someone who rotates off a team loses access next time they
sign in; SESSION_COOKIE_AGE (12h) bounds how long an already-issued session
outlives the change.
DASHBOARD_ALLOWED_USERS (in dashboard/settings/base.py, overridable as a
comma-separated env var; defaults to chenstan) always has access, regardless of
permissions. It is additive — it never removes the lead rule — and matching is
case-insensitive. Two cases need it, and neither is a corner case:
- Whoever operates this dashboard without leading a product team. Step 1 only
issues codenames to people who are both in a
*-leadsGitHub team and listed in team-sync'sroster.yaml, so a DevOps maintainer holds no*_adminat all and would be locked out of the tool they run. - A Platform lead. Platform's
ProductAdminViewspecial-cases theplatform_adminslug: it setsis_superuseron the Platform-side user andcontinues before creating aPermission. Soplatform_adminnever comes back from introspection, and introspection doesn't exposeis_staff/is_superusereither (see Platform'sUserSerializer.Meta.fields) — there is nothing for the*_admintest to match. A Platform lead who leads no other product is refused without this. Directors are unaffected, because team-sync hands them every other product codename too.
Enforcement is server-side: IsTeamLead is the project-wide default DRF
permission, so every endpoint is closed unless it explicitly opts out. Only
/api/health/ (liveness probes) and /api/me/ (so the SPA can discover whether
you're signed in) are public. AuthGate.tsx is a UX layer on top, not the
boundary.
Create a dashboard-leads team in the GitHub org. team-sync will start issuing
dashboard_admin, which the existing check already matches — no code change.
Logging in locally does a real OAuth round-trip against
platform-dev.pennlabs.org using Penn Labs' shared development client
(configured in dashboard/settings/development.py). There is no auth-bypass
switch, so you need network access and a Penn Labs account on platform-dev.
engineering
(engineering@upenn.edu, pennid 1000) — not your own PennKey. Your real pennkey
only comes back from production Platform. So an allowlist of real pennkeys can
never match locally, and every local login would otherwise land on "You don't
have access yet". development.py therefore appends engineering to
DASHBOARD_ALLOWED_USERS. That module is never loaded in production, so the
shared test account cannot reach the real dashboard.
If a local login lands on "You don't have access yet", check which user you actually became before assuming the OAuth flow is broken:
cd backend && sqlite3 db.sqlite3 'SELECT id, username, is_staff FROM auth_user;'- register
https://dashboard.pennlabs.org/api/accounts/callback/as a redirect URI, with scopesreadandintrospection - put the issued credentials in Vault as
LABS_CLIENT_ID/LABS_CLIENT_SECRETin thedashboardsecret, alongside aSECRET_KEY
dashboard.settings.production refuses to start without SECRET_KEY, since it
signs session cookies.
Note the backend runs a single replica (k8s/main.ts). Auth state lives in a
SQLite file inside the container, so a session minted on one pod would mean
nothing on another. The file is ephemeral by design — Platform is the source of
truth for identity and lead status, so a pod restart just means users silently
re-authenticate through their existing SSO session. Moving to Postgres is what
would let the API scale out again.
The deployment page shows each deployed commit's message and author, and links to GitHub's compare view between any two revisions. The rollback dialog links to the same diff, showing what the rollback would take away.
The repo is assumed to be pennlabs/<product>, the same convention the SHA links
use. Lookups go through the backend (deployments/github_client.py), in their own
request so a slow GitHub never delays the live Kubernetes data, and the SHAs come
from the cluster rather than from the browser.
GITHUB_TOKEN is optional, but production should have one. Without it, private
repos aren't visible and GitHub allows 60 requests an hour, shared with everything
else behind the cluster's NAT. Use a token with read-only access to repo contents
and add it to the Vault secret at secrets/production/default/dashboard; every key
there reaches the backend as an environment variable, so the chart doesn't change.
Locally, GITHUB_TOKEN=$(gh auth token) ./dev.sh does the job.
Unlike Kubernetes state, this data is cached, in memory per gunicorn worker, because a commit never changes. Anything GitHub can't answer falls back to the bare SHA. Compare links are plain links to github.com, so they need neither the token nor the API; viewing a private repo's diff needs the viewer's own GitHub access.
backend/
├── Dockerfile # multi-stage uv build
├── pyproject.toml # dependency declarations (uv)
├── uv.lock # fully-pinned resolution — commit this
├── manage.py
├── dashboard/ # Django project
│ ├── settings/
│ │ ├── base.py # shared; SQLite for auth only, DLA config
│ │ ├── development.py # local: ~/.kube/config, platform-dev OAuth
│ │ └── production.py # in-cluster: ServiceAccount token, real OAuth
│ └── urls.py # mounts DLA at /api/accounts/
└── deployments/ # the actual app
├── k8s_client.py # all calls to the K8s API live here
├── github_client.py # commit details from GitHub, cached
├── backends.py # auth backend: who counts as a team lead
├── permissions.py # IsTeamLead — the project-wide default
├── views.py # thin DRF views, no business logic
├── urls.py # routes keyed by (namespace, name)
└── tests.py # unit tests with mocked K8s client
frontend/
├── Dockerfile # node build → nginx serve (also proxies /api)
└── src/
├── setupProxy.js # dev-only: /api → Django, keeps OAuth same-origin
├── api/ # fetch wrappers (+ CSRF header)
├── hooks/ # React Query hooks (auto-refetch), useAuth
├── pages/ # Dashboard, ProductDetail, DeploymentDetail
├── components/ # AuthGate, HealthBadge, PodStatusList, ...
└── types/ # TypeScript shapes mirroring the API
k8s/
├── main.ts # CDK8s / Kittyhawk chart
├── package.json # `npm run build` synthesizes YAML to dist/
├── yarn.lock # pins cdk8s/constructs — CI needs this
├── sample-deployments.yaml # fake services for local demos
└── kind-cluster.yaml # kind cluster config
docker-compose.yml # one-command local stack
/database shows what the shared production database was doing at any point in
the history AWS keeps: load over time, then the top queries, database users and
wait events for that window. It's for answering "which query was that?" when the
RDS CPU alarm fires. Paste the alarm email into the box, or share the page's URL,
which holds the time window.
The data comes from Performance Insights
through deployments/pi_client.py, not from the database itself, so the dashboard
has no database credentials. In production the backend's ServiceAccount assumes
the dashboard-backend IAM role (terraform/dashboard.tf in
pennlabs/infrastructure). Locally, boto3 uses your own AWS profile:
AWS_PROFILE=pennlabs ./dev.shTwo things to keep in mind when reading it:
- Query text can include real values despite being "tokenized", which is why the endpoint is limited to team leads like the rest of the API.
- Load is sampled once a second. It shows what the database spent its time on, not how many times a query ran, and it can't say which request sent a query.
If the instance is resized, set DATABASE_VCPUS so the graph's vCPU line stays
right.
The dashboard relies on three labels Penn Labs' Kittyhawk library applies to every managed resource:
| Label | Used for |
|---|---|
app.kubernetes.io/managed-by=kittyhawk |
Filter — what to show vs. hide |
app.kubernetes.io/part-of=<repo> |
Group Deployments into "products" |
app.kubernetes.io/version=<git-sha> |
Display the deployed commit |
app.kubernetes.io/name=<deployment> |
Find the pods for a given Deployment |
The dashboard wouldn't exist without these conventions — they turn the cluster into a queryable database. Adding the dashboard cost zero changes to any existing service.
Running inside the cluster, the backend pod authenticates as a ServiceAccount,
which by default has zero API permissions — without a role, every list
call 403s and the dashboard renders empty. k8s/main.ts declares the narrowest
ClusterRole that makes it work:
get,list,watchon deployments and replicasets (apps/v1)get,list,watchon pods (core/v1)patchon deployments only (the rollback button)
No write access on pods or replicasets. No access to Secrets, ConfigMaps, Services, or anything else. A bug or an exploit in the dashboard cannot delete or modify any other workload in the cluster.
Two details worth knowing:
- The ServiceAccount, ClusterRole and ClusterRoleBinding are declared in the
chart, not as a separate hand-written YAML file, so CI applies them with
everything else. A standalone
rbac.yamlwould never be applied, since the deploy step only appliesk8s/dist/. - We do not use kittyhawk's
createServiceAccount: true. It is broken —application/base.jspasses the account down asserviceAccountName(a string) whiledeployment.jsonly emits that field whenprops.serviceAccount(an object) is set, so it creates a ServiceAccount and never attaches it, and the pod silently runs asdefault. It also annotates the account for IRSA, which expects a matching IAM role; we don't need one, because this backend calls the Kubernetes API, not any AWS API.
cd k8s
yarn install --frozen-lockfile
RELEASE_NAME=dashboard GIT_SHA=$(git rev-parse HEAD) AWS_ACCOUNT_ID=<acct> yarn build
# YAML appears in k8s/dist/All three environment variables are required. RELEASE_NAME in particular is
not optional: kittyhawk reads it to set app.kubernetes.io/part-of, which names
every resource and is what kubectl apply --prune selects on — omit it and you
get undefined_release-* names. GIT_SHA becomes the image tag.
Use yarn, not npm: cdk8s and constructs are pinned to exact versions with
a resolutions block, because kittyhawk's own caret ranges otherwise resolve to
newer copies that fail to compile against ours. yarn.lock is committed and CI
installs with --frozen-lockfile.
Inspect dist/dashboard.k8s.yaml to see exactly what deploys — 11 objects,
including the ServiceAccount and RBAC.
This project was descoped from a more ambitious original proposal (database + sync worker + CI/CD pipeline) after instructor feedback that "the simpler the better." Things that were intentionally cut and would be the natural next steps:
- CI/CD via GitHub Actions — build/test on PR, push images to Docker
Hub on merge,
kubectl applyto EKS. Was in the original proposal. - Deployment history — would require adding a small database back, since the K8s API doesn't retain deploy events beyond the lifetime of the ReplicaSets that produced them (typically ~10 revisions).
- Postgres instead of SQLite — auth state currently lives in a SQLite file
inside the container, which is why the API runs a single replica. Swapping in
Postgres (via
DATABASE_URL, the way Platform and Penn Clubs do it) would let it scale out again. See Authentication. - Wider access than team leads — the dashboard is deliberately restricted to
leads for now. A
dashboard-leadsGitHub team would widen it without a code change. - Alerts — turn the polling-based dashboard into push-based notifications when a deployment becomes unhealthy.
- Multi-cluster support — currently the backend reads one kubeconfig context at a time. Adding a context switcher would let one dashboard span staging + production.
MIT.