Add asynchronous full-site and URL-list accessibility audits

.github/workflows/ci.yml

@@ -53,6 +53,9 @@ jobs:
           print('all entry points import cleanly')
           "
 
+      - name: Unit tests
+        run: uv run python -m unittest discover -v
+
       # Exercises extractors, slicers, prompt templates and the programmatic
       # checkers end to end. --dry-run makes no API calls, so this needs no key
       # and costs nothing.

DEPLOY.md

@@ -21,11 +21,16 @@ environment variables below, attach a domain, and deploy.
 | `HOST` | No | Set to `0.0.0.0` by the image; do not override |
 | `PORT` | No | Defaults to 8000; Coolify may inject its own |
 | `HOST_PORT` | No | Local-only: host port for `docker compose up` (default 8000) |
+| `DAT_DAILY_MONITOR_EMAIL` | For daily monitoring | Recipient for the 24-hour usage and health report |
+| `DAT_MONITOR_COLLECTION` | No | Firestore monitor-run collection; defaults to `dat_site_audit_monitor_runs` |
+| `DAT_USAGE_COLLECTION` | No | Privacy-safe synchronous-audit usage events; defaults to `dat_audit_usage_events` |
+| `DAT_SITE_PASSWORD` | For administrator tools | Shared password for Analytics and both bulk-audit modes; store in Secret Manager |
+| `DAT_OPENAI_API_KEY` | For administrator tools | Saved OpenAI key available only to authenticated administrators; store in Secret Manager |
 
-**No provider API keys are configured, deliberately.** Each user pastes their
-own key into the web form, so the deployment never bills a shared account.
-Per-request keys take priority over the environment
-(`api_server.py:_resolve_api_key`).
+Public users paste their own key into the web form and never inherit provider
+keys from the service environment. An authenticated administrator may use the
+saved `DAT_OPENAI_API_KEY`; per-request keys take priority. The raw saved key is
+never returned to the browser.
 
 Consequence worth knowing: **a request with no resolvable key silently runs a
 dry run** — programmatic checks only, no LLM findings, no CSV report, and a
@@ -56,17 +61,16 @@ the clone), but local `docker compose up` is.
 
 ## Proxy configuration — read this one
 
-`POST /api/audit`, `/api/audit/url`, and `/api/audit/url/nested` return an
+`POST /api/audit` and `/api/audit/url` return an
 **NDJSON stream** that stays open for the whole audit, emitting progress events
 as it goes. Two default proxy behaviours will break this:
 
 1. **Response buffering.** A proxy that buffers will hold every progress event
    until the audit finishes, so the browser shows a frozen progress bar and then
    all results at once. Disable buffering for `/api/`.
 2. **Read timeouts.** A single-page audit runs 17+ sequential LLM calls and
-   commonly takes 1–3 minutes; a nested crawl audits every discovered page and
-   can run far longer. Anything under ~10 minutes risks cutting audits off
-   mid-run. Traefik's default is generous, but Coolify installs vary — verify.
+   commonly takes 1–3 minutes. Anything under ~10 minutes risks cutting audits
+   off mid-run. Traefik's default is generous, but Coolify installs vary — verify.
 
 In Coolify, set these as Traefik labels on the service, or raise the equivalent
 values in the proxy settings:
@@ -96,8 +100,52 @@ Audits are I/O-bound on the LLM API, not CPU-bound. 512 MB RAM and 0.5 vCPU is
 enough for light use. The server is a stdlib `ThreadingHTTPServer`, so each
 concurrent audit holds a thread and its own temp directory — fine for a handful
 of simultaneous users, not for public high traffic. The `tmpfs` mount is capped
-at 512 MB; a large nested crawl of very large pages could approach that, so
-raise it if you see failures writing temp files.
+at 512 MB; large pages could approach that, so raise it if you see failures
+writing temp files.
+
+## Daily DAT usage and health monitor
+
+The authenticated endpoint `POST /api/internal/site-audits/daily-monitor`
+collects audits active during the preceding 24 hours. It reports
+each scanned site, mode, status, model, pages processed, page failures, and
+estimated AI cost. It also checks the public health endpoint, Firestore, report
+storage, background-audit configuration, the saved model credential, email
+configuration, and jobs that have stopped progressing for more than six hours.
+It also saves an HTML copy in private report storage. The protected Analytics
+page lists the latest 30 copies; older daily report records and objects are
+removed automatically.
+Synchronous modes store only operational totals (site host when available,
+model, status, token totals, estimated cost, and page counts), never uploaded
+HTML, provider credentials, or requester information.
+
+Cloud Scheduler should call the endpoint at 6:00 a.m. in the
+`America/New_York` timezone. Use the same Secret Manager value already supplied
+to the service as `DAT_JOB_TOKEN`; never place the token in source control.
+
+```powershell
+$datJobToken = (gcloud secrets versions access latest `
+  --secret ability-bazaar-dat-job-token `
+  --project ability-bazaar-2026 | Out-String).Trim()
+
+gcloud scheduler jobs create http ability-bazaar-dat-daily-monitor `
+  --project ability-bazaar-2026 `
+  --location us-east1 `
+  --schedule "0 6 * * *" `
+  --time-zone "America/New_York" `
+  --uri "https://ability-bazaar-dat-621739355963.us-east1.run.app/api/internal/site-audits/daily-monitor" `
+  --http-method POST `
+  --headers "Content-Type=application/json,X-DAT-Job-Token=$datJobToken" `
+  --message-body "{}" `
+  --attempt-deadline 5m `
+  --max-retry-attempts 3
+
+$datJobToken = $null
+```
+
+The endpoint deduplicates successful scheduled reports by Eastern calendar
+date, so a Scheduler retry does not send a second daily email. Failed email
+attempts remain retryable. A manual verification can pass
+`{"send_email":false}` to exercise the health and usage checks without sending.
 
 ## Serving
 

Dockerfile

@@ -35,7 +35,15 @@ RUN uv sync --frozen --no-dev
 FROM python:3.12-slim
 
 ENV PYTHONUNBUFFERED=1 \
-    PYTHONDONTWRITEBYTECODE=1
+    PYTHONDONTWRITEBYTECODE=1 \
+    DAT_CHROMIUM_EXECUTABLE=/usr/bin/chromium
+
+# SiteGround and similar CDNs can return JavaScript security interstitials to
+# cloud data-center traffic. Chromium executes the legitimate public-site
+# challenge so the auditor receives the requested page, never the interstitial.
+RUN apt-get update \
+    && apt-get install -y --no-install-recommends chromium \
+    && rm -rf /var/lib/apt/lists/*
 
 # Single chowned COPY: the virtualenv lands in exactly one layer.  Copying it
 # after creating it, rather than chown -R'ing it in place, keeps the image from

README.md

@@ -244,9 +244,23 @@ GEMINI_API_KEY=AIza...
 uv run python entry_points/api_server.py      # http://localhost:8000
 ```
 
-Serves the UI and the audit API from one process. Users can paste their own API
-key into the form instead of configuring one server-side; per-request keys take
-priority over the environment.
+Serves the UI and the audit API from one process. The public page has no
+password and offers **HTML File** and **URL — Single Page**. Public users may
+paste their own provider key; public requests never inherit a server credential.
+
+The **Analytics** link opens the password-protected administrator area. It
+provides cumulative privacy-safe usage totals, the most recent 30 daily web
+reports, and two asynchronous bulk-audit modes with the same model, saved-key
+support, email, progress, consolidated findings, and token/cost reporting:
+
+- **Full Site — Crawl and Email**: starts with one public URL,
+  discovers same-site pages, and audits up to 200 pages.
+- **Full Site — Batch and Email**: accepts a UTF-8/plain-text `.txt` file or a
+  Word `.docx` file containing up to 200 public HTTP(S) URLs. It deduplicates
+  the list and audits the supplied pages directly without crawling.
+
+The list upload is limited to 2 MB. Legacy `.doc` files are not supported;
+save them as `.docx` or plain text first.
 
 The audit endpoints stream **NDJSON** — progress events, one JSON object per
 line, then a final `{"type":"result"}` object. Parsing that body with a single

analytics.html

@@ -0,0 +1,165 @@
+<!DOCTYPE html>
+<html lang="en">
+  <head>
+    <meta charset="UTF-8" />
+    <meta name="viewport" content="width=device-width, initial-scale=1" />
+    <meta name="robots" content="noindex,nofollow" />
+    <title>Analytics | Vision Aid DAT</title>
+    <link rel="stylesheet" href="/styles.css" />
+  </head>
+  <body class="analytics-page">
+    <a href="#main" class="skip-link">Skip to main content</a>
+    <header class="navbar">
+      <nav class="navbar-inner" aria-label="Analytics navigation">
+        <a href="/" class="navbar-brand">Vision Aid DAT</a>
+        <ul class="nav-links analytics-nav">
+          <li><a href="/analytics/full-site">Full Site — Crawl and Email</a></li>
+          <li><a href="/analytics/url-list">Full Site — Batch and Email</a></li>
+          <li><a href="/analytics" aria-current="page">Analytics</a></li>
+          <li><button class="nav-audit-btn" id="logoutBtn" type="button">Sign out</button></li>
+        </ul>
+      </nav>
+    </header>
+    <main id="main" class="analytics-main">
+      <div class="container">
+        <div class="section-header">
+          <div class="accent-bar" aria-hidden="true"></div>
+          <h1>Analytics</h1>
+          <p>Private usage totals, administrator audit tools, and the last 30 daily reports.</p>
+        </div>
+
+        <section aria-labelledby="totalsHeading">
+          <h2 id="totalsHeading">Usage totals</h2>
+          <div class="analytics-metrics" id="analyticsMetrics" aria-live="polite">
+            <p>Loading analytics…</p>
+          </div>
+          <p class="field-hint" id="historyNote"></p>
+        </section>
+
+        <section aria-labelledby="toolsHeading" class="analytics-section">
+          <h2 id="toolsHeading">Administrator audit tools</h2>
+          <div class="analytics-tools">
+            <a class="analytics-tool-card" href="/analytics/full-site">
+              <strong>Full Site — Crawl and Email</strong>
+              <span>Discover and audit up to 200 pages, then email the consolidated report.</span>
+            </a>
+            <a class="analytics-tool-card" href="/analytics/url-list">
+              <strong>Full Site — Batch and Email</strong>
+              <span>Upload a .txt or .docx list of up to 200 URLs for direct batch processing.</span>
+            </a>
+          </div>
+        </section>
+
+        <section aria-labelledby="reportsHeading" class="analytics-section">
+          <h2 id="reportsHeading">Daily reports</h2>
+          <p>Reports are retained here for 30 days.</p>
+          <div class="analytics-table-wrap">
+            <table class="analytics-table">
+              <thead>
+                <tr>
+                  <th scope="col">Date</th>
+                  <th scope="col">Health</th>
+                  <th scope="col">Audits</th>
+                  <th scope="col">Pages</th>
+                  <th scope="col">Cost</th>
+                  <th scope="col">Report</th>
+                </tr>
+              </thead>
+              <tbody id="dailyReports">
+                <tr><td colspan="6">Loading daily reports…</td></tr>
+              </tbody>
+            </table>
+          </div>
+          <p class="form-error" id="analyticsError" role="alert" hidden></p>
+        </section>
+      </div>
+    </main>
+    <script>
+      const metrics = [
+        ["users", "Users"],
+        ["reports", "Reports"],
+        ["sites", "Sites"],
+        ["pages_scanned", "Pages scanned"],
+        ["total_tokens", "Tokens"],
+        ["estimated_cost_usd", "Estimated cost"],
+      ];
+      const number = new Intl.NumberFormat("en-US");
+      const money = new Intl.NumberFormat("en-US", {
+        style: "currency",
+        currency: "USD",
+        minimumFractionDigits: 2,
+        maximumFractionDigits: 6,
+      });
+
+      function renderMetric(key, label, value) {
+        const card = document.createElement("div");
+        card.className = "analytics-metric";
+        const strong = document.createElement("strong");
+        strong.textContent = key === "estimated_cost_usd" ? money.format(value || 0) : number.format(value || 0);
+        const span = document.createElement("span");
+        span.textContent = label;
+        card.append(strong, span);
+        return card;
+      }
+
+      function renderReports(reports) {
+        const body = document.getElementById("dailyReports");
+        body.replaceChildren();
+        if (!reports.length) {
+          const row = body.insertRow();
+          const cell = row.insertCell();
+          cell.colSpan = 6;
+          cell.textContent = "No daily reports are available yet.";
+          return;
+        }
+        reports.forEach((report) => {
+          const row = body.insertRow();
+          [
+            report.report_date,
+            report.health_status,
+            number.format(report.audit_count || 0),
+            number.format(report.pages_processed || 0),
+            money.format(report.estimated_cost_usd || 0),
+          ].forEach((value) => {
+            row.insertCell().textContent = value;
+          });
+          const reportCell = row.insertCell();
+          if (report.report_available) {
+            const link = document.createElement("a");
+            link.href = report.report_url;
+            link.textContent = "View report";
+            reportCell.appendChild(link);
+          } else {
+            reportCell.textContent = "Unavailable";
+          }
+        });
+      }
+
+      async function loadAnalytics() {
+        try {
+          const response = await fetch("/api/admin/analytics", { cache: "no-store" });
+          if (response.status === 401) {
+            location.assign("/analytics");
+            return;
+          }
+          const data = await response.json();
+          if (!response.ok || !data.success) throw new Error(data.error || "Could not load analytics");
+          const container = document.getElementById("analyticsMetrics");
+          container.replaceChildren(...metrics.map(([key, label]) => renderMetric(key, label, data.summary[key])));
+          document.getElementById("historyNote").textContent = data.summary.history_note || "";
+          renderReports(data.reports || []);
+        } catch (error) {
+          const errorBox = document.getElementById("analyticsError");
+          errorBox.textContent = error.message;
+          errorBox.hidden = false;
+        }
+      }
+
+      document.getElementById("logoutBtn").addEventListener("click", async () => {
+        await fetch("/api/logout", { method: "POST" });
+        location.assign("/");
+      });
+      loadAnalytics();
+    </script>
+  </body>
+</html>

deploy/report-bucket-lifecycle.json

@@ -0,0 +1,8 @@
+{
+  "rule": [
+    {
+      "action": {"type": "Delete"},
+      "condition": {"age": 30}
+    }
+  ]
+}