Skip to content

feat(observability): emit quality signals and a write-journey SLO - #428

Open
sylphx-desk-studio[bot] wants to merge 3 commits into
mainfrom
w18294-quality-signals
Open

sylphx-desk-studio[bot] wants to merge 3 commits into
mainfrom
w18294-quality-signals

Conversation

@sylphx-desk-studio

@sylphx-desk-studio sylphx-desk-studio Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

What and why

Reworks this existing PR for the review finding that read-only Connect POSTs dilute write failures.

  • Emit puzzled.user.write.ok|failed only for explicitly enumerated user-mutating Connect RPCs. Read-only Connect POSTs, operator/job/Compute calls, bootstrap and induction are excluded. HTTP 5xx fails this availability journey; client rejection does not.
  • Replace the old method-only puzzled.api.write.* events; infra#3360 selects the dedicated semantic events.
  • Keep grouped issue events. The induction handler carries its nonce subject to the outer observer, so one induced 503 emits one fingerprint, not a second route fingerprint.
  • Document scope and the paired SLO. Add the mixed-traffic regression: ten failed submissions plus 1,000 successful reads still count ten writes and ten failures.

Verification

A local isolated smoke harness imports the production signal/induction modules directly because the initial remote repository test command timed out without executing a test. The semantic regression failed before the emitter change; repository gates and an actual-binary HTTP smoke subsequently passed locally.

The QA-owned post-deploy acceptance remains: call /signals:induce once and then again with the same nonce; read exactly one Work item and a repeat note. Local emission checks do not claim a production Work readback.

  • Repository unit/binary tests: 350 core and 291 server tests passed, zero failures.
  • CI clippy command (-D warnings with its two existing allowed lints) and cargo fmt --check passed.
  • Actual product binary built and served HTTP locally: read-only GetTodayProgress POST returned 200 with no journey line; empty SubmitGuess returned 400 with one availability-ok write; two induced failures returned 503 with exactly two occurrences of one nonce fingerprint and no induction write events.
  • Isolated exact-production-module suite: seven tests passed. The mixed-traffic regression was failing before the change and passes afterward.

Paired PR: SylphxAI/infra#3360. No migrations.

The api reports its own problems as structured log lines (owner
standards/quality-signals.md): one puzzled.issue.turn_failed.<route> line per
5xx, so the platform files one work item per fingerprint, and one
puzzled.api.write.ok|failed line per finished write request, the events of a
journey SLO. POST /signals:induce (key-guarded, changes nothing) is the
production readback.

@sylphx-desk-studio sylphx-desk-studio Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The production induction check does not satisfy the one-issue acceptance, and ordinary RPC failures lose their identity. Please fix: 1. crates/puzzled-server/src/bootstrap/router.rs:57 - Add /signals:induce to the api path_prefixes in sylphx.toml; the documented public URL currently routes to the web service, not this handler. 2. crates/puzzled-server/src/bootstrap/signals_induce.rs:36 - Emit only one issue for an authorized induction. This explicit emission is followed by a second fingerprint from observe() when the handler returns 503. Test the composed router so a repeat produces only the same fingerprint. 3. crates/puzzled-server/src/shared/signals.rs:130 - Preserve bounded registered Connect service/method identity for fallback RPC requests; they have no Axum MatchedPath, so unrelated RPC 5xx failures currently collapse into puzzled.issue.turn_failed.unknown. Non-blocking: docs/metrics.md:65 inserts prose before the Revenue table row; move that paragraph below the complete table. CI is green for the reviewed head; the current tests exercise synthetic matched routes rather than these production composition paths.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant