WebhookTrigger v2: provider-agnostic template verifier#1186
Closed
WebhookTrigger v2: provider-agnostic template verifier#1186
Conversation
Adds a declarative signing-template engine that covers every major HMAC-family
webhook provider — GitHub, Stripe, Slack, Shopify, Linear, Svix / Standard
Webhooks, Mailgun, PayPal, Clerk, and Twilio-style URL+param signing — with
zero provider-specific code in DM core.
What's new
- WebhookVerifier (new): single verify() entry point driven by `signed_template`
placeholders (`{body}` `{timestamp}` `{id}` `{url}` `{header:X}` `{param:X}`)
plus four extract kinds (`raw`, `prefix`, `kv_pairs`, `regex`). Signature
encodings: hex, base64, base64url. Algos: sha1, sha256, sha512.
- WebhookVerificationResult (new): structured result object with outcome codes
(ok, bad_signature, missing_header, missing_signature, missing_timestamp,
stale_timestamp, no_active_secret, payload_too_large, malformed_template,
malformed_config, unknown_mode) and diagnostic fields (secret_id, timestamp,
skew_seconds, detail).
- WebhookAuthResolver (new): compatibility layer. Accepts v2 `webhook_auth`
blocks, v2 `webhook_auth_preset` names (+ optional `webhook_auth_overrides`),
or legacy v1 fields — all resolve to a canonical verifier config.
- Replay protection: `timestamp_source` + `tolerance_seconds` reject any
request whose timestamp skew exceeds the window.
- Multi-secret rotation: `secrets: []` array with optional `expires_at`.
Any active secret that verifies wins; expired entries auto-skipped.
- Preset filter registry: `datamachine_webhook_auth_presets` — third parties
register provider shorthands. Core ships zero presets.
- Mode filter registry: `datamachine_webhook_verifier_modes` — plugins can
add non-HMAC modes (Ed25519, x509, JWT, mTLS) that core doesn't ship.
Ability surface (+ 3 new)
- executeEnable extended: `preset`, `secret_id` input fields.
- executeRotateSecret (new): zero-downtime rotation. Demotes current →
previous with a TTL (default 7d), installs a fresh current.
- executeForgetSecret (new): remove a specific secret id immediately.
- executeTest (new): offline verifier dry-run against a supplied body +
headers. No job spawn, no rate limiter touched.
- executeStatus: surfaces auth_mode, preset, and secret_ids (values never
exposed).
CLI surface
- `wp datamachine flows webhook enable --preset=<name>` (+ `--secret-id`).
- `wp datamachine flows webhook rotate <id> --generate [--previous-ttl-seconds=N]`.
- `wp datamachine flows webhook forget <id> <secret_id>`.
- `wp datamachine flows webhook test <id> --body=@file.json --header="X: y"`.
- `wp datamachine flows webhook presets` — list filter-registered presets.
- Status / list / enable output includes auth mode, preset, secret roster.
Integration
- WebhookTrigger::handle_trigger now routes all non-bearer auth through the
verifier. Structured `datamachine_webhook verification <reason>` log lines
include secret_id, timestamp, skew, and preset — no secrets or signatures.
Backward compatibility
- Every v1 field keeps working byte-for-byte. The resolver expands legacy
`webhook_auth_mode=hmac_sha256` + `webhook_signature_header` +
`webhook_signature_format` into the v2 template shape at read time.
- Legacy single-value `webhook_secret` continues to work alongside the new
`webhook_secrets[]` list; set-secret / rotate maintain both.
- Bearer flow is untouched.
Tests (+ 70 new tests; 1214/1214 relevant pass, same 37 pre-existing
unrelated failures as baseline)
- WebhookVerifierTest: 38 tests. Data-driven provider matrix for 9 providers
× (valid / tampered body / wrong secret) plus replay, rotation, expiry,
malformed-template, unknown-mode, URL+param, unix_ms timestamp.
- WebhookAuthResolverTest: 11 tests. v1 bearer / v1 hmac_sha256 expansion,
v2 pass-through, preset lookup, preset+overrides deep-merge, unknown
preset fallback, scheduling_config secret inheritance.
- WebhookTriggerV2Test: 17 tests. End-to-end REST handler with a Stripe
preset (valid / wrong signature / stale timestamp), rotation lifecycle
(previous keeps verifying until forget), offline test ability, status
never leaks secrets.
Docs
- docs/api/endpoints/webhook-triggers.md: new "v2 template-based verifier"
section with config grammar, provider coverage table, preset filter
example, rotation workflow, offline-test recipe, non-HMAC extension
point.
- docs/core-system/wp-cli.md: updated webhook command block.
- docs/core-system/abilities-api.md: +3 webhook abilities.
Out of scope (future follow-ups)
- Dedicated `{prefix}_webhook_secrets` table with migration.
- Structured verification log table.
- AgentPing outbound signing using the same template grammar.
- Nonce-based replay storage beyond timestamp window.
- Concrete Ed25519 / x509 implementations for the mode registry.
Refs: #1179
Contributor
Homeboy Results —
|
Member
Author
|
Closing without merging — see #1179 thread. Provider-specific knowledge leaked back in via v1 compat defaults (GitHub-style header/format defaults, hardcoded safe-header whitelist, enum switch in the resolver). Rebuilding clean: the v2 template config will be the ONLY HMAC shape, v1 migrates silently once, safe-headers becomes a pattern-based deny-list, no provider names anywhere in core. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1179.
Summary
Replaces the
signature_formatenum (v1:sha256=hex | hex | base64) with a declarative signing template + extraction rules engine that handles every HMAC-family provider in the wild — with zero provider-specific code in DM core.The insight
v1 was "GitHub + Shopify + Linear and that's it." Stripe, Slack, Svix, Mailgun, PayPal, and Clerk all sign composite strings (timestamp + body, or id + timestamp + body, or URL + params) that can't be expressed as a single format enum.
v2 describes how a provider signs instead of which provider is signing:
signed_template— placeholders: `{body}` `{timestamp}` `{id}` `{url}` `{header:X}` `{param:X}`signature_source— where the signature lives (header or param), with four extract kinds: `raw`, `prefix`, `kv_pairs`, `regex`timestamp_source— optional; its presence enables replay protectionsecrets: []— multi-secret rotation with optional `expires_at`tolerance_seconds— replay windowProof of coverage (one engine, nine providers, zero provider-specific code)
Every row below is exercised end-to-end by the PHPUnit provider matrix:
Plus a Twilio-style `{url}{param:From}{param:To}` test that exercises URL + body-param placeholders.
What's in the box
New classes
Filter-based extension points (core ships zero provider names)
Multi-secret rotation
```bash
wp datamachine flows webhook rotate 42 --generate [--previous-ttl-seconds=N]
wp datamachine flows webhook forget 42 previous
```
Both `current` and `previous` verify inbound signatures until `previous` expires or is forgotten — zero-downtime swap window.
Offline dry-run
```bash
wp datamachine flows webhook test 42 \\
--body=@fixtures/github-ping.json \\
--header="X-Hub-Signature-256: sha256=..."
```
Runs the verifier against captured payloads. No job spawn, no rate-limit state touched. Prints the verification outcome including which secret matched and the extracted timestamp skew.
Replay protection
Presence of `timestamp_source` + `tolerance_seconds > 0` rejects any request whose timestamp skew exceeds the window. Tested at fresh, stale, and unix_ms formats.
Backward compatibility
Zero behavior change for any shipped flow. Concretely:
v1 fields remain the documented shortcut for simple single-header providers — they're just sugar over the v2 engine now.
Tests
```
homeboy test data-machine -- --filter='Webhook'
OK (101 tests, 204 assertions)
```
+70 new tests, 0 new regressions against baseline:
Full-suite: 1255 total, 1214 passed, 37 pre-existing failures unchanged (NetworkSettings, ImportExportStepConfig — same baseline as #1178). Zero new regressions.
Lint
Clean for new/modified files. The only phpstan finding attributable to `WebhookTriggerAbility` comes from the shared `FlowHelpers` trait and pre-dates this PR.
Docs
Out of scope (future issues)
Verification
```bash
cd /var/lib/datamachine/workspace/data-machine@webhook-v2-template-verifier
homeboy lint data-machine --changed-only --summary # clean for our files
homeboy test data-machine -- --filter='Webhook' # 101/101
homeboy test data-machine # no new failures vs baseline
```