A deterministic evaluation harness that tests whether scheduling agents correctly recover when concurrent confirmations invalidate pending proposals across shared resources.
This is an independent, synthetic evaluation inspired by Vela's public description of concurrent scheduling cascades. It does not test Vela's production system and does not claim to identify a Vela bug. Both agent architectures and the priority hierarchy are this repo's own and are labeled as such throughout.
Vela publicly describes scheduling environments where a staffing coordinator runs many interviews across shared interviewers: confirming one interview can invalidate a pending proposal on another thread. The system must detect the cascade, decide which proposal wins, invalidate the affected proposal, and re-propose an alternative.
I built the honest, measurable version of that failure surface: two deliberately different scheduling agents, a deterministic scenario generator, ten machine-checkable invariants, and real computed results. No number here is hand-written — every one is produced by the program (npm run evaluate).
Can a scheduling agent correctly handle concurrent scheduling cascades when a confirmed reservation invalidates pending proposals on other threads?
Concretely, can it detect the conflict, identify the invalidated proposal, preserve the confirmed reservation, resolve priority correctly, generate a valid alternative, and leave no conflicting confirmed state?
Actual output of npm run evaluate -- --seed 12345 --scenarios 500 (500 scenarios, 479 cascade scenarios, 713 conflicts generated):
Agent Detected Accuracy Confirmed DblBk PrioV RecFail FalseInv InvarFail
NaiveAgent 0 0.0% 1314 888 841 0 0 3505
CascadeAwareAgent 713 100.0% 1193 0 0 121 0 0
Determinism (1000 checks): PASS — byte-identical
| NaiveAgent (baseline) | CascadeAwareAgent (reference) | |
|---|---|---|
| Cascade detection | 0.0% | 100.0% |
| Double bookings | 888 | 0 |
| Priority violations | 841 | 0 |
| False invalidations | 0 | 0 |
| Invariant failures | 3505 | 0 |
| Recovery failures | 0 (never attempts) | 121 (true resource exhaustion) |
What this honestly shows: a baseline with no global pending-hold registry misses every cascade and lets losers get confirmed onto the same slot (888 double-bookings, 841 priority violations). A cascade-aware agent detects all 713 conflicts, preserves winners, and produces zero safety violations — while honestly showing 121 recovery failures where time is genuinely exhausted (e.g. three threads competing for the only slot). It cannot conjure time; that is a hard constraint, not a bug.
Real, regenerated artifacts: results/latest.json and results/analysis.md.
Two systems run over the same 500+ deterministic scenarios. They share identical slot-selection order, hard-constraint checking, and priority ordering — they differ in exactly one property: whether they maintain and reconcile a global view.
- System 1 — NaiveAgent (baseline). Treats each scheduling thread in isolation. No global pending-hold registry; slot choices ignore global confirmed state; a confirmation does not reconcile other threads.
- System 2 — CascadeAwareAgent (reference implementation — NOT a claim about Vela's architecture). Maintains a coherent global view: a pending-hold registry, a confirmed-reservation registry, conflict detection, priority-based ranking, invalidation of losers, preservation of confirmed reservations, and a re-proposal/recovery event.
Scenario classes: simple-cascade · multi-cascade · priority-conflict · stale-proposal · rescheduling-cascade · shared-resources · event-ordering · slot-exhaustion · no-conflict (control).
Thread A (Candidate A + Interviewer X) proposes Monday 10:00
Thread B (Candidate B + Interviewer X) proposes Monday 10:00
Thread A is confirmed => Thread B's proposal is now invalid
npm run replay -- --scenario pre-simple-thread-a-b prints exactly how both systems behave on this one.
Each is computed by the evaluator and returns structured failure lists (never a hollow "PASS"). I8 is checked by running the whole evaluation twice and comparing every trace byte-for-byte.
| ID | Invariant | Naive | Cascade |
|---|---|---|---|
| I1 | No double booking | 888 | 0 |
| I2 | Confirmed state consistency | 0 | 0 |
| I3 | Invalidation propagation | 888 | 0 |
| I4 | No stale belief | 0 | 0 |
| I5 | Priority consistency | 841 | 0 |
| I6 | Recovery (REPROPOSAL_REQUIRED emitted) | 0 | 0 |
| I7 | No false invalidation | 0 | 0 |
| I8 | Determinism | — | — |
| I9 | Event-trace consistency | 0 | 0 |
| I10 | Terminal consistency | 888 | 0 |
Priority is an injected, configurable hierarchy (default VIP_CLIENT > EXTERNAL_CLIENT > INTERNAL > PEER) — a synthetic convention of this experiment, not a claim about any real product's ordering.
npm install
npm test # 37 tests across 7 suites
npm run evaluate -- --seed 12345 --scenarios 500 # full eval + regenerates results/
npm run replay -- --scenario pre-simple-thread-a-b # replay a failing/successful tracesrc/
├── domain/ types.ts - states.ts (state machine + priority) - constraints.ts
│ events.ts - scenario.ts
├── engine/ agent.ts (AgentRun/SchedulingAgent) · world-state.ts (global resource state)
│ base-agent.ts · naive-agent.ts · cascade-aware-agent.ts · event-engine.ts
├── evaluator/ invariants.ts - metrics.ts - evaluator.ts
├── scenarios/ generator.ts (seeded multi-class) - predefined.ts
├── replay/ replay.ts
├── util/ random.ts (mulberry32 / fnv1a)
└── cli/ evaluate.ts - replay.ts
tests/ state-machine - cascade - invariants - priority - determinism - regression - replay (+ helpers)
results/ latest.json - analysis.md (generated)
Design decisions worth calling out:
- Determinism is structural, not accidental. All randomness comes from a seeded
mulberry32; the event engine owns a single global sequence counter; the hot path never callsDate.now(); traceseqis strictly increasing by construction. - One synthetic difference. The two agents differ only in "maintain and reconcile a global view", so the measured gap is cleanly attributable to cascade-awareness.
- Invariants are trace-based.
simulateCascades()reconstructs live pending holds over time and works out, for each confirmation, which other holds became impossible — agent-agnostic and replayable. - Honest failure model. No invariant is weakened to pass; the reference system is allowed to genuinely fail (recovery exhaustion), and those still count as recovery failures in the metrics.
npm test runs 37 tests with Vitest: thread state machine legality, canonical cascade behaviour of both systems, each invariant, priority conflict resolution, byte-identical determinism over 100 scenarios, replay stability, and regression guards over 250 generated scenarios. The tests target the evaluator itself; they do not weaken assertions to pass.
- Independent, synthetic — inspired by Vela's public description; does not test Vela's production system and makes no claim about a Vela bug.
- Both agent architectures are this repo's own;
CascadeAwareAgentis our reference implementation, not asserted to be Vela's architecture. - The priority hierarchy is a synthetic convention, injected and configurable.
- Slots are abstract integer time units; timezone/duration/recurrence semantics are out of scope.
- Scenario generation is deterministic but not a substitute for testing against real scheduling data.