Flaky-test triage: repetition-as-signal + Monitor 'deferred (nondeterministic)' disposition

[azalio]

Source loop: flaky-test-triage (loops.elorm.xyz)

What the loop does

Re-runs a failing suite N times, classifies each failure as flaky vs real, and fixes only confirmed regressions. Exit condition is non-binary: "every failure classified and real regressions fixed or explicitly deferred."

Why valuable for MAP

MAP currently treats a test/check as a single-pass deterministic verdict — one red run either hard-stops the workflow (Monitor valid=false) or, worse, pressures the Actor to "fix" a nondeterministic failure that isn't real. Neither path handles flakiness. Two additive ideas:

1. Repetition-as-signal — re-run a failing test K times to separate non-deterministic noise from a deterministic regression before acting.
2. A deferred (nondeterministic) disposition — a third Monitor outcome besides pass / hard-stop, with a recorded reason, so a confirmed flake doesn't force a false-green and doesn't block the run.

What MAP has today

• Deterministic gates + Monitor binary valid verdict.
• No flakiness classifier, no repeat-N-times, no defer-with-reason disposition.

Proposed scope

• Add an opt-in "re-run failing test K times" step in the test-gate path (config knob, default off or K=1).
• Extend the Monitor/gate verdict schema with a deferred disposition carrying reason + evidence (K runs, P pass / Q fail).
• Document: a flake must be classified and recorded, never silently skipped or test-weakened (reuse existing anti-gaming guardrails).

Notes

Single-source render invariant applies (edit .jinja, make render-templates). Honors "fix every surfaced error" — defer is recorded, not a silent skip.

---

Part of #251

[azalio]

Delivered first slice via merged PR #263.

Shipped:

• record_flaky_test_triage and validate_flaky_test_triage runner commands
• branch-scoped .map/<branch>/flaky_test_triage.json
• manifest stage flaky_test_triage
• dispositions: deferred_nondeterministic, deterministic_failure, not_reproduced, insufficient_evidence
• explicit monitor_verdict_policy=not_valid_without_explicit_triage plus operator requirements forbidding weaken/skip/delete/silent-green handling
• schemas, run-health inventory, docs, skill guidance, generated template copies, and regression tests

Verification:

• focused uv run pytest ...flaky_test_triage... passed
• make check passed: 2714 passed, 3 skipped, 12 deselected, generated trees match templates_src
• make test-e2e passed: 33 passed
• make test-integration passed: 33 passed, 12 deselected
• GitHub CI passed on build, validate-version, macOS/Ubuntu Python 3.11/3.12

Leaving this issue open because the merged slice records and validates repeated-run evidence, but does not yet add an automatic repeat-K command runner or wire a new non-binary disposition into the core Monitor/orchestrator verdict path.

[azalio]

Delivered next #252 slice via merged PR #266.

Shipped:

• automatic run_flaky_test_triage repeat-K runner
• argv execution with shell=False and explicit bash -lc escape hatch only when shell behavior is intentional
• bounded tempfile-backed stdout/stderr tail capture plus per-run timed_out and duration_seconds evidence
• CLI support for both trailing argv after -- and --command-json
• schema support for the new optional evidence fields
• Claude/Codex generated skill references plus docs/USAGE.md / docs/ARCHITECTURE.md
• regression tests for shell safety, mixed/all-pass classifications, timeout evidence, bounded tail capture, command-json validation, and trailing argv UX

Verification:

• uv run pytest tests/test_map_step_runner.py -q passed: 558 passed
• uv run pytest tests/test_map_step_runner.py tests/test_artifact_schemas.py -q passed: 574 passed
• make check passed: 2722 passed, 3 skipped, 12 deselected; generated trees match templates_src
• make test-e2e passed: 33 passed
• make test-integration passed: 33 passed, 12 deselected
• GitHub CI passed on build, validate-version, macOS/Ubuntu Python 3.11/3.12

Remaining follow-up before closing this issue: core Monitor/orchestrator non-binary deferred_nondeterministic disposition integration. llm-council recommended keeping that as a separate cross-cutting refactor after the runner slice, rather than half-wiring state-machine behavior in this PR.

[azalio]

Taking this issue now. I’m working on the remaining slice: wiring the core Monitor/orchestrator path so a classified deferred_nondeterministic flaky-test disposition is explicit non-green/non-hard-stop behavior, not a silent pass.

[azalio]

Беру следующий (и последний core) срез: wiring deferred_nondeterministic в verdict-путь Monitor/orchestrator.

Сейчас триаж-примитивы (run_/record_/validate_flaky_test_triage) и defer_flaky_subtask есть, но они разъединены с verdict-путём: Monitor отдаёт только valid: true/false, у него нет поля сигналить flaky-defer, а validate_step 2.4 не умеет роутить в deferral — оператор обязан в обход вызвать defer_flaky_subtask вручную. Третий исход — внешнее operator-суждение, а не часть структурного вердикта.

Срез (дизайн сверен с llm-council, conv d3ddca63):

• Monitor schema (monitor.md.jinja): опциональное структурное поле disposition: {kind, check_id} (kind enum {deferred_nondeterministic}), отсутствует для обычных вердиктов; guidance — эмитить при confirmed mixed pass/fail, а не требовать фейк-фикс от Actor.
• validate_step 2.4: ветка disposition между envelope-гейтом и recommendation-гейтом; делегирует в defer_flaky_subtask in-process (единый владелец close+advance транзакции).
• Anti-gaming (council-driven, must-have): defer чтится ТОЛЬКО если (a) sidecar валиден с mixed-evidence (защита от downgrade детерминированного фейла — уже есть), (b) disposition.check_id == sidecar.check_id, (c) Monitor valid:false + failed_checks непуст (failed_checks — это 11 dimension'ов, не check-id namespace, поэтому привязка адаптирована к схеме). Иначе valid:false hard-stop.
• Verdict vs routing (ключевая правка): deferred-ран возвращает valid:false + deferred:true + non_green_outcome:true (НЕ valid:true). Роутинг на deferred, CLI exit 0 на defer, skill-консьюмеры valid:false обновлены.
• SSOT-dict MONITOR_DISPOSITIONS + drift-guard тест; тесты three-way split; render 4 деревьев; docs (USAGE/ARCHITECTURE/CHANGELOG).

PR пойдёт с Part of #252 / Part of #251 — issue остаётся открытым до закрытия эпика.

[azalio]

✅ Core verdict-path slice landed via #299 (squash-merged to main, 1ff4d22).

This closes the remaining follow-up named in the prior comments — core Monitor/orchestrator non-binary deferred_nondeterministic disposition integration. The third Monitor outcome is now part of the structured verdict, not an out-of-band command:

• Monitor schema: optional structured disposition: {kind, check_id} (kind enum {deferred_nondeterministic}), absent for normal verdicts; guidance to emit on confirmed mixed pass/fail instead of demanding a fake Actor fix.
• validate_step 2.4 --disposition deferred_nondeterministic --check-id <id> --monitor-envelope - routes to the existing defer_flaky_subtask in-process (single owner of close+advance), BEFORE the recommendation gates so a defer with recommendation=needs_investigation is not hard-stopped.
• Anti-gaming: honored only when the envelope structurally backs it (valid:false + non-empty failed_checks + matching structured disposition) AND the sidecar holds mixed pass/fail evidence for that check_id (re-validated from disk). Can't dodge a deterministic failure or a green check; revise/block + disposition rejected as contradictory. (failed_checks = quality dimensions, a different namespace from a flaky check id, so the bind is admit-failure + dispositions-match.)
• Verdict vs routing: deferred run returns valid:false+deferred:true+non_green_outcome:true; CLI exits 0 on a deferral, 1 only on a true invalid verdict.
• SSOT MONITOR_DISPOSITIONS dict + drift-guard test; codex parity (separate source tree); docs.

Design llm-council-reviewed (conv d3ddca63). make check green (ruff/mypy/pyright 0, 2961 tests, render ✅) and GitHub CI green (build + validate-version + macos/ubuntu × 3.11/3.12).

Recommend closing #252 — all three proposed-scope items are now shipped: repeat-K runner (#266), deferred disposition in the Monitor verdict (#299), and docs. Leaving the close call to you per the epic-tracked-issue pattern.