Add Validate-DocsMsToc.ps1 guardrail to fail-closed on a corrupt overview ToC by shageev-msft · Pull Request #16328 · Azure/azure-sdk-tools

Pavel Shageev (shageev-msft) · 2026-07-13T00:45:03Z

What

Adds eng/common/scripts/Validate-DocsMsToc.ps1, a fail-closed structural
guardrail run immediately after Update-DocsMsToc.ps1 writes the unified
Docs.MS ToC. The validator exits non-zero (and the publish pipeline fails)
when the emitted toc.yml is structurally broken or suspiciously degraded.

Why

Follow-up to PR #15244 "Update-DocsMsToc.ps1 - Use powershell-yaml 0.4.12,
include fixes for breaking behavior change". That PR mitigated a specific
breaking change in powershell-yaml's ConvertTo-Yaml that caused
children items to be serialized as mappings instead of plain scalars. As a
result, on the affected publish, every named service node on the Azure SDK
for .NET overview was dropped on render and only the Other / Uncategorized Packages sink survived — on both Latest and Preview monikers.

Two contributing factors made this a silent, customer-detected failure
rather than a build-time failure:

Update-DocsMsToc.ps1 had no structural assertion on its own output, so
the corrupted YAML was committed and published as if it were valid.
Install-ModuleIfNotInstalled accepts any cached module version
>= requested, so a newer breaking powershell-yaml already present on
the agent was silently preferred over the requested version. (Tracked
separately — see follow-up note below.)

This PR closes the first gap: produce a loud, actionable failure at the
moment the bad bytes are emitted, before they hit the content repo.

What the validator catches

Code	Failure
TOC001	ToC file is empty
TOC002	ToC is not parseable YAML
TOC010	`children` was serialized as a mapping (the exact PR #15244 corruption shape) — detected on the raw YAML text
TOC011	A `children` item is not a scalar string
TOC020	Root node name is not `Reference`
TOC021	Root is not a mapping
TOC022	ToC is not a non-empty sequence
TOC030	Service node with empty name
TOC031	Service-level entry is not a named mapping
TOC040	Fewer than `MinTopLevelNodes` non-`Other` top-level nodes (the original incident's symptom)
TOC041	`Other` is the only top-level node
TOC050	Leaf-node count dropped > `MaxNodeCountDropPercent` % versus the previous good ToC (only if `-PreviousTocPath` set)

The TOC010/TOC011 check is intentionally done by raw-text scanning, not
just by ConvertFrom-Yaml, so it remains trustworthy even when the
powershell-yaml module installed on the agent has a behavioral regression
that broke ConvertTo-Yaml in the first place.

How it's wired

Update-DocsMsToc.ps1 calls the validator immediately after
Set-Content -Path $OutputLocation -Value $outputYaml. Non-zero exit fails
the pipeline. Every per-language repo that consumes eng/common picks the
gate up automatically through the standard sync flow, so the protection
applies to every language pipeline, not just .NET.

Rollout

Default behavior is enforce. For teams that want a calibration window
the validator accepts -SoftFail, which logs violations but exits 0.
Suggested rollout if needed:

Land this PR with -SoftFail in callers for one publish cycle.
Inspect telemetry / build logs for any unexpected hits on legitimate
intentional changes.
Remove -SoftFail from the call in Update-DocsMsToc.ps1 to enforce.

(The patch in this PR enforces from the start; the soft-fail path is left
available for the per-language repos to opt into during sync if their
custom hooks need it.)

Tests

New Pester suite under eng/common-tests/:

Validate-DocsMsToc.Tests.ps1 — six tests covering: happy path,
structural collapse (TOC040/TOC041), children-mapping corruption
(TOC010/TOC011), empty service name (TOC030), -SoftFail reporting mode,
and the drift gate (TOC050) both above and below threshold.
Validate-DocsMsToc.Fixtures/ — four small toc.yml fixtures, including
one that reproduces the exact PR Update-DocsMsToc.ps1 - Use powershell-yaml 0.4.12, include fixes for breaking behavior change #15244 corruption shape (children
items emitted as { Length: …, Chars: … } mappings).

Run locally:

Invoke-Pester ./eng/common-tests/Validate-DocsMsToc.Tests.ps1

Follow-up (separate PR, not in this one)

eng/common/scripts/Helpers/PSModule-Helpers.ps1 — moduleIsInstalled
currently treats a cached module as satisfying the request when its version
is >= requested, but installModule itself uses -RequiredVersion. That
asymmetry is what let a newer breaking powershell-yaml slip in. A
follow-up PR should tighten moduleIsInstalled to -eq semantics (or add
a -RequiredVersion switch) so the helper actually matches its own install
behavior.

Files changed

A eng/common/scripts/Validate-DocsMsToc.ps1
M eng/common/scripts/Update-DocsMsToc.ps1    (+8 lines at end of file)
A eng/common-tests/Validate-DocsMsToc.Tests.ps1
A eng/common-tests/Validate-DocsMsToc.Fixtures/good.yml
A eng/common-tests/Validate-DocsMsToc.Fixtures/bad-only-other.yml
A eng/common-tests/Validate-DocsMsToc.Fixtures/bad-children-mapping.yml
A eng/common-tests/Validate-DocsMsToc.Fixtures/bad-empty-name.yml

Adds a fail-closed structural guardrail run immediately after Update-DocsMsToc.ps1 writes the unified Docs.MS ToC. The validator exits non-zero when the emitted toc.yml is structurally broken or suspiciously degraded, plus Pester tests and fixtures.

github-actions · 2026-07-13T00:45:28Z

Thank you for your contribution Pavel Shageev (@shageev-msft)! We will review the pull request and get back to you soon.

Pavel Shageev (shageev-msft) · 2026-07-13T01:14:19Z

@microsoft-github-policy-service agree company="Microsoft"

Copilot

Pull request overview

This PR adds a new fail-closed validation step for the unified Docs.MS toc.yml generated by Update-DocsMsToc.ps1, aiming to prevent structurally corrupted or suspiciously degraded ToCs from being published. It introduces a dedicated validator script plus a Pester test suite with fixtures to cover key corruption and collapse scenarios.

Changes:

Add Validate-DocsMsToc.ps1 to validate emitted ToC YAML (raw-text corruption detection + parsed structural checks + optional drift gate).
Wire the validator into Update-DocsMsToc.ps1 immediately after writing toc.yml, failing the pipeline on violations.
Add Pester tests and YAML fixtures covering happy path, known corruption shapes, structural collapse, soft-fail behavior, and drift detection.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
eng/common/scripts/Validate-DocsMsToc.ps1	New guardrail script to validate generated Docs.MS ToC structure and detect known corruption patterns.
eng/common/scripts/Update-DocsMsToc.ps1	Invokes the validator after writing the unified `toc.yml` and fails publishing on validation errors.
eng/common-tests/Validate-DocsMsToc.Tests.ps1	New Pester suite exercising validator behavior across key failure modes and options.
eng/common-tests/Validate-DocsMsToc.Fixtures/good.yml	Valid ToC fixture for baseline passing behavior.
eng/common-tests/Validate-DocsMsToc.Fixtures/bad-only-other.yml	Fixture representing structural collapse to only “Other”.
eng/common-tests/Validate-DocsMsToc.Fixtures/bad-children-mapping.yml	Fixture reproducing the “children rendered as mappings” corruption shape.
eng/common-tests/Validate-DocsMsToc.Fixtures/bad-empty-name.yml	Fixture for empty service-name detection.

Pavel Shageev (shageev-msft) · 2026-07-15T00:12:06Z

+$badChildPattern = '(?ms)^\s*children:\s*\r?\n(?:\s*-\s*(?:Length|Chars|value)\s*:|\s*-\s*!!?map\b)'
+if ($rawYaml -match $badChildPattern) {
+  Add-Violation 'TOC010' `
+    "`children` was serialized as a mapping (probable powershell-yaml regression). First match near offset $($Matches[0].Length)."
+}


Fixed in 3fcf3a3 - switched to [regex]::Match(...) and now report the actual .Index.

Pavel Shageev (shageev-msft) · 2026-07-15T00:12:08Z

+$childBlockPattern = '(?ms)^( +)children:\s*\r?\n((?:\1 +- .*\r?\n)+)'
+foreach ($m in [regex]::Matches($rawYaml, $childBlockPattern)) {
+  $block = $m.Groups[2].Value
+  foreach ($line in $block -split "`r?`n") {
+    if ($line -match '^\s*-\s*\{' -or $line -match '^\s*-\s*\S+\s*:\s*\S') {
+      Add-Violation 'TOC011' "`children` item is not a scalar string: '$($line.Trim())'"
+      break
+    }
+  }
+}


Fixed in 3fcf3a3 - relaxed the pattern to \1 *- (same or greater indent) and constrained the wildcard to a single line ([^\r\n]*) so unrelated items: blocks are no longer swept in. TOC011 now actually fires on bad-children-mapping.yml.

Pavel Shageev (shageev-msft) · 2026-07-15T00:12:09Z

+if ($LASTEXITCODE -ne 0) {
+    Write-Error "Validate-DocsMsToc reported violations; refusing to publish a corrupt ToC."
+    exit $LASTEXITCODE
+}


Fixed in 3fcf3a3 - reindented to 2 spaces to match surrounding style.

- TOC010: report actual regex match offset (was reporting match length) - TOC011: fix childBlockPattern so items at same indent as children: are captured; also constrain wildcard to a single line so unrelated items: blocks are not swept in - Update-DocsMsToc.ps1: use 2-space indent to match surrounding style

github-actions Bot added Community Contribution Community members are working on the issue customer-reported Issues that are reported by GitHub users external to the Azure organization. labels Jul 13, 2026

Merge branch 'main' into pavelshageev/guard-docsms-toc-publish

9d7a15f

Pavel Shageev (shageev-msft) marked this pull request as ready for review July 13, 2026 01:14

Pavel Shageev (shageev-msft) requested a review from a team as a code owner July 13, 2026 01:14

Copilot AI review requested due to automatic review settings July 13, 2026 01:14

Copilot started reviewing on behalf of Pavel Shageev (shageev-msft) July 13, 2026 01:15 View session

Copilot AI reviewed Jul 13, 2026

View reviewed changes

Navigation Menu

Uh oh!

Add Validate-DocsMsToc.ps1 guardrail to fail-closed on a corrupt overview ToC - #16328

Add Validate-DocsMsToc.ps1 guardrail to fail-closed on a corrupt overview ToC#16328
Pavel Shageev (shageev-msft) wants to merge 3 commits into
Azure:mainfrom
shageev-msft:pavelshageev/guard-docsms-toc-publish

Pavel Shageev (shageev-msft) commented Jul 13, 2026

Uh oh!

github-actions Bot commented Jul 13, 2026

Uh oh!

Pavel Shageev (shageev-msft) commented Jul 13, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Pavel Shageev (shageev-msft) Jul 15, 2026

Uh oh!

Pavel Shageev (shageev-msft) Jul 15, 2026

Uh oh!

Pavel Shageev (shageev-msft) Jul 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

Pavel Shageev (shageev-msft) commented Jul 13, 2026

What

Why

What the validator catches

How it's wired

Rollout

Tests

Follow-up (separate PR, not in this one)

Files changed

Uh oh!

github-actions Bot commented Jul 13, 2026

Uh oh!

Pavel Shageev (shageev-msft) commented Jul 13, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Pavel Shageev (shageev-msft) Jul 15, 2026

Choose a reason for hiding this comment

Uh oh!

Pavel Shageev (shageev-msft) Jul 15, 2026

Choose a reason for hiding this comment

Uh oh!

Pavel Shageev (shageev-msft) Jul 15, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants