Skip to content

feat(scripts): add instruction applyTo scope Pester suite#1640

Open
WilliamBerryiii wants to merge 2 commits into
mainfrom
sibling/planner-linter-hardening
Open

feat(scripts): add instruction applyTo scope Pester suite#1640
WilliamBerryiii wants to merge 2 commits into
mainfrom
sibling/planner-linter-hardening

Conversation

@WilliamBerryiii
Copy link
Copy Markdown
Member

Pull Request

Description

This PR adds a single Pester regression guard, scripts/tests/linting/Test-InstructionApplyToScope.Tests.ps1, that enforces phase-narrowed applyTo globs on every instruction file under .github/instructions/security/** and .github/instructions/rai-planning/**. Any file whose frontmatter declares applyTo: '**' is rejected unless its repo-relative path is on the in-script allowlist (currently empty, matching the Phase 6 audit-matrix migration map).

The suite is data-driven (It -ForEach) so each scanned file produces a discrete test case (currently 18 cases, all passing). It uses powershell-yaml (already required by scripts/linting/Validate-MarkdownFrontmatter.ps1 and other existing suites — no new repo-level dependency) and explicitly throws on missing frontmatter so silent skips cannot mask regressions.

This PR is a sibling of #1497, branched off main. It is independent of PRs A/C/B in the post-#1497 stack and can land in any order relative to them.

Scope reduction

The originating plan listed four Pester suites under "planner linter hardening." Three are deferred to PR B because they assert content not yet on main:

  • Rule 5 "discover" keyword in security/identity.instructions.md and sssc-identity.instructions.md.
  • ## Startup H2 in six prompts under .github/prompts/security/ and .github/prompts/security/sssc/.
  • "Informational" bucket in security-model.instructions.md.

Shipping those tests now would produce a red base. They will land in PR B alongside the content edits they exercise.

Related Issue(s)

Sibling of #1497. No other issue references.

Type of Change

Code & Documentation:

  • New feature (non-breaking change adding functionality)

Infrastructure & Configuration:

  • Linting configuration (markdown, PowerShell, etc.)

Other:

  • Script/automation (.ps1, .sh, .py)

Testing

Automated validation performed by the agent:

  • npm run test:ps -- -TestPath scripts/tests/linting/Test-InstructionApplyToScope.Tests.ps118/18 passed.
  • npm run lint:ps — clean for repository code.

Security analysis findings:

  • No secrets, credentials, or customer data in the diff.
  • No new runtime dependencies. powershell-yaml is already required by existing linting and test suites.
  • No changes that broaden privilege boundaries; the test reads files only.

Diff-based assessments:

  • The new file follows the existing scripts/tests/linting/Test-*.Tests.ps1 naming and structure.

Note

Manual testing was not performed.

Checklist

Required Checks

  • Documentation is updated (if applicable) — (N/A — passive regression test.)
  • Files follow existing naming conventions.
  • Changes are backwards compatible.
  • Tests added for new functionality (if applicable).

Required Automated Checks

  • PowerShell analysis: npm run lint:ps

Security Considerations

  • This PR does not contain any sensitive or NDA information
  • Any new dependencies have been reviewed for security issues — (N/A — no new dependencies.)
  • Security-related scripts follow the principle of least privilege — (N/A — read-only test.)

Additional Notes

Companion PRs in the post-#1497 work stream:

@WilliamBerryiii WilliamBerryiii requested a review from a team as a code owner May 23, 2026 21:45
@codecov-commenter
Copy link
Copy Markdown

codecov-commenter commented May 23, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 85.49%. Comparing base (1102901) to head (e5f1a84).

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #1640      +/-   ##
==========================================
- Coverage   85.50%   85.49%   -0.01%     
==========================================
  Files          82       82              
  Lines       11805    11805              
==========================================
- Hits        10094    10093       -1     
- Misses       1711     1712       +1     
Flag Coverage Δ
pester 83.65% <ø> (-0.02%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Document that planner-related Pester tests are intentionally split between
`scripts/tests/linting/` (rule validators) and `scripts/tests/security/`
(artifact signing). Consolidation under a single `planner/` subdirectory
is deferred to backlog item IV-011.

Reference: .copilot-tracking/reviews/2026-05-22/stacked-prs-from-pr-1497-review.md (IV-011)
}
}
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Pester instructions require -Tag 'Unit' on all Describe blocks. Without it, CI runs filtered by -Tag Unit will skip this suite silently.

Describe 'Phase-narrowed applyTo regression guard' -Tag 'Unit' {

Comment on lines +20 to +27
$script:cases = foreach ($root in $script:scanRoots) {
if (-not (Test-Path $root)) { continue }
Get-ChildItem -Path $root -Filter '*.md' -Recurse -File | ForEach-Object {
@{
Path = $_.FullName
Rel = ($_.FullName.Substring($script:repoRoot.Length + 1) -replace '\\','/')
}
}
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

$script:cases is populated at file scope before the Pester container runs. If either scan root is absent (renamed, shallow clone, etc.), the continue silently skips it. With zero cases, -ForEach produces zero It executions and Pester reports a green suite, so the guard gives no signal that it is covering nothing.

Consider moving the population into BeforeAll and asserting the collection is non-empty:

BeforeAll {
    # ... existing setup ...
    $script:cases = foreach ($root in $script:scanRoots) {
        if (-not (Test-Path $root)) { throw "Scan root not found: $root" }
        Get-ChildItem -Path $root -Filter '*.md' -Recurse -File | ForEach-Object {
            @{ Path = $_.FullName; Rel = ($_.FullName.Substring($script:repoRoot.Length + 1) -replace '\\','/') }
        }
    }
    $script:cases | Should -Not -BeNullOrEmpty -Because 'at least one instruction file must be discovered'
}

#>

$script:repoRoot = (Resolve-Path (Join-Path $PSScriptRoot '../../..')).Path
Import-Module powershell-yaml -Force -ErrorAction SilentlyContinue
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Import-Module powershell-yaml -Force -ErrorAction SilentlyContinue appears twice: once at file scope (line 13) and again inside BeforeAll (line 31). Both calls use -ErrorAction SilentlyContinue, so a missing module fails silently. The first visible error would be ConvertFrom-Yaml: The term 'ConvertFrom-Yaml' is not recognized, which obscures the actual cause.

The idiomatic fix is a single #Requires directive at the top of the file, replacing both calls:

#Requires -Modules Pester, powershell-yaml
# Copyright (c) Microsoft Corporation.
# SPDX-License-Identifier: MIT

Comment thread scripts/tests/README.md
Comment on lines +122 to +124
Consolidation into a single `planner/` subdirectory is tracked as backlog
item IV-011 in
`.copilot-tracking/reviews/2026-05-22/stacked-prs-from-pr-1497-review.md`.
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The sentence references .copilot-tracking/reviews/2026-05-22/stacked-prs-from-pr-1497-review.md, but .copilot-tracking/ is gitignored and never committed. The IV-011 ID is an Implementation Validator finding number that lives in that same local, unshared tree. Both references are inaccessible to any other contributor after merge.

Two options to consider:

Option A: Remove the sentence. The preceding lines already convey the rationale for the split, and the tracking reference adds no durable value.

Option B: Open a GitHub issue to track the consolidation work and replace the sentence with a link to it, giving any contributor a live pointer they can actually follow.

Which approach works for you?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants