Skip to content

Conversation

@jpower432
Copy link
Contributor

@jpower432 jpower432 commented Nov 4, 2025

Description

This PR introduces support for multiple assessment evaluators in Layer 4 Evaluation Plans, enabling tools and manual methods to execute assessment procedures with configurable conflict resolution strategies when multiple evaluators provide results. This add a new directory spec to documentation how to use the Layer 4 schema for finding determination (how it could be used by Layer 5 tools). This PR precedes Layer 5 to ensure Layer 5 tools can form an opinion about the returned result.

Schema Changes

Schema Changes Made

  • No schema changes
  • Layer 1 schema (schemas/layer-1.cue) changes
  • Layer 2 schema (schemas/layer-2.cue) changes
  • Layer 3 schema (schemas/layer-3.cue) changes
  • Layer 4 schema (schemas/layer-4.cue) changes

Schema Change Details

Fields Added:
Evaluators - Introducing an Evaluator concept that can represent both automated tools and manual review processes.
Confidence Level - Allow Layer4 tools to attach confidence level to a specific result in the AssessmentLog
EvaluationDocument - Adds a new top-level object to represent Layer 4 EvaluationDocument to ensure the pattern is consistent for each layer.
ConflictResolution - Providing three distinct strategies that user can choose based on their security posture and tool maturity

Field Modified:
AssessmentPlan - Add optional conflict resolution between procedures
Evaluation Plan - Added required evaluators
AssessmentLog - Added optional confidence level
AssessmentProcedures - Added required evaluator mapping

Field Removed:
None

Consumer Impact: The new evaluator fields in Evaluation Plan are requirement making this change not backward compatible.

Testing

  • Unit tests added/updated
  • Manual testing performed
  • Test data updated (if applicable)

Run cue vet -d "#EvaluationDocument" -s schemas/layer-4.cue layer4/test-data/multi-tool-plan.yaml

Related Issues

Partially resolves #170
Closes #175

Reviewer Hints

The new multi-tool-plan.yaml test data demonstrates configuring multiple executors with different roles and conflict resolution strategies:

executors:
  - id: "primary-scanner"
    type: Automated
    authoritative: true
  - id: "experimental-tool"
    type: Automated
    
conflict_resolution:
  strategy: AuthoritativeConfirmation

Assisted by: Cursor AI

@jpower432 jpower432 marked this pull request as ready for review November 12, 2025 01:11
@jpower432 jpower432 requested a review from a team as a code owner November 12, 2025 01:11
@jpower432
Copy link
Contributor Author

Documenting some feedback discussed with @eddie-knight:

  • Adding confidence scores to AssessmentLog to allow a tool express confidence in a particular result. Related to [Question] Should we support confidence levels in the Layer 4 EvaluationLog? #175.
  • Explore confidence scores for the ExecutorMapping on how much a tool is trusted performing a particular procedure
  • Layer4 is breaking a pattern because it has two top level objects (Plan and Log). Create a top-level document that would serve an input into Layer5.

@jpower432 jpower432 force-pushed the feat/layer4-tools branch 3 times, most recently from 8221cff to 98eae49 Compare November 15, 2025 04:02
@jpower432 jpower432 marked this pull request as draft November 20, 2025 22:01
@jpower432
Copy link
Contributor Author

Temporarily converting to draft while working through feedback

Layer 4 currently breaks the pattern established by
the other layers. This add a top-level documentation
that can be passed into Layer 5 for full context to
execute an enforcement action.

Introduce support for multiple assessment executors
(tools/manual methods) that can execute assessment procedures,
with configurable conflict resolution strategies when multiple executors provide results.

Signed-off-by: Jennifer Power <[email protected]>
Adding confidence to EvaluationLog allows an evaluation to
return a confidence level in the result based on inputs

Signed-off-by: Jennifer Power <[email protected]>
Assisted by: Cursor Agent
Signed-off-by: Jennifer Power <[email protected]>
The finding specificiation details how to use the EvaluationPlan
and deconfliction strategies to determine findings for Layer5
action.

Assisted by: Cursor Agent
Signed-off-by: Jennifer Power <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Question] Should we support confidence levels in the Layer 4 EvaluationLog? Enhance multi-source evaluation support in Layer 4/5

2 participants