Add missing schema validation fixtures for async, OpenAPI, and workflow steps#476
Add missing schema validation fixtures for async, OpenAPI, and workflow steps#476nashjain wants to merge 5 commits intoOAI:v1.1-devfrom
Conversation
Add seven fixture-driven schema tests covering async step validation that was already encoded in the v1.1 schema but not explicitly exercised by the suite. The new fail fixtures cover mutually exclusive operation targets, missing action on channelPath steps, and dependsOn validation for empty arrays, invalid reference syntax, and duplicate entries. The new pass fixtures cover receive steps without correlationId and reusable parameter references on async steps. Each negative fixture was authored from an otherwise-valid baseline and checked by applying the minimal invalidating change, so the failures remain attributable to the intended schema branch rather than unrelated missing fields or malformed top-level structure.
Add five fixture-driven schema tests covering OpenAPI step validation paths that were already enforced by the v1.1 schema but were not explicitly exercised by the test suite. The new pass fixtures cover operationPath-only steps and reusable parameter references for OpenAPI operations. The new fail fixtures cover mutually exclusive operationId/operationPath targets, missing operation targets, and duplicate parameter entries. Each negative fixture was validated against a repaired baseline first so the final failing YAML is attributable to the intended schema branch rather than unrelated structural mistakes.
Add six fixture-driven schema tests covering workflow-step validation paths that were already enforced by the v1.1 schema but were not explicitly exercised by the suite. The new pass fixtures cover the minimal valid workflow-step shape and reusable parameter references on workflow steps. The new fail fixtures cover missing workflowId, duplicate workflow-step parameters, and rejection of async-only fields such as action and channelPath on workflow steps. Each negative fixture was validated against a repaired baseline first so the final failing YAML remains attributable to the intended schema branch rather than unrelated document-shape problems.
Add six pass fixtures that intentionally capture current JSON Schema validator behavior in cases where structural validation succeeds without proving full semantic compliance. These characterization cases cover source-type mismatches between OpenAPI and AsyncAPI step usage, permissive operationPath and channelPath string handling, and the current acceptance of requestBody on workflow steps because requestBody is defined on step-object-base. Also add a short informative note to the specification text clarifying that JSON Schema validation alone does not establish full Arazzo conformance and that some constraints require source resolution or other semantic validation across referenced documents.
Drop the recently added note in src/arazzo.md while keeping the characterization fixtures unchanged.
frankkilcommins
left a comment
frankkilcommins
left a comment
There was a problem hiding this comment.
Looks good, thanks @nashjain. Good to have more tests in this area.
It's good for all to remember the following statement in ~/Arazzo-Specification/src/schemas/validation/README.md:
As a reminder, the JSON Schema is not the source of truth for the Specification.
In cases of conflicts between the Specification itself and the JSON Schema, the
Specification wins. Also, some Specification constraints cannot be represented
with the JSON Schema so it's highly recommended to employ other methods to
ensure compliance.
| # Invalid on purpose: dependsOn accepts a plain local stepId (`sendEvent`), | ||
| # `$workflows.<workflowId>.steps.<stepId>`, or | ||
| # `$sourceDescriptions.<name>.<workflowId>.steps.<stepId>`, but not `$steps.<stepId>`. | ||
| - $steps.sendEvent |
There was a problem hiding this comment.
@frankkilcommins/ @nashjain
I remember there was a discussion about what context $step.STEP_ID refers to.
Would $workflows..steps. be equivalent to $steps. when referencing a step within the current workflow, or is there a difference in scope or resolution?
There was a problem hiding this comment.
It would be the same, and the expectation is that evaluation would work similarly. When within the context of a workflow, $steps. refers to steps within the current workflow only. You could still use $worklows.<workflowId>.steps to get you to the same list assuming the is the current workflow.
| workflows: | ||
| - workflowId: parentWorkflow | ||
| steps: | ||
| - stepId: nestedWorkflow |
There was a problem hiding this comment.
Please check test name, it seems misleading, probably smth about empty step should be mentioned.
| workflows: | ||
| - workflowId: parentWorkflow | ||
| steps: | ||
| # Characterization test: this passes today because requestBody is defined |
There was a problem hiding this comment.
Seems like this test is false positive and don't really check anything except the lack of possible verification. Do we need to have it still?
3 participants
Summary
Adds fixture coverage for schema validation branches that were already enforced by the Arazzo v1.1 schema but were not explicitly exercised by the test suite.
This PR expands coverage in four areas:
What’s added
Async fixtures
Adds coverage for:
operationIdandchannelPathmutual exclusivityactionon asyncchannelPathstepsdependsOnempty array rejectiondependsOnreference syntax rejectiondependsOnentry rejectionreceivestep withoutcorrelationIdOpenAPI fixtures
Adds coverage for:
operationPath-only stepsoperationIdandoperationPathoperationIdandoperationPathWorkflow step fixtures
Adds coverage for:
workflowIdactionon workflow stepschannelPathon workflow stepsCharacterization fixtures
Adds pass fixtures that document current validator behavior where structural schema validation succeeds even though deeper semantic validation would still be required:
asyncapioperation step withoutactionactioncorrelationIdoperationPathstring handlingchannelPathstring handlingrequestBodyThese are intentional characterization tests for current behavior, not normative conformance examples.
Notes
For each new negative fixture, I first verified an otherwise-valid baseline and then introduced the single invalidating change. This was done to ensure failures are attributable to the intended schema branch rather than unrelated document-shape issues.
Validation
npm test62/62tests passing