test(openai): expand native Responses vLLM integration coverage by leseb · Pull Request #1129 · praxis-proxy/ai

leseb · 2026-09-11T14:58:04Z

Summary

Expands OpenAI SDK integration coverage for native Responses and Conversations, and splits deterministic protocol coverage from tests that require real model inference or vLLM-specific behavior.

Pull request coverage

Each SQLite and PostgreSQL simulator job runs 45 Responses tests and all 51 Conversations tests.
The 51 Conversations tests include six authenticated tenant-isolation scenarios. The test-only conversations_tenant_proxy is built only in these two simulator jobs.
The normal live CPU-vLLM pull-request lane remains at the eight critical tests introduced by ci: split Responses tests across simulator and vLLM #1195. This change adds no critical_vllm cases.

Full live coverage

The complete 85-test Responses SDK suite runs exclusively against real CPU vLLM:

Nightly at 04:17 UTC, clear of the repository nightly and CodeQL schedules.
On demand through workflow dispatch with run_live_vllm=true.
On demand for a pull request by adding the vllm-full-suite label.

A label-triggered run posts one PR comment with its Actions run ID when it starts and a second comment with the final result. The tests execute in the unprivileged pull_request context; a separate trusted workflow_run listener that never checks out PR code posts both comments with the praxis-bot-app token. The simulator jobs are skipped in every full-live mode; an unlabeled pull request continues to run only the eight critical live tests.

The added Responses scenarios are classified as follows:

real_inference: the nine-case structured-output schema matrix.
vllm_compat: finite and streaming logprobs, guided choice, response-field echoing, and invalid model, temperature, and tool-choice behavior.
Simulator: three stream-options/usage scenarios and the strict max_tool_calls validation xfail.

Simulator-compatible cases assert SDK lifecycle, persistence, usage, and response shapes without depending on exact generated text.

Conversations

The expanded Conversations suite covers:

Metadata and 20-item limits
Atomic batch failures and duplicate IDs
Function-call and function-call-output round trips
Cursor pagination and repeated-delete behavior
Item-to-conversation isolation
Full conversation lifecycle behavior
Tenant-scoped conversation and item access, identical item IDs across tenants, post-denial integrity, invalid credentials, and resistance to tenant-header spoofing

Validation

actionlint .github/workflows/vllm-integration.yaml
uv run python -m py_compile tests/integration/sdk/openai/conftest.py tests/integration/sdk/openai/test_openai_conversations.py tests/integration/sdk/openai/test_openai_responses_vllm.py
Simulator Responses selection: 45 tests collected
Critical live selection: 8 tests collected
Complete nightly/manual/label-triggered live selection: 85 tests collected
Conversations selection: 51 tests collected
cargo build -p praxis-ai-proxy
cargo build -p praxis-test-utils --example conversations_tenant_proxy
cargo test -p praxis-test-utils --example conversations_tenant_proxy — 1 passed
PRAXIS_TENANT_TEST_BIN=target/debug/examples/conversations_tenant_proxy uv run tests/integration/sdk/openai/test_openai_conversations.py -q — 51 passed

Notes

No production filter behavior changes
No new external dependencies
No recordings required

Add native-path Responses API integration cases to the vLLM SDK suite: structured-output schema shapes, logprobs (streaming and non-streaming), guided-choice extra_body, prompt_cache_key, top_p, truncation, parallel tool-call toggles, stream_options usage, and an error matrix for invalid model/max_tool_calls/temperature/tool_choice and conflicting previous_response_id + conversation. Six cases are strict xfails marking real parity gaps. Signed-off-by: Sébastien Han <seb@redhat.com>

Signed-off-by: Sébastien Han <seb@redhat.com>

franciscojavierarceo

the new selector-conflict test is still marked xfail(strict=True), but both vllm-responses jobs now show it as an XPASS, so the PR fails its required checks. the implementation already satisfies the assertion; we should remove this stale xfail marker and its outdated reason so the test is treated as a passing regression.

The conflicting previous_response_id + conversation validation already returns 400 before the previous-response lookup, so the case XPASSes under strict xfail and fails required checks. Remove the marker so it is treated as a passing regression test. Signed-off-by: Sébastien Han <seb@redhat.com>

test_conflicting_history_selectors_error_shape already covers the mutually exclusive previous_response_id + conversation conflict for both buffered and streaming requests and asserts the exact error body, so the standalone buffered-only test_conflicting_previous_response_and_conversation_raises_bad_request is duplicate coverage. Remove it. Signed-off-by: Sébastien Han <seb@redhat.com>

leseb · 2026-09-14T13:13:57Z

the new selector-conflict test is still marked xfail(strict=True), but both vllm-responses jobs now show it as an XPASS, so the PR fails its required checks. the implementation already satisfies the assertion; we should remove this stale xfail marker and its outdated reason so the test is treated as a passing regression.

fixed in 9fa06ae

franciscojavierarceo

the earlier selector-conflict failure is fixed and both vLLM jobs are green. one remaining expected-failure marker can hide a structured-output regression.

praxis-bot

PR Review: test(openai): expand native Responses vLLM integration coverage

Solid expansion of SDK integration coverage. The tenant isolation test adapter is well-designed: deterministic credentials, clean separation of concerns, and thorough cross-tenant boundary verification. The structured-output schema validation utility and parametrized cases are comprehensive.

Two findings below.

#	Severity	File	Finding
1	Medium	`test_openai_responses_vllm.py`	`stream_options` tests use non-standard field `include_obfuscation` instead of `include_usage`
2	Medium	`conversations_tenant_proxy.rs`	Doc comment on test function violates project convention

Run deterministic gateway and protocol coverage against llm-d-inference-sim in the SQLite and PostgreSQL jobs. Keep a focused CPU vLLM smoke suite on relevant changes, including critical OGX file resolution and file search scenarios. Leave the remaining live inference and compatibility cases available through workflow dispatch until GPU runners are available. Add test-only simulator adapters for deterministic Chat tool turns and hosted file search. Signed-off-by: Sébastien Han <seb@redhat.com>

Signed-off-by: Sébastien Han <seb@redhat.com>

Keep deterministic stream-options and max-tool-calls coverage in the simulator jobs. Route schema generation and vLLM-specific behavior to the live lanes, without adding critical PR cases.\n\nMake simulator assertions independent of exact generated wording while preserving lifecycle, usage, response-shape, and continuation checks. Signed-off-by: Sébastien Han <seb@redhat.com>

Keep the eight-test critical live lane on pull requests. Run all 85 Responses SDK cases once against real CPU vLLM on the nightly schedule or an explicit run_live_vllm dispatch. Skip both simulator jobs in full-live mode so scheduled and requested runs exercise only the real vLLM backend. Signed-off-by: Sébastien Han <seb@redhat.com>

Signed-off-by: Sébastien Han <seb@redhat.com>

Keep agentic-loop read_timeout_ms single-declaration while adopting main's translate_to_chat backend_endpoint wiring from praxis-proxy#1129. Signed-off-by: mkoushni <mkoushni@redhat.com>

leseb requested review from a team and crstrn13 September 11, 2026 14:58

leseb enabled auto-merge September 11, 2026 14:59

franciscojavierarceo approved these changes Sep 11, 2026

View reviewed changes

leseb disabled auto-merge September 11, 2026 15:05

leseb added 3 commits September 11, 2026 17:08

test(openai): expand conversations SDK coverage

0e58dcd

Signed-off-by: Sébastien Han <seb@redhat.com>

test(openai): add tenant-isolated conversations coverage

111ec96

Signed-off-by: Sébastien Han <seb@redhat.com>

test(openai): stabilize Responses vLLM coverage

b2c2036

Signed-off-by: Sébastien Han <seb@redhat.com>

leseb enabled auto-merge September 11, 2026 15:47

leseb disabled auto-merge September 11, 2026 16:02

test(openai): tolerate Qwen structured-output truncation

0d9d0a3

Signed-off-by: Sébastien Han <seb@redhat.com>

leseb added this pull request to the merge queue Sep 11, 2026

github-merge-queue Bot removed this pull request from the merge queue due to no response for status checks Sep 11, 2026

leseb added this pull request to the merge queue Sep 14, 2026

github-merge-queue Bot removed this pull request from the merge queue due to no response for status checks Sep 14, 2026

Merge branch 'main' into leseb/openai-responses-tests

7c988bf

franciscojavierarceo requested changes Sep 14, 2026

View reviewed changes

leseb added 2 commits September 14, 2026 11:56

leseb requested a review from franciscojavierarceo September 14, 2026 13:13

franciscojavierarceo requested changes Sep 14, 2026

View reviewed changes

Comment thread tests/integration/sdk/openai/test_openai_responses_vllm.py

praxis-bot reviewed Sep 14, 2026

View reviewed changes

Comment thread tests/integration/sdk/openai/test_openai_responses_vllm.py

Comment thread tests/utils/examples/conversations_tenant_proxy.rs

franciscojavierarceo approved these changes Sep 15, 2026

View reviewed changes

leseb added 3 commits September 16, 2026 10:58

Merge pull request praxis-proxy#1195 into praxis-proxy#1129

7b414c4

Signed-off-by: Sébastien Han <seb@redhat.com>

leseb mentioned this pull request Sep 16, 2026

ci: split Responses tests across simulator and vLLM #1195

Closed

leseb added 2 commits September 16, 2026 11:37

ci(vllm): trigger full suite from PR label

051f7eb

Signed-off-by: Sébastien Han <seb@redhat.com>

leseb enabled auto-merge September 16, 2026 09:48

github-advanced-security Bot found potential problems Sep 16, 2026

View reviewed changes

Comment thread .github/workflows/vllm-integration.yaml Fixed

Comment thread .github/workflows/vllm-integration.yaml Fixed

Comment thread .github/workflows/vllm-integration.yaml Fixed

ci(vllm): isolate label-triggered PR execution

111a6a7

Signed-off-by: Sébastien Han <seb@redhat.com>

leseb disabled auto-merge September 16, 2026 10:18

leseb merged commit 423fbea into praxis-proxy:main Sep 16, 2026
27 checks passed

leseb deleted the leseb/openai-responses-tests branch September 16, 2026 10:25

Navigation Menu

test(openai): expand native Responses vLLM integration coverage - #1129

test(openai): expand native Responses vLLM integration coverage#1129
leseb merged 14 commits into
praxis-proxy:mainfrom
leseb:leseb/openai-responses-tests

leseb commented Sep 11, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

franciscojavierarceo left a comment

Uh oh!

leseb commented Sep 14, 2026

Uh oh!

franciscojavierarceo left a comment

Uh oh!

Uh oh!

praxis-bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

leseb commented Sep 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Pull request coverage

Full live coverage

Conversations

Validation

Notes

Uh oh!

Uh oh!

Uh oh!

franciscojavierarceo left a comment

Choose a reason for hiding this comment

Uh oh!

leseb commented Sep 14, 2026

Uh oh!

franciscojavierarceo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

praxis-bot left a comment

Choose a reason for hiding this comment

PR Review: test(openai): expand native Responses vLLM integration coverage

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

leseb commented Sep 11, 2026 •

edited

Loading