Headless enhancement: add stream-json as input-format/output-format to support programmatically use
#926
+14,074
−456
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
feat: enhance build process and update .gitignore for Python caches
feat: add support for stream-json format and includePartialMessages flag in CLI arguments
feat: add StreamJsonWriter and associated types for structured JSON streaming
feat: implement stream-json session handling and control requests
Implement control request handling and refactor related functions - Added `handleIncomingControlRequest` method to `StreamJsonController` for processing control requests. - Created `input.test.ts` and `session.test.ts` to test control request handling. - Refactored `runStreamJsonSession` to delegate control requests to the controller. - Moved `extractUserMessageText` and `writeStreamJsonEnvelope` to a new `io.ts` file for better organization. - Updated tests to ensure proper functionality of control responses and message extraction.
Add user envelope handling in runNonInteractive function
Add tests for runStreamJsonSession and enhance session handling - Implement tests for runStreamJsonSession to validate user prompts and message handling. - Improve session termination logic to ensure all active runs are awaited. - Log user prompts with additional metadata for better tracking.
chore: update .gitignore to remove Python cache entries
x22x22
force-pushed
the
feature/stream-json-migration
branch
from
ea0fb43 to
567b73e
Compare
Mingholy
changed the title
stream-json as input-format/output-format to support programmatically use
- remove SDK files - moved to mingholy/feat/sdk-draft branch - revert package.json & launch.json changes
…n-code into feature/stream-json-migration
tanzhenxin
approved these changes
Nov 12, 2025
| }); | ||
|
|
||
| it('should return a valid JSON with response and stats', async () => { | ||
| it('should return a valid JSON array with result message containing response and stats', async () => { |
Collaborator
There was a problem hiding this comment.
It would be better if we can have some integration tests for the new stream-output feature.
|
|
||
| # Log files | ||
| patch_output.log | ||
| QWEN.md |
Collaborator
There was a problem hiding this comment.
Do not filter out QWEN.md here. It is designed to be git managed in nature, otherwise if would be in .qwen folder.
|
|
||
| const newToolCalls: ToolCall[] = requestsToProcess.map( | ||
| (reqInfo): ToolCall => { | ||
| // Check if the tool is excluded due to permissions/environment restrictions |
Collaborator
There was a problem hiding this comment.
If a tool is excluded using tools.excludes configuration, it would not registered and fallback to the logic below. So this logic may be not necessary.
| ); | ||
| this.setStatusInternal(reqInfo.callId, 'scheduled'); | ||
| } else { | ||
| /** |
| "types": "./dist/index.d.ts", | ||
| "import": "./dist/index.js" | ||
| }, | ||
| "./protocol": { |
Collaborator
There was a problem hiding this comment.
Cannot find this file.
|
|
||
| await runNonInteractiveStreamJson( | ||
| nonInteractiveConfig, | ||
| settings, |
Collaborator
There was a problem hiding this comment.
A question here:
Should we ignore configuration like settings.json by default like Claude Agent SDK? It might be a good idea for SDK.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
TLDR
This PR implements a comprehensive Stream JSON migration that enables structured JSON I/O for the CLI, supporting both standard JSON and streaming JSON formats. It introduces a new non-interactive mode framework with session management and extensive test coverage. Key additions include:
--output-format stream-jsonand--input-format stream-jsonoptions for bidirectional JSON communicationBaseJsonOutputAdapter,JsonOutputAdapter,StreamJsonOutputAdapter) for flexible output handlingImportant for reviewers: This PR focuses on JSON/stream-json output functionality and resolves all known issues. SDK code has been moved to a separate branch to keep this PR focused on CLI output format capabilities.
Known Issues
is_errorstate of failing tool usemessage_deltaevent emittingTodo Actions
nonInteractive/types.ts, align them with claude-code-like types to gain compatibilitygemini.tsxandnonInteractiveCli.tsfor better support of the arguments matrix:[input-format, output-format] × [text, json, stream-json]Control Plane Architecture (Under Construction)
The control plane components are included in this PR to provide the foundation for bidirectional communication between the CLI and external SDK clients. However, reviewers should focus primarily on the JSON output format functionality, which is the main focus of this PR.
Control Plane Components
control/ControlContext.ts): Shared session state (Layer 1)control/ControlDispatcher.ts): Protocol-level routing and message handling (Layer 2)control/ControlService.ts): Programmatic API for internal CLI usage (Layer 3)control/controllers/): Modular controllers for different domainsDive Deeper
Background: Existing Implementation
The previous commits (task1-2-2 through task1-2-5) implemented initial stream-json support in
packages/cli/src/streamJson/:types.ts: Message envelopes and type definitionswriter.ts: StreamJsonWriter for output serializationcontroller.ts: StreamJsonController for control request handlinginput.ts: Input parsing and validationsession.ts: Session management (runStreamJsonSession)io.ts: I/O utilities (extractUserMessageText, writeStreamJsonEnvelope)What's New: Non-Interactive Framework Architecture
This PR introduces a comprehensive non-interactive mode framework in
packages/cli/src/nonInteractive/:1. Non-Interactive Mode Framework (
packages/cli/src/nonInteractive/):session.ts):SessionManagerhandles multi-turn conversations in stream-json mode2. JSON Output Adapters (
nonInteractive/io/):BaseJsonOutputAdapter (
BaseJsonOutputAdapter.ts): Core functionality for message serialization, block management, and state tracking (1,172 lines)JsonOutputAdapter (
JsonOutputAdapter.ts): Standard JSON output for single-request responses--output-format jsonStreamJsonOutputAdapter (
StreamJsonOutputAdapter.ts): Streaming JSON that emits messages incrementally--include-partial-messages--output-format stream-jsonStreamJsonInputReader (
StreamJsonInputReader.ts): Reads JSON protocol messages from stdin in stream-json mode3. Protocol Types (
nonInteractive/types.ts):4. Configuration Enhancements:
--input-format(text/stream-json),--include-partial-messages(boolean)-iflag → JSON formats with query/prompt → TTY detectionpackages/cli/src/config/config.tspackages/core/src/config/config.tspackages/core/src/output/types.ts5. Core Integration:
coreToolSchedulerto support permission denial trackingKey Features
CLIPermissionDenialinterface for tracking denied tool callsTesting
Extensive test coverage added:
BaseJsonOutputAdapter.test.tsStreamJsonOutputAdapter.test.tsJsonOutputAdapter.test.tsControlDispatcher.test.tssession.test.tsStreamJsonInputReader.test.tsReviewer Test Plan
Basic Functionality Tests
Standard JSON Output:
Verify: Output is valid JSON with complete assistant message
Stream JSON Output:
Verify: Multiple JSON lines emitted, each line is valid JSON
Stream JSON with Partial Messages:
Verify: Partial messages are included in the stream
Stream JSON Input Mode:
Verify: CLI reads and processes JSON input correctly
Advanced Scenarios
Multi-Turn Session:
Permission Denial Tracking:
Error Handling:
Integration Tests
Run the existing test suite:
npm testFocus on:
packages/cli/src/nonInteractive/io/*.test.ts: JSON adapter testspackages/cli/src/nonInteractive/session.test.ts: Session management testspackages/cli/src/nonInteractive/control/ControlDispatcher.test.ts: Control dispatcher testspackages/cli/src/nonInteractiveCli.test.ts: Enhanced test coveragepackages/core/src/core/coreToolScheduler.test.ts: Permission tracking testsKey Areas to Review
New Architecture Implementation:
nonInteractive/io/BaseJsonOutputAdapter.ts: Core message serialization and state managementnonInteractive/io/JsonOutputAdapter.ts: Standard JSON output implementationnonInteractive/io/StreamJsonOutputAdapter.ts: Streaming JSON output implementationnonInteractive/io/StreamJsonInputReader.ts: JSON input parsingnonInteractive/session.ts: Session state machine and message routingnonInteractive/types.ts: Protocol message typesIntegration Points:
nonInteractiveCli.tswith format detectiongemini.tsx)Testing:
Testing Matrix
Note: Primary testing done on macOS with
npm runandnpx.Linked issues / bugs
This PR implements a comprehensive Stream JSON migration and non-interactive framework refactoring originally proposed in #810
Development Context:
Phase 0 (commits task1-2-2 through task1-2-5): Initial stream-json implementation
packages/cli/src/streamJson/Phase 1 (this PR): Non-Interactive Framework Architecture
nonInteractive/directory with comprehensive frameworknonInteractive/types.tsFuture Phases: