fix: keep agent output streaming alive on an unreadable line, and honor LOG_LEVEL

[chakrris]

Summary

agentex agents run could leave an agent's worker frozen with no error, no traceback and no CPU use. Two defects combined to make that possible and then to make it undiagnosable, so both are fixed here.

The failure

An agent emitted an 81,988 character log line. asyncio's StreamReader defaults to 65,536 bytes, so readline() raised:

ValueError('Separator is found, but chunk is longer than limit')

stream_process_output is the only reader of the child's stdout pipe. The except Exception caught that, the loop exited, and 64 KiB later the child was wedged in a write() syscall. From the outside: 0% CPU, log output stops mid-run, health check stops answering, no traceback. A stack dump of the frozen process shows the main thread parked in _io_FileIO_write inside a logging call.

The handler did log the cause, with logger.debug. But make_logger pinned every logger to INFO and read no override, so that message could not be emitted on any configuration. The one signal that would have explained the freeze was unreachable by construction, which is why three separate freezes produced no clue.

What changed

src/agentex/lib/cli/handlers/run_handlers.py

• readline() and decode failures are handled per line rather than per loop, so one bad line cannot end the stream.
• The ValueError handler does not assume the cause. Skipping is only known-safe for a limit overrun, where readline() has already discarded the line and resumed the transport. readline() flattens LimitOverrunError into a bare ValueError, so the two are indistinguishable at the call site, and any other ValueError consumes nothing. Retrying that forever would spin at 100% CPU while still not draining the pipe, which is worse than the freeze being fixed. Consecutive failures are bounded at MAX_CONSECUTIVE_READ_ERRORS (100), then re-raised.
• The outer try/except is load-bearing rather than vestigial: it is the escalation path for that re-raise, it still covers console.print failures and non-ValueError transport errors, and it names the consequence. It does not swallow cancellation, since CancelledError derives from BaseException, so the auto-reload path that cancels these tasks is unaffected.
• All four spawn sites, normal and debug, pass limit=SUBPROCESS_STREAM_LIMIT (8 MiB), so ordinary large lines stream through instead of being dropped. Agents legitimately emit them: serialized charts, payloads echoed back by validation errors. SUBPROCESS_STREAM_LIMIT lives in cli/utils/cli_utils.py, which imports only typer and rich, because run_handlers imports cli.debug and so neither handler can export it to the other.

src/agentex/lib/utils/logging.py

• make_logger reads LOG_LEVEL from the environment, defaulting to INFO so nothing changes for anyone who does not set it.
• Unprefixed to match the SDK's other variables (ENVIRONMENT, REDIS_URL, AGENT_NAME), and read directly rather than through EnvVarKeys, because environment_variables imports this module and the reverse would be a cycle.
• getLevelName returns the string "Level FOO" for an unrecognized name, so an unusable value falls back to INFO rather than being handed to setLevel, where a typo would silently disable logging.

Review guide

Start with stream_process_output, and specifically the ValueError branch. The interesting question there is not whether skipping a line is safe, but that the handler cannot tell whether it is: readline() reports a limit overrun and any other failure as the same bare ValueError. The bound is what makes the unsafe case survivable rather than an infinite spin.

resolve_log_level in logging.py is the other crux. Worth knowing that make_logger has 94 call sites, so this does change behaviour for anyone who already sets LOG_LEVEL for their own application: they will now see SDK logs at that level. Keeping INFO as the default means no change for anyone who does not set it.

SUBPROCESS_STREAM_LIMIT is 8 MiB, which is a judgement call: large enough for the payload-shaped lines that caused this, small enough to bound memory per line. Happy to change it.

Tests are in tests/lib/cli/test_run_handlers_streaming.py and tests/lib/utils/test_logging_level.py, and each pins a behaviour that fails against the previous code: the child never exits because the reader stops draining its pipe; a reader that raises without consuming spins forever instead of giving up at the bound; cancelling the streaming task must still raise out of it; and the logger sees INFO where DEBUG was configured.

The limit and the retry bound have to stay in step, which is the one non-obvious coupling here. The bound counts every consecutive ValueError, including overruns that do make progress. That is harmless at 8 MiB and dangerous at 64 KiB: a spawn left on the default overruns easily enough that a burst of large lines can exhaust the bound, stop the reader, and deadlock the child. An earlier revision of this PR had exactly that gap, because the debug spawns kept the default. test_every_spawn_uses_the_larger_limit covers all four sites so a new spawn cannot be added on the default without failing.

Greptile Summary

This revision completes the large-line streaming fix by applying the shared 8 MiB stream limit to both debug subprocess helpers and extending the regression test across all four spawn sites.

• Moves SUBPROCESS_STREAM_LIMIT into a dependency-safe shared CLI utility module.
• Applies the larger stream limit to normal and debug subprocesses.
• Keeps output draining after isolated read or decoding failures while bounding repeated non-progressing errors.
• Honors LOG_LEVEL, with INFO as the fallback for missing or invalid values.
• Adds regression coverage for streaming continuity, cancellation, retry bounds, spawn configuration, and log-level resolution.

Confidence Score: 5/5

The PR appears safe to merge; the previously reported debug streaming freeze is fully addressed and no new actionable failures remain.

Both debug subprocess helpers now pass the enlarged stream limit, and the regression test verifies all four normal and debug spawn paths. The shared constant introduces no import cycle, while the logging and per-line streaming changes retain safe defaults and cancellation behavior.

Important Files Changed

Filename	Overview
src/agentex/lib/cli/debug/debug_handlers.py	Both debug subprocess helpers now use the shared enlarged stream limit, resolving the prior debug-mode freeze path.
src/agentex/lib/cli/handlers/run_handlers.py	Output streaming now tolerates isolated unreadable lines, bounds repeated non-progressing failures, and uses the shared subprocess stream limit.
src/agentex/lib/cli/utils/cli_utils.py	Defines the shared 8 MiB stream limit in a dependency-safe utility module.
src/agentex/lib/utils/logging.py	Resolves logger levels from LOG_LEVEL while safely defaulting missing or invalid values to INFO.
tests/lib/cli/test_run_handlers_streaming.py	Covers large-line draining, bounded read failures, cancellation propagation, and stream-limit configuration at every spawn site.
tests/lib/utils/test_logging_level.py	Covers default, normalized, invalid, and applied environment log levels.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Spawn agent subprocess] --> B[StreamReader with 8 MiB limit]
    B --> C{readline result}
    C -->|Valid bytes| D{UTF-8 decoding succeeds?}
    D -->|Yes| E[Print prefixed output]
    D -->|No| F[Warn and continue]
    C -->|ValueError| G[Increment consecutive error count]
    G --> H{More than 100?}
    H -->|No| B
    H -->|Yes| I[Warn that streaming stopped]
    E --> B
    F --> B
    C -->|EOF| J[Finish streaming]

Loading

_{Reviews (4): Last reviewed commit: "Address greptile: give the debug spawns ..." | Re-trigger Greptile}

[greptile-apps]

Debug streaming can still freeze

In debug mode, both subprocess helpers retain asyncio's 64 KiB stream limit. A sufficiently large newline-free log entry can therefore cause repeated limit-overrun errors. Once this counter exceeds 100, the only stdout reader stops without terminating the subprocess, so its pipe can fill and freeze the worker. The debug subprocesses need the larger limit too, or progress-making overruns must not count toward the cutoff for non-consuming failures.

Prompt To Fix With AI

This is a comment left during a code review.
Path: src/agentex/lib/cli/handlers/run_handlers.py
Line: 275-277

Comment:
**Debug streaming can still freeze**

In debug mode, both subprocess helpers retain asyncio's 64 KiB stream limit. A sufficiently large newline-free log entry can therefore cause repeated limit-overrun errors. Once this counter exceeds 100, the only stdout reader stops without terminating the subprocess, so its pipe can fill and freeze the worker. The debug subprocesses need the larger limit too, or progress-making overruns must not count toward the cutoff for non-consuming failures.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[chakrris]

Fixed in d0a1c2e. Both debug helpers now pass limit=SUBPROCESS_STREAM_LIMIT, so all four spawn sites use 8 MiB rather than asyncio's default.

The constant moved to cli/utils/cli_utils.py, which imports only typer and rich. It could not live in either handler, since run_handlers imports cli.debug and the reverse would cycle. That import problem is why the debug path was left out of the first revision.

On the alternative you offered, not counting progress-making overruns toward the cutoff: I looked at it and did not take it. readline() reports a limit overrun and any other failure as the same bare ValueError, and the buffer state that would distinguish them is private, so the only signal available is the exception message. Raising the limit everywhere fixes the reachable case without that.

test_every_spawn_uses_the_larger_limit now asserts all four sites, and fails with a spawn is missing limit=: [8388608, 8388608, 8388608, None] if one is added on the default.