Agentic RL with black-box harnesses in OpenEnv via LLM-call interception

[rycerzes]

Goal

Enable agentic RL through black-box agent harnesses (Pi, OpenCode, Claude Code, …) inside OpenEnv. The harness issues LLM calls at an OpenAI-compatible boundary; an interception server fronts a trainer-controlled engine, captures token-faithful traces of those calls, and feeds them to TRL — with rewards graded inside SWE environment. This is the installed-agent training path (train through the harness as-is), the counterpart to the external-agent pattern TRL's Harbor integration already covers.

The end state — one env, one agent, plus the core infra they need:

• swe_rl_env — the single SWE environment for this work (SWE-rebench V2 task substrate, sandboxed test execution, binary all-pass grading). Works black-box on its own.
• swe_rl_agent — the agent harness swe_rl_env drives (Pi, OpenCode, Claude Code, … via the RFC 005 CLIHarnessAdapter seam). Black-box: only its base URL + API key are injected.
• Sandbox backends in core — SandboxBackend/SandboxHandle/BgJob with E2B, Docker, and HF backends, reusable across envs.
• Interception + token-faithful trace (OpenEnv core) — an OpenEnv-hosted interception server (session-ID-as-API-key auth) fronting TRL's vLLM, and a trace recorder emitting prompt_ids, response_ids, loss_mask, response_logprobs, reward with prefix-break forking. TRL keeps generation (return_token_ids), advantages, weight sync, and IS correction; the seam is rollout_func via CLIHarnessAdapter.run_white_box.
• **A new RFC ** documenting the interception/trainer-integration design with cross-framework evidence (verifiers, AReaL, NVIDIA Polar, TRL-native rollout). Extends RFC 005's run_white_box seam (doesn't parallel it); records draft [Draft] core: harness-rollout contract + runnable example for online RL with TRL #864 as superseded prior art.

Ownership split with TRL: OpenEnv owns the env, sandbox backends, interception server, and token-faithful trace recorder; TRL owns generation (vLLM return_token_ids), advantages/GRPO, weight sync, IS correction, and chat-template correctness; the rollout_func seam is co-designed. The RFC and swe_rl_env (black-box) start now with no dependency; the interception + trace PRs land against TRL's stabilized rollout_func/vllm-serve seam and the RFC 005 core/harness/ runtime.

CC: @burtenshaw

[varshini2305]

@rycerzes Thanks for the detailed issue. I have some experience working with TRL and some exposure with OpenEnv, can I pick this up?

[rycerzes]

@rycerzes Thanks for the detailed issue. I have some experience working with TRL and some exposure with OpenEnv, can I pick this up?

Hey @varshini2305 thanks for showing interest, but I am already working on this PR (which will be mostly a refactor and standardization, you can refer to PR #694 #695 )

[varshini2305]

Any good open issues you'd recommend for a beginner in OpenEnv?

[rycerzes]

Any good open issues you'd recommend for a beginner in OpenEnv?

i think you can look into #63

[sergiopaniego]

hey, @rycerzes, small heads-up: #998 adds an HFSandboxBackend for the OpenCode harness (per-env for now, alongside the existing E2B one, same SandboxBackend / SandboxHandle / BgJob protocol). It overlaps the "sandbox backends in core" item on this roadmap, so just flagging so it's on your radar. Actually moving the protocol + the E2B/HF backends into core/harness/sandbox/ would be a separate follow-up PR (kept out of #998 to keep that one focused) 😄