Skip to content

docs(frd): add FRD 0005 for web_request system tool#87

Closed
larohra wants to merge 7 commits into
mainfrom
larohra-http-call-system-tool
Closed

docs(frd): add FRD 0005 for web_request system tool#87
larohra wants to merge 7 commits into
mainfrom
larohra-http-call-system-tool

Conversation

@larohra

@larohra larohra commented Jul 9, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Why

Tracks the design for a built-in web_request system tool (issue #1176) so agents can call HTTP/REST endpoints directly instead of generating and running code. This PR is the FRD only (medium+ feature pipeline, AGENTS.md §1) — no product code yet. The FRD stays In review, gated on human sign-off before implementation.

What's here

docs/frds/0005-web-request-system-tool.md — the full Feature Requirements Document: motivation, proposed design mapped to the discover -> translate -> register -> execute pipeline, an always-on SSRF security floor, a security analysis (trust model, SSRF validator, header/redirect/secret handling, execution-surface boundary, reflection residual, and an out-of-process-custody / ACA egress-proxy alternatives study), an append-only Decisions log, a test plan, and docs impact.

Naming (latest)

Renamed the tool and config key http_call -> web_request, and the scheme flag allow_http (default false) -> require_https (default true, inverted polarity). This drops the overloaded "http" (tool name vs URL scheme) and the misread where allow_http: false looked like it disabled a now default-on tool. The FRD file slug was renamed to match; Decisions row J8 records the change, and historical log rows keep the original identifiers (append-only).

Revision (scope reduction + default-on)

  • v1 is narrowed to a public, unauthenticated fetch primitive. Deferred to v2/v3+: per-host auth + secret custody, redirect following, the per-agent override object, wildcard/IDNA host matching, and full telemetry parity. The always-on SSRF floor + https-only + caps still ship in v1. A "Phased delivery (v1/v2/v3+)" table sets the cut line.
  • web_request is now on by default. The reduced, SSRF-floored public-fetch scope makes it safe as a standard built-in (unlike the opt-in sandbox, which needs an Azure resource). Operators opt out with system_tools.web_request: false (app-wide), per-agent web_request: false, or the tools: false kill-switch. This is a behavior change on upgrade, called out in Compatibility.
  • Examples restructured. Section 4 now shows a v1-minimal authoring surface inline; the full auth/wildcard/override example moved to a "Target authoring surface (once all versions ship)" subsection marked illustrative-not-v1. Security subsections and the test plan carry (v1) / (v2) tags so readers aren't told to configure things the runtime rejects.
  • Decisions J1-J8 appended to the log; historical rows are untouched (append-only). Prior arch-review rows remain Agent-decided and pending human confirmation.

Status

In review — ready for human sign-off. Not flipped to Finalized; no implementation until sign-off.

Refs #1176

larohra and others added 5 commits July 8, 2026 22:49
@larohra
docs(frd): add FRD 0004 for http_call system tool
bc69183 
Design doc for a built-in, opt-in http_call system tool that lets agents invoke HTTP endpoints declaratively instead of code-generating requests in the sandbox. Includes SSRF security floor, per-host auth via typed env secret-refs, per-agent override model, and full telemetry parity with the sandbox.

Incorporates two independent architecture reviews (gpt-5.4 + gpt-5.5). Status: In review, pending human sign-off.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@larohra
docs(frd): capture secret-custody & egress-proxy analysis in FRD 0004
225d9dd 
Fold the follow-up security discussion into FRD 0004 for the http_call system tool:

- New section 4 subsections: 'Execution surfaces & the in-worker-code boundary'
  (model code runs only in the ACA sandbox, never the worker, with no in-worker
  fallback; tools/*.py is the sole in-worker author-code path, in-trust-domain and
  not model-reachable at runtime; standing invariant + author guideline),
  'Residual threat: reflection / confused-deputy exfiltration' (named, with a
  mitigation ladder), and 'Alternatives considered: out-of-process secret custody'
  (ACA egress-proxy pros/cons table, why v1 is independent of it, compute re-host
  rejection, and a tiered custody roadmap).
- New section 5 Decisions rows I1-I4 (Human-decided, 2026-07-09).
- Section 7 docs impact + section 8 review record updated.

No product-code or scope changes; the sign-off gate is unchanged.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@larohra
docs(frd): WIP v1 scope reduction (summary/motivation/goals)
7cc1899 
In-progress: reduce v1 to a public, unauthenticated fetch primitive (SSRF floor + caps + exact-host allowlist + minimal telemetry). Summary, motivation, and Goals updated; Non-goals/Phased-delivery table, section 4 v2 tags, and Decisions rows still pending in a follow-up commit.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 6c80e793-f72f-4629-ba8e-16e5847dbe53
@larohra
Merge origin/main into larohra-http-call-system-tool
4b85b89 
Resolved docs/frds/README.md index conflict. main added FRD 0004 (Dynamic workflows, Finalized), which collides with this branch's FRD 0004 (http_call). Renumbered the http_call FRD to 0005 (git mv + frontmatter + heading) so main's 0004 owns the slot; README lists both 0004 and 0005.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 6c80e793-f72f-4629-ba8e-16e5847dbe53
@larohra
docs(frd): reduce v1 scope and make http_call default-on
ff4ecd4 
Narrow FRD 0005 v1 to a public, unauthenticated fetch primitive and make the tool on-by-default (opt out via `http_call: false` or `tools: false`). Restructure the section 4 examples to a v1-minimal surface, moving the full auth/wildcard/override example into a 'once all versions ship' subsection. Tag the section 4 security subsections and section 6 tests with v1/v2 markers, append Decisions J1-J7 (scope reduction + default-on), and refresh the docs-impact and status sections. FRD stays In review; the human sign-off gate is unchanged.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 6c80e793-f72f-4629-ba8e-16e5847dbe53
@larohra larohra changed the title docs(frd): add FRD 0004 for http_call system tool docs(frd): add FRD 0005 for http_call system tool Jul 11, 2026
@larohra
docs(frd): rename http_call system tool to web_request
7146190 
Rename the FRD 0005 tool and config key http_call -> web_request, and the
scheme flag allow_http (default false) -> require_https (default true,
inverted polarity). This removes the overloaded "http" (tool name vs URL
scheme) and the misread where allow_http: false looked like it disabled a
now default-on tool.

Also renames the FRD file slug, updates the docs/frds README index, and
appends Decisions row J8. Historical log rows A1-J7 intentionally keep the
original identifiers (append-only log); a naming note points to J8.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 6c80e793-f72f-4629-ba8e-16e5847dbe53
@larohra larohra changed the title docs(frd): add FRD 0005 for http_call system tool docs(frd): add FRD 0005 for web_request system tool Jul 11, 2026
@larohra
docs(frd-0005): trim web_request FRD to v1 scope
c8ae6bf 
Remove deep v2/v3+ why-analysis prose that duplicated the append-only Decisions log: drop the Trust-model subsection, collapse Redirect/Secret-custody/Execution-surface/Reflection/egress-proxy-alternatives (~200 lines) into a compact 'Deferred to v2/v3+' notes block, and cut the review-history retrospective from section 8. Also compress the section 2 motivation and section 3 non-goals prose.

Section 5 Decisions log (A1-J8) is preserved verbatim per the append-only convention; the full rationale remains there and in git history. No design change; FRD stays In review. 865 -> 594 lines.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 6c80e793-f72f-4629-ba8e-16e5847dbe53
@larohra larohra mentioned this pull request Jul 13, 2026
Merged
@larohra

larohra commented Jul 13, 2026

Copy link
Copy Markdown
Contributor Author

Superseded by #96, which combines this FRD (0005) with the v1 implementation, tests, and docs in a single feature PR to main. Closing this FRD-only PR to consolidate review there.

@larohra larohra closed this Jul 13, 2026
larohra added a commit that referenced this pull request Jul 13, 2026
@larohra
docs(frd): finalize FRD 0005 web_request (sign-off + PR #96)
86c9fc7 
Human sign-off received: flip status In review -> Finalized, confirm the Agent-decided rows, add PR #96 to pull_requests, and record the sign-off / PR-consolidation decision as append-only row K1. Update section 8 to reflect completed implementation (587 tests) shipped via #96; PR #87 closed as superseded.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 6c80e793-f72f-4629-ba8e-16e5847dbe53
larohra added a commit that referenced this pull request Jul 14, 2026
@larohra
feat: web_request system tool (v1) + FRD 0005 (#96)
5092ca8 
* docs(frd): add FRD 0004 for http_call system tool

Design doc for a built-in, opt-in http_call system tool that lets agents invoke HTTP endpoints declaratively instead of code-generating requests in the sandbox. Includes SSRF security floor, per-host auth via typed env secret-refs, per-agent override model, and full telemetry parity with the sandbox.

Incorporates two independent architecture reviews (gpt-5.4 + gpt-5.5). Status: In review, pending human sign-off.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* docs(frd): capture secret-custody & egress-proxy analysis in FRD 0004

Fold the follow-up security discussion into FRD 0004 for the http_call system tool:

- New section 4 subsections: 'Execution surfaces & the in-worker-code boundary'
  (model code runs only in the ACA sandbox, never the worker, with no in-worker
  fallback; tools/*.py is the sole in-worker author-code path, in-trust-domain and
  not model-reachable at runtime; standing invariant + author guideline),
  'Residual threat: reflection / confused-deputy exfiltration' (named, with a
  mitigation ladder), and 'Alternatives considered: out-of-process secret custody'
  (ACA egress-proxy pros/cons table, why v1 is independent of it, compute re-host
  rejection, and a tiered custody roadmap).
- New section 5 Decisions rows I1-I4 (Human-decided, 2026-07-09).
- Section 7 docs impact + section 8 review record updated.

No product-code or scope changes; the sign-off gate is unchanged.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* docs(frd): WIP v1 scope reduction (summary/motivation/goals)

In-progress: reduce v1 to a public, unauthenticated fetch primitive (SSRF floor + caps + exact-host allowlist + minimal telemetry). Summary, motivation, and Goals updated; Non-goals/Phased-delivery table, section 4 v2 tags, and Decisions rows still pending in a follow-up commit.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 6c80e793-f72f-4629-ba8e-16e5847dbe53

* docs(frd): reduce v1 scope and make http_call default-on

Narrow FRD 0005 v1 to a public, unauthenticated fetch primitive and make the tool on-by-default (opt out via `http_call: false` or `tools: false`). Restructure the section 4 examples to a v1-minimal surface, moving the full auth/wildcard/override example into a 'once all versions ship' subsection. Tag the section 4 security subsections and section 6 tests with v1/v2 markers, append Decisions J1-J7 (scope reduction + default-on), and refresh the docs-impact and status sections. FRD stays In review; the human sign-off gate is unchanged.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 6c80e793-f72f-4629-ba8e-16e5847dbe53

* docs(frd): rename http_call system tool to web_request

Rename the FRD 0005 tool and config key http_call -> web_request, and the
scheme flag allow_http (default false) -> require_https (default true,
inverted polarity). This removes the overloaded "http" (tool name vs URL
scheme) and the misread where allow_http: false looked like it disabled a
now default-on tool.

Also renames the FRD file slug, updates the docs/frds README index, and
appends Decisions row J8. Historical log rows A1-J7 intentionally keep the
original identifiers (append-only log); a naming note points to J8.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 6c80e793-f72f-4629-ba8e-16e5847dbe53

* docs(frd-0005): trim web_request FRD to v1 scope

Remove deep v2/v3+ why-analysis prose that duplicated the append-only Decisions log: drop the Trust-model subsection, collapse Redirect/Secret-custody/Execution-surface/Reflection/egress-proxy-alternatives (~200 lines) into a compact 'Deferred to v2/v3+' notes block, and cut the review-history retrospective from section 8. Also compress the section 2 motivation and section 3 non-goals prose.

Section 5 Decisions log (A1-J8) is preserved verbatim per the append-only convention; the full rationale remains there and in git history. No design change; FRD stays In review. 865 -> 594 lines.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 6c80e793-f72f-4629-ba8e-16e5847dbe53

* config+observability: add web_request system tool schema, resolution, and telemetry

Adds WebRequestConfig, SystemToolsConfig.web_request, SystemToolsAgentOverride.web_request,
and ResolvedAgent.web_request_config to config/schema.py; adds default-on _resolve_web_request()
to config/merge.py; adds FaultDomain.WEB_REQUEST and record_web_request() counter plumbing to
_observability.py.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 53035269-5774-46e6-a0bc-a7eba5601967

* system_tools: implement web_request outbound HTTP tool (v1)

Default-on, unauthenticated, single-request web_request tool with an always-on SSRF
security floor: exact-host allowlist, global-unicast-only IP validation via an injectable
async resolver, IP pinning through a custom aiohttp resolver/connector (DNS caching
disabled), request-header denylist, response-header redaction, and capped/truncating
response reads. Config is clamped to operator-defined absolute ceilings (timeout 120s max,
request/response body 10MB max; defaults: 30s timeout, 5MB max response, 1MB max request).
Built once per agent at registration (no Azure resource required).

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 53035269-5774-46e6-a0bc-a7eba5601967

* wire web_request tool through registration, runner, and app

Adds AgentCapabilities.web_request_tools (built once per agent, lazily importing the
factory), threads a dedicated web_request_tools parameter through runner.run_agent()/
run_agent_stream() parallel to sandbox_tools, passes it at every registration/_handlers.py
and registration/endpoints.py call site, tracks 'web_request' in app.py's system_tools_used
reporting (respecting default-on + global/per-agent disable), and exports
create_web_request_tools from the package __init__.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 53035269-5774-46e6-a0bc-a7eba5601967

* tests: add comprehensive coverage for the web_request system tool

New tests/test_web_request.py: param-schema validation, response shaping (JSON/text/
binary/HEAD/truncation), the full SSRF suite (IMDS/loopback/private/link-local/ULA/CGNAT/
unspecified/multicast/reserved, IPv4-mapped IPv6 unwrap, non-canonical numeric IP literal
rejection, embedded-userinfo rejection, exact-host allowlist, https floor), IP pinning
(validated IPs, DNS cache disabled, Host/SNI preserved), header policy, telemetry
(span + counter, no query/secret leakage), config clamping, and default-on registration.
New tests/test_config_merge.py cases for _resolve_web_request's default-on resolution.
New fixtures/config_scenarios/14_web_request scenario plus a test_config_fixtures.py case
exercising global config + per-agent opt-out through compose(). Updates
test_package_imports.py and test_registration_capabilities.py for the new export/field.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 53035269-5774-46e6-a0bc-a7eba5601967

* docs: document the web_request system tool

Regenerates docs/front-matter-reference.md (schema-driven) after adding WEB_REQUEST_DESCRIPTIONS
and a system_tools.web_request table to eng/scripts/generate_config_reference.py. Adds a
system_tools.web_request section with default-on/disable/opt-out examples to
front-matter-spec.md (v2 items called out as future work), updates docs/architecture.md's
module map and pipeline narrative for the new default-on, built-once-at-registration tool,
adds a web_request span/metric/fault-domain reference to docs/observability.md, and adds a
one-line quickstart mention to README.md.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 53035269-5774-46e6-a0bc-a7eba5601967

* test: harden web_request SSRF/header/capability regression coverage

Address gpt-5.5 FIX-THEN-SHIP validation feedback (test-only, no product
code changes):

- tests/test_web_request.py: add explicit regression tests for
  trailing-dot allowlist normalization (single dot stripped ->
  match; double dot -> no match) and non-HTTP(S) scheme rejection
  (ftp://, file://) prior to any DNS/connect. Parameterize the
  request-header-denylist and response-header-redaction tests over
  the actual module-level '_REQUEST_HEADER_DENYLIST' /
  '_RESPONSE_HEADER_REDACT' constants (canonical + mixed-case
  variants) so coverage can't silently drift from the implementation.
- tests/test_registration_capabilities.py: add a direct
  build_capabilities() regression for the web_request_tools channel —
  default WebRequestConfig() builds exactly one tool; None config and
  tools_disabled=True both suppress the tool AND skip the lazy
  'import_module' factory call entirely (asserted via a monkeypatched
  import_module that raises if invoked).

Gate (via uv run, Python 3.13.7): ruff clean, mypy --strict clean
(37 files), pytest 587 passed (+35 vs. 552 baseline).

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 53035269-5774-46e6-a0bc-a7eba5601967

* docs(frd): finalize FRD 0005 web_request (sign-off + PR #96)

Human sign-off received: flip status In review -> Finalized, confirm the Agent-decided rows, add PR #96 to pull_requests, and record the sign-off / PR-consolidation decision as append-only row K1. Update section 8 to reflect completed implementation (587 tests) shipped via #96; PR #87 closed as superseded.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 6c80e793-f72f-4629-ba8e-16e5847dbe53

* fix: address PR reviewer feedback (URL leakage, global stmt, docs wording, ceiling values)

* fix: split global statement in _ensure_metrics across two lines for readability

---------

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@larohra