feat: web_request system tool (v1) + FRD 0005

[larohra]

Purpose

Adds a built-in web_request system tool so agents can make outbound HTTP(S) calls directly, instead of hand-writing a custom tool per endpoint or spinning up the Dynamic Sessions code interpreter (an ACA sessions resource + managed identity) just to hit a REST API. Calling an HTTP API is the single most common integration need, and making it safe is genuinely hard (SSRF, IMDS token theft, DNS rebinding). Centralizing it as a governed system tool means every agent inherits the security floor for free.

This PR delivers the v1 scope from FRD 0005 and also brings in the FRD document itself (this PR supersedes the FRD-only PR #87, which can be closed).

What v1 gives you:

• A single tool call web_request(method, url, headers?, query?, body?|json?) that makes ONE outbound request to a public host and returns a structured status + headers + parsed body.
• Default-on (see note below): agents get the tool automatically; it can be disabled app-wide (system_tools.web_request: false) or per-agent (web_request: false), and configured via a system_tools.web_request object.
• An always-on SSRF floor that cannot be turned off: rejects non-HTTP(S)/userinfo/bad-port URLs, enforces an optional exact-host allowlist before DNS, resolves and validates every IP as global-unicast-only (blocks loopback, link-local, RFC1918, ULA, CGNAT 100.64/10, IMDS 169.254.169.254, unspecified, reserved, multicast), unwraps IPv4-mapped IPv6, rejects non-canonical numeric IPs, then pins the connection to the validated IP (custom aiohttp resolver/connector, DNS cache off) so DNS rebinding cannot swap the address after validation. HTTPS is required unless require_https: false.

Approach: mirrors the existing dynamic_sessions_code_interpreter (sandbox) system tool across the config -> merge -> impl -> registration -> telemetry pipeline, with two deliberate differences: (1) resolution is default-on rather than opt-in, and (2) because the tool is stateless it is built once per agent at registration and threaded through a dedicated web_request_tools channel (never merged with sandbox_tools), rather than rebuilt per request.

Note on default-on: this is additive and does not break any existing config or API, but it does mean agents gain outbound-HTTP capability by default. The SSRF floor (public hosts only, private/link-local/IMDS blocked) is always enforced, and operators can disable globally or per-agent. Flagging it explicitly so reviewers can weigh the posture.

Validated and works E2E -

Does this introduce a breaking change?

[ ] Yes
[x] No

Pull Request Type

What kind of change does this Pull Request introduce?

[ ] Bugfix
[x] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no api changes)
[ ] Documentation content changes
[ ] Other... Please describe:

How to Test

• Get the code

git clone https://github.com/Azure/azure-functions-agents-runtime.git
cd azure-functions-agents-runtime
git checkout larohra-web-request-system-tool
pip install -e .[dev]

• Test the code

The repo requires Python 3.13+ (PEP 695 syntax). Run the canonical gate:

python -m ruff check src tests
python -m mypy src
python -m pytest tests -q

Expected: ruff clean, mypy strict clean (37 source files), 587 tests passed. The new SSRF/response/telemetry suite lives in tests/test_web_request.py; config resolution in tests/test_config_merge.py + fixture tests/fixtures/config_scenarios/14_web_request/; default-on registration in tests/test_registration_capabilities.py.

What to Check

Verify that the following are valid

• SSRF floor is fail-closed and validates every resolved IP, pins the connection to it, and preserves the Host header / TLS SNI. Allowlist is enforced before DNS.
• Default-on resolution (config/merge.py _resolve_web_request): global absent/None/true -> defaults; global false -> disabled app-wide; global object -> that config; per-agent false -> opt-out regardless of global.
• v1 scope discipline: no auth / per-host credentials, no {env: VAR} secret refs, no redirect following (allow_redirects=False, redirect_count always 0), no per-agent override object (per-agent is bool-only), no wildcard/suffix host matching, no IDNA/punycode. These are documented in the FRD as v2/v3+ and must be absent from product code.
• Telemetry redaction: the web_request span and counter record success/error without leaking full URL, query string, or secret header values.
• Caps clamping: timeout and request/response byte sizes are clamped to absolute operational maximums.

Other Information

• Design: docs/frds/0005-web-request-system-tool.md (Section 4 documents the full target design; only untagged items are v1, v2/v3+ items are tagged). Deferred to later versions: authenticated requests + per-host secrets, redirect following with per-hop re-validation, per-agent override objects, wildcard host matching, IDNA.
• Refs Azure/azure-functions-bucees-planning#1176 (this delivers v1 of that phased ask; later versions remain open, so this does not auto-close it).
• Docs updated: front-matter-reference.md (regenerated), front-matter-spec.md, architecture.md, observability.md, README.md.
• Validation: implemented by a claude-sonnet-5 session, then independently reviewed by a gpt-5.5 rubber-duck pass (no blocking correctness or SSRF defect, no v2 scope leakage) plus a re-run of the full gate. The last commit hardens regression coverage on the security-critical paths (trailing-dot allowlist normalization, non-HTTP(S) scheme rejection, full header denylist/redaction sets with mixed-case, and default-on capability build/suppress).

[Copilot]

Pull request overview

Introduces a new built-in web_request system tool (default-on) that lets agents make outbound HTTP(S) requests to public hosts with an always-on SSRF safety floor, and wires it through config resolution, capability building, runner/tool assembly, and observability. The PR also adds FRD 0005 and updates user/docs surfaces to reflect the new tool and configuration model.

Changes:

• Add web_request tool implementation with SSRF validation + IP pinning + response shaping + telemetry/metrics.
• Add config schema + merge logic for default-on enablement with global disable and per-agent opt-out.
• Add comprehensive tests, fixture scenarios, and docs/FRD updates.

Reviewed changes

Copilot reviewed 26 out of 26 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
tests/test_web_request.py	New unit suite covering params schema, response shaping, SSRF validator, header policy, and telemetry behavior.
tests/test_registration_capabilities.py	Verifies default-on tool creation at registration and suppression when disabled.
tests/test_package_imports.py	Adds create_web_request_tools to public export surface checks.
tests/test_config_merge.py	Tests _resolve_web_request default-on merge/override semantics.
tests/test_config_fixtures.py	Adds config scenario fixture assertions for global config + per-agent opt-out.
tests/fixtures/config_scenarios/14_web_request/opted_out.agent.md	New per-agent opt-out fixture.
tests/fixtures/config_scenarios/14_web_request/main.agent.md	New default agent fixture inheriting global web_request config.
tests/fixtures/config_scenarios/14_web_request/agents.config.yaml	New global config fixture for system_tools.web_request.
src/azure_functions_agents/system_tools/web_request.py	New tool implementation (SSRF floor, IP pinning via aiohttp resolver, caps, response shaping, telemetry).
src/azure_functions_agents/runner.py	Adds web_request_tools channel to tool assembly for run + stream paths.
src/azure_functions_agents/registration/endpoints.py	Threads capabilities.web_request_tools into builtin endpoint execution paths.
src/azure_functions_agents/registration/capabilities.py	Builds web_request tools once per agent at registration via lazy import.
src/azure_functions_agents/registration/_handlers.py	Threads web_request_tools through trigger handlers into runner calls.
src/azure_functions_agents/config/schema.py	Adds WebRequestConfig and web_request fields to global/per-agent system tool config + resolved agent.
src/azure_functions_agents/config/merge.py	Implements default-on _resolve_web_request resolution semantics.
src/azure_functions_agents/app.py	Adds web_request to system tools indexing summary.
src/azure_functions_agents/_observability.py	Adds FaultDomain.WEB_REQUEST and new metrics counters for web_request.
src/azure_functions_agents/init.py	Exposes create_web_request_tools in the package public API.
README.md	Documents the new default-on web_request capability and observability surface.
eng/scripts/generate_config_reference.py	Updates config reference generator to include system_tools.web_request.
docs/observability.md	Documents web_request span attributes and metrics.
docs/front-matter-spec.md	Adds authoring/spec documentation for system_tools.web_request.
docs/front-matter-reference.md	Regenerated reference including system_tools.web_request.
docs/frds/README.md	Adds FRD 0005 index entry.
docs/frds/0005-web-request-system-tool.md	Adds FRD 0005 (design + decisions log + scope).
docs/architecture.md	Updates module map/pipeline narrative to include web_request wiring.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[larohra]

@copilot fix this

[Copilot]

@copilot fix this

Done in commits fix: address PR reviewer feedback and fix: split global statement:

1. URL leakage (web_request.py:648-652): aiohttp.ClientError handler now uses type(exc).__name__ only — the full exception message (which can embed the request URL + query string) is no longer surfaced in the telemetry span, log, or the JSON error returned to the model.

2. global backslash continuations (_observability.py:434): replaced with two clean global statements (Python doesn't support parenthesized global).

3. allowed_hosts default (front-matter-reference.md:40): was [], now null — matching the schema (list[str] | None = None) and the actual semantic (omit = any public host allowed). Fixed in generate_config_reference.py via a custom_defaults entry, then regenerated.

4. "operator-defined maximum" wording (front-matter-reference.md:42-44, front-matter-spec.md:440, generate_config_reference.py:252-255): replaced with "runtime-defined ceiling (N)" with the concrete values (timeout: 120 s, response/request: 10 MB). front-matter-reference.md regenerated.