Skip to content

azd foundry app insights VNET support - #10819

Open
Kayla Seager (kseager) wants to merge 16 commits into
Azure:mainfrom
kseager:feat/foundry-vnet-monitoring
Open

Kayla Seager (kseager) wants to merge 16 commits into
Azure:mainfrom
kseager:feat/foundry-vnet-monitoring

Conversation

@kseager

@kseager Kayla Seager (kseager) commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Scope

  1. New project, customer VNet
  2. New project, managed VNet
    • It creates the AMPLS and PE and adds the  ampls-monitor-rule  outbound rule.
    • It grants the account identity the Network Connection Approver role.
    • It also adds the foundry-account-pe outbound rule (account to itself), required for hosted agents.
    • If no isolation mode is set, it defaults to  AllowInternetOutbound .
  3. BYO connection only: parity with the previous PR.
    • This covers connecting an existing App Insights component. It creates no new component for an existing project.
    • Existing-project AMPLS, PE and rule creation stays out of scope and is documented

Problem

A project with a network: block (VNet-injected agents) could not send traces: native azd monitoring created public-ingestion App Insights, and #10743 skips monitoring in that case. This PR adds a private path so agent traces arrive in App Insights and can be read from inside the network.

What azd now provisions (new project, network: with your own agent subnet, monitoring enabled)

Customer VNet
├─ agent subnet (delegated Microsoft.App/environments)   ← agent containers
└─ pe subnet
     ├─ private endpoint → Foundry account
     ├─ private endpoint → Storage (blob)
     └─ private endpoint "azuremonitor" ──┐
                                          ▼
                    Azure Monitor Private Link Scope (AMPLS)
                      ingestionAccessMode: PrivateOnly
                      queryAccessMode:     Open
                      scoped resources:
                        ├─ Application Insights   public ingestion Disabled, public query Enabled
                        └─ Log Analytics workspace public ingestion Disabled, public query Enabled

Private DNS zones linked to the VNet (Monitor):
  privatelink.monitor.azure.com, privatelink.oms.opinsights.azure.com,
  privatelink.ods.opinsights.azure.com, privatelink.agentsvc.azure-automation.net

Project connection: AppInsights, ApiKey (project-scoped, azd-appinsights-<hash>)
Project capability host (BYO egress)

Ingestion is private only. Traces are sent through the AMPLS private endpoint and the component rejects public ingestion. Query matches Foundry sample 15 (AMPLS query Open). Difference with S2 bug bash sample for VNET is that our set-up connects the App Insight/workspace to the project vs Foundry.

Component re-sync (reliability fix)

In earlier fresh environments, ingestion intermittently returned 403 Component private link id does not match incoming link for 45 minutes or more (one environment stayed stuck for hours) after ARM reported everything Succeeded. Touching the component's tags cleared it immediately.

Suspected cause: a stale routing cache. Private Link data-plane traffic is handled by internal distributed proxies (the Private Link mapping layer), which ARM does not expose. When the component is created, it registers routing metadata with no AMPLS link. Linking it through the AMPLS scoped resource updates the AMPLS side (the scopedServices table), but the proxy mapping for the component appears to be treated as a cache that is only refreshed by a modification event on the component itself. Until one arrives, the mapping can stay stale and ingestion keeps being rejected as a link-id mismatch. ARM only sees control-plane state, so it reports Succeeded while the data plane is still stale.

A second PUT on the component, even one that only changes a tag, makes ARM send a metadata sync to the underlying resource provider. That should invalidate the stale mapping and reload the link ID.

This mechanism is inferred from the symptoms. It is not confirmed by Azure Monitor or Private Link documentation, and I have not verified it with the service team. The only evidence is the live runs below.

To fix this issue reliablymonitoring-resync.bicep re-declares the component with identical properties after the AMPLS scoped resources exist, adding a deterministic tag azd-ampls-link (a hash of the scope ID and component name; no utcNow, so re-provision is a no-op). The project connection depends on it.

Test evidence (westus3, jump VM inside the VNet, public ingestion off)

Flow per environment: azd provision from a laptop; azd deploy and azd ai agent invoke from the jump VM; traces read privately from the VM.

Env Build Ingestion after provision Deploy + invoke Private trace read
vtrk07 no re-sync 403 for about 45 min, needed a second provision pass pass
vtrk08 with re-sync accepted about 74 s after provision pass pass (29 rows)
vtrk09 with re-sync accepted about 74 s after provision not run not run
vtrk10 with re-sync accepted about 74 s after provision pass pass (34 rows)

Notes and limits:

  • vtrk09 has ingestion evidence only. My local azd client was killed mid-run, so deploy and invoke were not run.
  • Hosted-agent invokes hit the model rate limit (the fixture's gpt-4.1-mini deployment has capacity 1). Retries spaced 75 s apart produced traces. This is a fixture limit, not this PR.
  • Three fresh environments with the re-sync saw no 403, against roughly 2 of 5 stuck earlier. There is no controlled run without the re-sync, so causation is not proven.
  • The unit tests check template shape only. Live behavior was verified by hand.

Behavior and limits

  • Supported on a customer VNet (own agent subnet, BYO egress) or a managed VNet, with an azd-created component or an existing component (azd creates its own Private Link Scope, private endpoint and DNS zones and links the existing component and its workspace). Shared private DNS zones (network.dns), a VNet that already has an azuremonitor private endpoint, or a network azd cannot inspect do not block provisioning: monitoring is skipped with a warning (opt-out: monitoring.enabled: false).
  • Managed egress adds two outbound private-endpoint rules and grants the account identity the Network Connection Approver role:
    • ampls-monitor-rule (target azuremonitor) lets the agent runtime reach the Private Link Scope.
    • foundry-account-pe (target account, to the account itself) is required for hosted agents. Without it they fail with Public access is disabled. Prompt agents are not affected.
  • Hosted agents export telemetry with their own agent identity, which exists only after deploy. That identity needs Monitoring Metrics Publisher on the Application Insights component. azd does not grant it yet. In the live run it was granted by hand, so hosted-agent traces can return 403 until it is granted.
  • A VNet can connect to only one AMPLS; a second one overrides Monitor DNS.
  • With AMPLS ingestion PrivateOnly, telemetry from the VNet to Azure Monitor resources that are not in the scope is blocked.
  • Users reading traces need Log Analytics Reader (plus Privileged Monitoring Data Reader if tables are protected). Because AMPLS query is Open, queries are not restricted to the private network. Verified live: private ingestion plus a read from inside the VNet, not a private-only read.

Changes

  • synthesis/templates/modules/monitor-private-link.bicep: AMPLS, scoped resources, azuremonitor private endpoint, four DNS zones (create-and-link or reuse existing).
  • synthesis/templates/modules/monitoring-resync.bicep (new), wired into monitoring.bicep; added to both eject module lists.
  • Project capability host in resources.bicep.
  • Go: private-link conflict preflight and monitoring policy for private networks.
  • Regenerated main.arm.json and existing-project.arm.json.
  • Tests: TestMonitorPrivateLinkAccessModes, TestMonitoringResyncMatchesComponent, TestCheckMonitorPrivateLinkConflict, TestMonitoringKeyPolicyForPrivateNetwork, VNet requirement tests, preflight tests.
  • README.md and azure.ai.agents/docs/private-networking.md.

Not covered

One region. A fresh managed environment on the final template (including foundry-account-pe) was not provisioned end to end; the managed rules and the agent-identity role were validated by hand on existing environments. Terraform parity and eject for this module follow the same shape but were not live-tested. The project-scoped connection was exercised live only in the runs above.

@azure-pipelines

azure-pipelines Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Azure Pipelines:
5 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

@microsoft-github-policy-service microsoft-github-policy-service Bot added the customer-reported identify a customer issue label Nov 3, 2026
@microsoft-github-policy-service

Copy link
Copy Markdown
Contributor

Thank you for your contribution Kayla Seager (@kseager)! We will review the pull request and get back to you soon.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Critical gaps in private-monitoring conflict checks can disrupt shared DNS and existing telemetry, while workspace ingestion remains publicly accessible despite the private-only configuration.

9 open issues
What changed in this PR

Extends the Foundry infrastructure synthesis shared by the Projects and Agents extensions to support private Application Insights ingestion for new projects using customer-provided agent subnets.

Changes:

  • Adds Azure Monitor private-link infrastructure, DNS integration, a project capability host, and an ordered component re-sync.
  • Enables BYO-network monitoring with endpoint-conflict checks and private-ingestion policy validation.
  • Updates embedded ARM templates, Bicep exports, tests, and networking documentation.

Extension user-contract review: blocked (23 files inspected; 0 new contract comments; 0 matched threads; review history unavailable).

File Description
cli/​azd/​extensions/​azure.ai.projects/​synthesis/​vnet_requirements_test.go Checks capability-host template requirements.
cli/​azd/​extensions/​azure.ai.projects/​synthesis/​templates/​modules/​resources.bicep Adds the project capability host and network outputs.
cli/​azd/​extensions/​azure.ai.projects/​synthesis/​templates/​modules/​monitoring.bicep Wires private monitoring and component re-sync.
cli/​azd/​extensions/​azure.ai.projects/​synthesis/​templates/​modules/​monitoring-resync.bicep Reapplies the component after private-link setup.
cli/​azd/​extensions/​azure.ai.projects/​synthesis/​templates/​modules/​monitor-private-link.bicep Defines the Monitor scope, endpoint, and DNS resources.
cli/​azd/​extensions/​azure.ai.projects/​synthesis/​templates/​main.bicep Passes network settings into monitoring.
cli/​azd/​extensions/​azure.ai.projects/​synthesis/​templates/​main.arm.json Regenerates embedded new-project infrastructure.
cli/​azd/​extensions/​azure.ai.projects/​synthesis/​templates/​existing-project.arm.json Regenerates the embedded monitoring module.
cli/​azd/​extensions/​azure.ai.projects/​synthesis/​synthesizer.go Enables monitoring creation for BYO networks.
cli/​azd/​extensions/​azure.ai.projects/​synthesis/​monitoring_test.go Updates network-specific monitoring expectations.
cli/​azd/​extensions/​azure.ai.projects/​synthesis/​monitoring_resolver.go Applies private-network component validation.
cli/​azd/​extensions/​azure.ai.projects/​synthesis/​monitoring_private_link.go Adds Monitor endpoint-conflict discovery.
cli/​azd/​extensions/​azure.ai.projects/​synthesis/​monitoring_private_link_test.go Tests conflicts, template settings, and ingestion policy.
cli/​azd/​extensions/​azure.ai.projects/​synthesis/​monitoring_policy.go Allows disabled public ingestion on private paths.
cli/​azd/​extensions/​azure.ai.projects/​README.md Documents private monitoring behavior and limits.
cli/​azd/​extensions/​azure.ai.projects/​internal/​provisioning/​monitoring_preflight.go Integrates private-network monitoring checks and skips.
cli/​azd/​extensions/​azure.ai.projects/​internal/​provisioning/​monitoring_preflight_test.go Retargets the unsupported-network skip test to reuse.
cli/​azd/​extensions/​azure.ai.projects/​internal/​provisioning/​monitoring_graph.go Applies private-network policy during graph validation.
cli/​azd/​extensions/​azure.ai.projects/​internal/​cmd/​project_existing_infra.go Exports the new monitoring Bicep dependencies.
cli/​azd/​extensions/​azure.ai.projects/​cspell.yaml Adds Monitor terminology to spell checking.
cli/​azd/​extensions/​azure.ai.agents/​internal/​cmd/​init_infra.go Includes both new modules in Bicep exports.
cli/​azd/​extensions/​azure.ai.agents/​docs/​private-networking.md Describes private monitoring support and exclusions.
cli/​azd/​extensions/​azure.ai.agents/​cspell.yaml Adds the Azure Monitor group identifier.

Review effort: Max

Copilot AI max review requested due to automatic review settings November 6, 2026 21:06
Kayla Porter and others added 5 commits November 6, 2026 13:17
Adds AMPLS, azuremonitor private endpoint, private DNS zones, project
capability host, and a deterministic App Insights component re-sync after
the AMPLS scoped resources so ingestion is reliable on fresh environments.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c1d0003f-40c8-481d-92a5-ecb8d7afb491
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c1d0003f-40c8-481d-92a5-ecb8d7afb491
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c1d0003f-40c8-481d-92a5-ecb8d7afb491
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c1d0003f-40c8-481d-92a5-ecb8d7afb491
… ingestion, docs

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c1d0003f-40c8-481d-92a5-ecb8d7afb491
@kseager
Kayla Seager (kseager) force-pushed the feat/foundry-vnet-monitoring branch from 9257128 to ee24b66 Compare November 6, 2026 21:17

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed to evaluate interacting template-compatibility, network-preflight, and shared-DNS risks that could prevent trace ingestion or disrupt existing environments.

No open issues 🎉

1 resolved since last review

Review effort: Max

Add an ampls-monitor-rule outbound rule to the managed network, grant the account identity the Network Connection Approver role, and stop skipping monitoring for managed egress.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c1d0003f-40c8-481d-92a5-ecb8d7afb491
Copilot AI max review requested due to automatic review settings November 7, 2026 00:25
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c1d0003f-40c8-481d-92a5-ecb8d7afb491

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The role-assignment GUID omits the principal ID, so recreating the account with the same project ID reuses an assignment name for a different identity and fails with RoleAssignmentUpdateNotPermitted.

1 open issue

Review effort: Max

Copilot AI max review requested due to automatic review settings November 7, 2026 00:42

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Default-on managed-egress monitoring risks silent trace loss on an unverified path, while role-assignment names that omit the principal ID can block provisioning after account recreation.

1 open issue

Review effort: Max

…name

Deploy the assignment through a module so the GUID is derived from
scope, principal and role, avoiding RoleAssignmentUpdateNotPermitted
when the account is recreated with the same project ID.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c1d0003f-40c8-481d-92a5-ecb8d7afb491
Copilot AI max review requested due to automatic review settings November 7, 2026 01:40

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Existing-project exports omit a required Bicep module, causing compilation failures, and unresolved vnetId values can silently drop an existing monitoring ownership binding.

2 open issues
1 resolved since last review

Review effort: Max

monitoring.bicep references the module, so both eject paths must copy it or compilation fails.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: c1d0003f-40c8-481d-92a5-ecb8d7afb491
Copilot AI max review requested due to automatic review settings November 7, 2026 03:20
Hosted agents fail with 'Public access is disabled' under a managed network unless the account has an outbound private endpoint rule to itself.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: c1d0003f-40c8-481d-92a5-ecb8d7afb491

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed because unpublished findings identify critical VNet and DNS validation gaps that could bypass network safeguards or redirect another VNet’s telemetry to the wrong endpoint.

No open issues 🎉

2 resolved since last review

Review effort: Max

Copilot AI max review requested due to automatic review settings November 7, 2026 03:39

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The networking documentation omits foundry-account-pe and leaves live-validation status unclear, while the PR description contradicts the implementation by saying managed egress is skipped.

1 open issue

Review effort: Max

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: c1d0003f-40c8-481d-92a5-ecb8d7afb491
Copilot AI max review requested due to automatic review settings November 7, 2026 16:08
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: c1d0003f-40c8-481d-92a5-ecb8d7afb491

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed because private monitoring can be approved without a verified network path, with additional connectivity and ownership risks across configuration changes.

No open issues 🎉

1 resolved since last review

Review effort: Max

Copilot AI max review requested due to automatic review settings November 7, 2026 16:27

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Unpublished findings include a critical risk of re-enabling public queries and interacting monitoring lifecycle and network-ownership concerns that require human assessment.

No open issues 🎉

Review effort: Max

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

customer-reported identify a customer issue ext-agents azure.ai.agents extension ext-projects azure.ai.projects extension

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants