Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 53 additions & 0 deletions cli/azd/.vscode/cspell.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -632,6 +632,59 @@ overrides:
- filename: extensions/azure.ai.content-understanding/internal/testutil/tempdir.go
words:
- testutil
- filename: "extensions/azure.ai.projects/internal/onboarding/*.go"
words:
- armcompute
- armnetwork
- armstorage
- deploymenttemplate
- GZRS
- hasgroups
- Kata
- katagen
- kubelet
- kubeletidentity
- OSSKU
- parseable
- RAGRS
- RAGZRS
- filename: "extensions/azure.ai.projects/{README.md,internal/provisioning/aks_hosting.go}"
words:
- Kata
- katagen
- kubelet
- filename: "extensions/azure.ai.projects/internal/provisioning/{aks_hosting,byo_plan_bindings,byo_plan_expression,byo_template_plan,onboarding_naming}.go"
words:
- ARMID
- copyindex
- createarray
- createobject
- extensionresourceid
- federatedidentityroledefinitionid
- hostingprincipalid
- kubelet
- kubeletobjectid
- managedclusters
- managedidentity
- privateendpoints
- privatednszonegroups
- privatednszones
- resourceid
- resourceinfo
- shallowmerge
- storageaccounts
- subscriptionresourceid
- tolower
- toupper
- tryget
- uniquestring
- userassignedidentities
- virtualnetworklinks
- vnets
- workloadprincipalid
- filename: extensions/azure.ai.projects/internal/testutil/bicep.go
words:
- testutil
- filename: pkg/update/msi_windows.go
words:
- norestart
Expand Down
2 changes: 1 addition & 1 deletion cli/azd/extensions/azure.ai.agents/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -1978,7 +1978,7 @@ deployment.

Foundry project services can be provisioned as network-secured, VNet-bound accounts by adding a `network:` block to the `host: azure.ai.project` service in `azure.yaml`. The `azure.ai.projects` extension owns that service and the `microsoft.foundry` provider; this extension still authors the block during agent init. See [Private networking for `host: azure.ai.project`](docs/private-networking.md) for the schema reference, BYO-image requirements, and VNet deployment cheatsheet.

For an existing AKS cluster, set `agentHosting.clusterResourceId` on that project service. [Existing AKS hosting](../azure.ai.projects/README.md#existing-aks-hosting) describes default or independently supplied Storage and hosting/workload identities, required grants, and lifecycle limitations. A native registry is created only when an Agent requires building a container image, following the standard Foundry rules. Foundry networking is optional through `network:` and can be combined with an automatically created registry when using AKS hosting. Terraform eject is not supported for this mode.
For an existing AKS cluster, set `agentHosting.clusterResourceId` on that project service. [Existing AKS hosting](../azure.ai.projects/README.md#existing-aks-hosting) describes default or independently supplied Storage and hosting/workload identities, read-only preflight, required grants, and lifecycle limitations. A native registry is created only when an Agent requires building a container image, following the standard Foundry rules. Foundry networking is optional through `network:` and can be combined with an automatically created registry when using AKS hosting. Terraform eject is not supported for this mode.

## Skill optimization from traces

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,9 @@ import "embed"

// Re-generate main.arm.json from main.bicep. Run via `go generate ./...`
// from the extension root after editing any *.bicep file under templates/.
// The compiled ARM JSON is embedded so the provisioning provider never
// needs a bicep CLI at user runtime.
// Deployment uses embedded ARM JSON. Preflight may use azd-managed Bicep for
// constant name expressions in the actual compiled plan, including plans
// that reference supplied resources.
//
//go:generate bicep build templates/main.bicep --outfile templates/main.arm.json
//go:generate bicep build templates/existing-project.bicep --outfile templates/existing-project.arm.json
Expand Down
44 changes: 35 additions & 9 deletions cli/azd/extensions/azure.ai.projects/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -191,7 +191,7 @@ services:

`agentHosting` does not require a `network` block or additional subnets. Omit `network` for public networking, or supply it to use the standard account network injection and private-endpoint resources. Existing AKS hosting supports an automatically created ACR with private account networking. Without `agentHosting`, private networking supports prebuilt images, code/ZIP deployments, and source builds using an external registry; it does not support an automatically created ACR.

Use an existing AKS cluster configured for Foundry agent hosting. When effective hosting is enabled, the provider always reads and validates the selected cluster's kubelet principal object ID, regardless of `includeAcr` or an explicit kubelet binding. It resolves the built-in federation role when that deployment binding is needed. It does not validate AKS or network readiness or reconfigure the cluster.
Use an existing AKS cluster configured for Foundry agent hosting. Before deployment, the provider reads the cluster, its AgentPool children and regional VM-size capabilities to check the Foundry hosting prerequisites. When effective hosting is enabled, it always reads and validates the selected cluster's actual kubelet principal object ID, regardless of `includeAcr` or an explicit kubelet binding. It resolves the built-in federation role when that deployment binding is needed. These checks do not create or reconfigure AKS, inspect Kubernetes node Ready conditions, or test runtime connectivity.

The default resource graph creates:

Expand All @@ -202,8 +202,6 @@ The default resource graph creates:

The account receives a `ManagedCluster` / `default` hosting configuration. Its data-plane access follows the optional `network` configuration. **Storage and any automatically created ACR use public authenticated endpoints**, so the cluster must be able to reach them. Registry creation uses API `2023-07-01` and relies on the default registry RBAC mode. The deploying principal needs read access to AKS, the built-in role definitions and target resource group, creation rights for the new resources, and role-assignment write rights at the cluster, workload identity and Storage scopes. Creating an ACR additionally requires registry role-assignment write rights. Configuring private networking additionally requires the corresponding network permissions.

When the hosting-management identity is omitted, the deploying principal also needs `Microsoft.Resources/deployments/write` at the existing AKS resource-group scope. The cluster role grants are created through nested ARM deployments in that group, including when it differs from the Foundry target resource group.

Storage and the two identities can each be supplied independently using complete ARM resource IDs:

```yaml
Expand All @@ -218,7 +216,7 @@ Omit any supporting-resource field to retain its default creation/update path. E

Agents using pre-built images retain their standard registry-connection and authentication requirements; azd does not automatically grant the AKS cluster access to an external registry.

A supplied hosting-management identity needs Reader and Kubernetes Extension Contributor on the cluster. A supplied hosting-management identity and a supplied workload identity require an existing federation-management grant between them. A supplied workload identity and supplied Storage require an existing Storage Blob Data Contributor grant. ARM creates the federation or Storage grant when either corresponding resource selection is omitted; two explicit endpoints require existing grants. A supplied hosting-management identity skips the cluster-grant deployments. Cross-subscription grants use the selected workload or Storage resource's subscription and resource group.
When the hosting-management identity is omitted, provision and preview check the deployer's `Microsoft.Resources/deployments/write` permission at the AKS resource-group scope, where the cluster role-assignment modules are deployed. Cluster-scope role-assignment permissions alone are insufficient, including when AKS and Foundry use different resource groups. This check also applies when all supporting resources use defaults; a supplied hosting-management identity skips these cluster-grant deployments.

The user or automation identity running the deployment also needs the following permissions:

Expand All @@ -230,12 +228,14 @@ The user or automation identity running the deployment also needs the following

These permissions are additional to the applicable resource and role-assignment permissions above. Resource-scoped role-assignment write permission alone does not authorize the resource-group-scoped nested deployments, including in another subscription.

For missing principal bindings declared by the template, the provider reads the effectively selected supplied identities to resolve their principal IDs. A missing federation-role binding is resolved in the effective workload identity's subscription, or the deployment subscription when that identity is omitted. Explicit bindings do not require these lookups. This is deployment binding resolution, not a readiness, networking, or permission preflight. Supplied resources and their existing grants must already be configured for Foundry agent hosting.
For missing principal bindings declared by the template, the provider uses matching supplied-identity metadata collected by preflight. The kubelet binding uses preflight's AKS metadata. Required bindings fail if matching preflight evidence is unavailable or the principal ID is not a GUID. A missing federation-role binding is resolved in the actual workload identity's subscription when the compiled plan identifies that scope, including when its name is unresolved. Otherwise, it uses the effective supplied workload identity's subscription, or the deployment subscription when that identity is omitted. Explicit bindings take precedence; applicable preflight evidence reads still run. Supplied resources and their existing grants must already be configured for Foundry agent hosting.

After local validation, preview opt-in, and credential initialization, independent AKS, federation-role,
and supplied-identity reads run concurrently. The first failed read cancels outstanding
reads and returns its original error; errors are selected by completion order.
All required reads must succeed before bindings are merged and ARM deployment or what-if can proceed.
After local validation, preview opt-in, and credential initialization, preflight collects resource
and authorization evidence. Required bindings must resolve successfully before they are merged
and ARM deployment or what-if can proceed.
Preflight and federation-role binding share built-in role-name lookups within that preparation,
keyed by target subscription and role name. Binding resolves the role on demand if preflight did
not need it; role-definition permission reads and role-assignment checks remain separate.

`azd provision --preview` resolves the host bindings and returns the standard ARM what-if changes without creating resources. `azd provision` uses the same ARM deployment path as a project without AKS hosting. The Bicep template derives names for omitted supporting resources and creates them with the required grants. Successful provisioning returns the standard Project ID, endpoint and registry outputs used by `azd deploy`. Registry outputs are empty when no registry is selected.

Expand All @@ -247,6 +247,32 @@ BYO AKS uses the standard Foundry resource-group creation/update behavior. The d

The provider fills `federatedIdentityRoleDefinitionId` and supplied identities' `hostingPrincipalId` / `workloadPrincipalId` only when the template declares the parameter and the user has not provided it. The same rule applies to `kubeletObjectId` when `includeAcr=true`. Explicit parameter values and envelopes take precedence, with parameter names matched ignoring case, and are forwarded unchanged. User parameters absent from the template are also forwarded so ARM reports the error. Location, resource-name salt and resource-group parameters follow the normal on-disk parameter precedence and ARM validation.

Provision and preview run read-only preflight for every BYO AKS plan, including plans that supply only AKS and create all supporting resources. Checks cover resource existence and identity metadata, subscription owning tenants, supplied-identity grants, AKS hosting prerequisites, the selected existing Agent subnet, and deploying-principal permissions at the actual operation scopes. Ordinary Foundry projects without `agentHosting` do not enter this pipeline.

| Checks | Required evidence |
|---|---|
| A01-A04 | AKS Base rather than Automatic; actual running Kubernetes version at least 1.31; Azure CNI Overlay or a qualifying Agent pool's Pod Subnet; supported managed cluster identity and kubelet object ID; enabled OIDC with issuer and Workload Identity. |
| A05 | Required System pools' applicable VM sizes provide at least 4 available vCPU and 16 GiB. |
| A06-A10 | One complete User Agent pool has Azure Linux, KataVmIsolation, at least one node, at least 8 available vCPU and 32 GiB, an `AKSAzureLinux-V3katagen2-` image at least `202608.14.0`, `agents.ai.azure.com/sandbox-node=true`, disabled SSH, and successful cluster/required pool ARM provisioning states. Requirements cannot be combined across different pools. |
| A11 | Azure or Calico network policy, or the supported Cilium dataplane. |
| N02 | The selected existing `network.agentSubnet` is private IPv4, at least /27, contained in its VNet, non-overlapping with the selected AKS Node subnet, and delegated to `Microsoft.App/environments`. Its provisioning state is not required. New or unselected Agent subnets are not applicable; unrelated Node subnets are not compared. |
| P02, P04, P06 | Identity lifecycle, Storage writes, and unconditional grant authority follow actual creation/reference decisions and current existence. Existing targets use their resource scopes; future targets use their existing parent scope. |
| P03 | Actual selected VNet reads, subnet writes/joins, private endpoints, private-DNS groups, zones and links use the selected resource or deployment-parent scopes. Existing DNS zones include an explicitly selected external DNS subscription's owning tenant check. |
| P05, P07 | Native registry writes and native pull-grant authority apply only to the registry and grants selected by the actual plan. A plan without a native registry does not query a future registry or force ACR creation. |
| P09 | Actual cluster-grant modules require deployment-write at the AKS resource group even when it is also the Foundry resource group. Cross-resource-group/subscription grant deployments retain their scoped deployment read/write and operation-read checks. |

Determinate mandatory failures, including nonexistent references, incompatible configuration and missing required permissions, block deployment and ARM what-if. Unreadable or incomplete Azure evidence retains the corresponding mandatory check result. An unresolved local template-plan dependency instead produces a visible `WARNING` naming the resource ID, or its type and template/module position, the check ID and the reason; only that dependent check is recorded as `SKIPPED`, not `PASS`. Later independent checks still run, and other mandatory failures still block deployment.

Supplied Storage must have a successful provisioning state, StorageV2 kind, Hot tier, HNS explicitly disabled, HTTPS-only transport, TLS 1.2 or later, Shared Key disabled, and anonymous Blob access disabled. Anonymous Blob access is verified as disabled only when Azure returns `allowBlobPublicAccess: false`; `true` fails the security check, while a missing or `null` value is unverified and blocks provisioning. Other missing security properties use the Storage API's documented defaults. Preflight does not change supplied Storage settings. Zone redundancy is recommended but does not block provisioning. The network check requires a valid HTTPS Blob endpoint and explicit network settings: public access with an allow-default firewall passes the static baseline, not a connectivity test. Restricted/private network connectivity is skipped with a warning and does not block provisioning.

A supplied hosting-management identity needs Reader and Kubernetes Extension Contributor on the cluster, plus federation-management access when the workload identity already exists. A supplied workload identity needs Storage Blob Data Contributor when Storage already exists. Applicable inherited permissions, including management-group assignments returned by the target scope's Azure query, or provably equivalent unconditional permissions are accepted; missing or unsupported evidence blocks provisioning. If the dependency does not yet exist, preflight checks the deployer's creation and unconditional role-assignment permissions and ARM creates the dependency and grant. ARM creates the federation or Storage grant when either corresponding resource selection is omitted; two explicit endpoints require existing grants.

Preflight uses current Azure state, not a persisted deployment journal. An existing target missing a required supplied-identity grant can block a retry, including after an interrupted deployment. Omitted Storage retains the deterministic `st<resourceToken>` name and ordinary ARM creation/update path even if it already exists; current existence does not convert it into a supplied reference or establish ownership.

For both default and supplied supporting resources, preflight identifies resources from the actual compiled ARM template and merged parameters, including local nested modules, resolvable conditions, names, resource scopes, selected network resources and native ACR connections. It does not compare the template against a standard-template digest or substitute standard names for rewritten resources.

The local plan adapter is bounded, not a general ARM interpreter. Inner-scoped nested templates use their own `deployment().name`. Ordinary `reference()` returns resource properties; `reference(..., 'Full')` returns the full resource shape, preserving symbolic identity principal IDs for grant checks. Runtime-only reference properties, unavailable linked templates and unsupported expressions remain unresolved; dependent checks warn and skip individually. Supported constant `uniqueString` and `guid` expressions use the existing azd-managed Bicep tool. Explicit deployment bindings retain their normal precedence.

### Infrastructure layers

For projects that use `infra.layers`, declare exactly one layer with
Expand Down
3 changes: 3 additions & 0 deletions cli/azd/extensions/azure.ai.projects/go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,12 @@ require (
github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.15.0
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/authorization/armauthorization/v2 v2.2.0
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/cognitiveservices/armcognitiveservices/v2 v2.0.0
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/compute/armcompute/v7 v7.3.0
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/containerservice/armcontainerservice/v9 v9.0.0
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/msi/armmsi v1.3.0
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/network/armnetwork/v7 v7.2.0
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/resources/armresources v1.2.0
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/storage/armstorage v1.8.1
github.com/azure/azure-dev/cli/azd v1.37.0
github.com/drone/envsubst v1.0.3
github.com/fatih/color v1.19.0
Expand Down
Loading
Loading