diff --git a/cli/azd/.vscode/cspell.yaml b/cli/azd/.vscode/cspell.yaml index 65de200620d..2eb8b4ce866 100644 --- a/cli/azd/.vscode/cspell.yaml +++ b/cli/azd/.vscode/cspell.yaml @@ -632,6 +632,59 @@ overrides: - filename: extensions/azure.ai.content-understanding/internal/testutil/tempdir.go words: - testutil + - filename: "extensions/azure.ai.projects/internal/onboarding/*.go" + words: + - armcompute + - armnetwork + - armstorage + - deploymenttemplate + - GZRS + - hasgroups + - Kata + - katagen + - kubelet + - kubeletidentity + - OSSKU + - parseable + - RAGRS + - RAGZRS + - filename: "extensions/azure.ai.projects/{README.md,internal/provisioning/aks_hosting.go}" + words: + - Kata + - katagen + - kubelet + - filename: "extensions/azure.ai.projects/internal/provisioning/{aks_hosting,byo_plan_bindings,byo_plan_expression,byo_template_plan,onboarding_naming}.go" + words: + - ARMID + - copyindex + - createarray + - createobject + - extensionresourceid + - federatedidentityroledefinitionid + - hostingprincipalid + - kubelet + - kubeletobjectid + - managedclusters + - managedidentity + - privateendpoints + - privatednszonegroups + - privatednszones + - resourceid + - resourceinfo + - shallowmerge + - storageaccounts + - subscriptionresourceid + - tolower + - toupper + - tryget + - uniquestring + - userassignedidentities + - virtualnetworklinks + - vnets + - workloadprincipalid + - filename: extensions/azure.ai.projects/internal/testutil/bicep.go + words: + - testutil - filename: pkg/update/msi_windows.go words: - norestart diff --git a/cli/azd/extensions/azure.ai.agents/README.md b/cli/azd/extensions/azure.ai.agents/README.md index 1683dce9f4c..f8beb0c198f 100644 --- a/cli/azd/extensions/azure.ai.agents/README.md +++ b/cli/azd/extensions/azure.ai.agents/README.md @@ -1978,7 +1978,7 @@ deployment. Foundry project services can be provisioned as network-secured, VNet-bound accounts by adding a `network:` block to the `host: azure.ai.project` service in `azure.yaml`. The `azure.ai.projects` extension owns that service and the `microsoft.foundry` provider; this extension still authors the block during agent init. See [Private networking for `host: azure.ai.project`](docs/private-networking.md) for the schema reference, BYO-image requirements, and VNet deployment cheatsheet. -For an existing AKS cluster, set `agentHosting.clusterResourceId` on that project service. [Existing AKS hosting](../azure.ai.projects/README.md#existing-aks-hosting) describes default or independently supplied Storage and hosting/workload identities, required grants, and lifecycle limitations. A native registry is created only when an Agent requires building a container image, following the standard Foundry rules. Foundry networking is optional through `network:` and can be combined with an automatically created registry when using AKS hosting. Terraform eject is not supported for this mode. +For an existing AKS cluster, set `agentHosting.clusterResourceId` on that project service. [Existing AKS hosting](../azure.ai.projects/README.md#existing-aks-hosting) describes default or independently supplied Storage and hosting/workload identities, read-only preflight, required grants, and lifecycle limitations. A native registry is created only when an Agent requires building a container image, following the standard Foundry rules. Foundry networking is optional through `network:` and can be combined with an automatically created registry when using AKS hosting. Terraform eject is not supported for this mode. ## Skill optimization from traces diff --git a/cli/azd/extensions/azure.ai.agents/internal/synthesis/templates_embed.go b/cli/azd/extensions/azure.ai.agents/internal/synthesis/templates_embed.go index 66feb2ea6d3..f13b8c9bdb7 100644 --- a/cli/azd/extensions/azure.ai.agents/internal/synthesis/templates_embed.go +++ b/cli/azd/extensions/azure.ai.agents/internal/synthesis/templates_embed.go @@ -7,8 +7,9 @@ import "embed" // Re-generate main.arm.json from main.bicep. Run via `go generate ./...` // from the extension root after editing any *.bicep file under templates/. -// The compiled ARM JSON is embedded so the provisioning provider never -// needs a bicep CLI at user runtime. +// Deployment uses embedded ARM JSON. Preflight may use azd-managed Bicep for +// constant name expressions in the actual compiled plan, including plans +// that reference supplied resources. // //go:generate bicep build templates/main.bicep --outfile templates/main.arm.json //go:generate bicep build templates/existing-project.bicep --outfile templates/existing-project.arm.json diff --git a/cli/azd/extensions/azure.ai.projects/README.md b/cli/azd/extensions/azure.ai.projects/README.md index ef1ade57aa5..a486a329db0 100644 --- a/cli/azd/extensions/azure.ai.projects/README.md +++ b/cli/azd/extensions/azure.ai.projects/README.md @@ -191,7 +191,7 @@ services: `agentHosting` does not require a `network` block or additional subnets. Omit `network` for public networking, or supply it to use the standard account network injection and private-endpoint resources. Existing AKS hosting supports an automatically created ACR with private account networking. Without `agentHosting`, private networking supports prebuilt images, code/ZIP deployments, and source builds using an external registry; it does not support an automatically created ACR. -Use an existing AKS cluster configured for Foundry agent hosting. When effective hosting is enabled, the provider always reads and validates the selected cluster's kubelet principal object ID, regardless of `includeAcr` or an explicit kubelet binding. It resolves the built-in federation role when that deployment binding is needed. It does not validate AKS or network readiness or reconfigure the cluster. +Use an existing AKS cluster configured for Foundry agent hosting. Before deployment, the provider reads the cluster, its AgentPool children and regional VM-size capabilities to check the Foundry hosting prerequisites. When effective hosting is enabled, it always reads and validates the selected cluster's actual kubelet principal object ID, regardless of `includeAcr` or an explicit kubelet binding. It resolves the built-in federation role when that deployment binding is needed. These checks do not create or reconfigure AKS, inspect Kubernetes node Ready conditions, or test runtime connectivity. The default resource graph creates: @@ -202,8 +202,6 @@ The default resource graph creates: The account receives a `ManagedCluster` / `default` hosting configuration. Its data-plane access follows the optional `network` configuration. **Storage and any automatically created ACR use public authenticated endpoints**, so the cluster must be able to reach them. Registry creation uses API `2023-07-01` and relies on the default registry RBAC mode. The deploying principal needs read access to AKS, the built-in role definitions and target resource group, creation rights for the new resources, and role-assignment write rights at the cluster, workload identity and Storage scopes. Creating an ACR additionally requires registry role-assignment write rights. Configuring private networking additionally requires the corresponding network permissions. -When the hosting-management identity is omitted, the deploying principal also needs `Microsoft.Resources/deployments/write` at the existing AKS resource-group scope. The cluster role grants are created through nested ARM deployments in that group, including when it differs from the Foundry target resource group. - Storage and the two identities can each be supplied independently using complete ARM resource IDs: ```yaml @@ -218,7 +216,7 @@ Omit any supporting-resource field to retain its default creation/update path. E Agents using pre-built images retain their standard registry-connection and authentication requirements; azd does not automatically grant the AKS cluster access to an external registry. -A supplied hosting-management identity needs Reader and Kubernetes Extension Contributor on the cluster. A supplied hosting-management identity and a supplied workload identity require an existing federation-management grant between them. A supplied workload identity and supplied Storage require an existing Storage Blob Data Contributor grant. ARM creates the federation or Storage grant when either corresponding resource selection is omitted; two explicit endpoints require existing grants. A supplied hosting-management identity skips the cluster-grant deployments. Cross-subscription grants use the selected workload or Storage resource's subscription and resource group. +When the hosting-management identity is omitted, provision and preview check the deployer's `Microsoft.Resources/deployments/write` permission at the AKS resource-group scope, where the cluster role-assignment modules are deployed. Cluster-scope role-assignment permissions alone are insufficient, including when AKS and Foundry use different resource groups. This check also applies when all supporting resources use defaults; a supplied hosting-management identity skips these cluster-grant deployments. The user or automation identity running the deployment also needs the following permissions: @@ -230,12 +228,14 @@ The user or automation identity running the deployment also needs the following These permissions are additional to the applicable resource and role-assignment permissions above. Resource-scoped role-assignment write permission alone does not authorize the resource-group-scoped nested deployments, including in another subscription. -For missing principal bindings declared by the template, the provider reads the effectively selected supplied identities to resolve their principal IDs. A missing federation-role binding is resolved in the effective workload identity's subscription, or the deployment subscription when that identity is omitted. Explicit bindings do not require these lookups. This is deployment binding resolution, not a readiness, networking, or permission preflight. Supplied resources and their existing grants must already be configured for Foundry agent hosting. +For missing principal bindings declared by the template, the provider uses matching supplied-identity metadata collected by preflight. The kubelet binding uses preflight's AKS metadata. Required bindings fail if matching preflight evidence is unavailable or the principal ID is not a GUID. A missing federation-role binding is resolved in the actual workload identity's subscription when the compiled plan identifies that scope, including when its name is unresolved. Otherwise, it uses the effective supplied workload identity's subscription, or the deployment subscription when that identity is omitted. Explicit bindings take precedence; applicable preflight evidence reads still run. Supplied resources and their existing grants must already be configured for Foundry agent hosting. -After local validation, preview opt-in, and credential initialization, independent AKS, federation-role, -and supplied-identity reads run concurrently. The first failed read cancels outstanding -reads and returns its original error; errors are selected by completion order. -All required reads must succeed before bindings are merged and ARM deployment or what-if can proceed. +After local validation, preview opt-in, and credential initialization, preflight collects resource +and authorization evidence. Required bindings must resolve successfully before they are merged +and ARM deployment or what-if can proceed. +Preflight and federation-role binding share built-in role-name lookups within that preparation, +keyed by target subscription and role name. Binding resolves the role on demand if preflight did +not need it; role-definition permission reads and role-assignment checks remain separate. `azd provision --preview` resolves the host bindings and returns the standard ARM what-if changes without creating resources. `azd provision` uses the same ARM deployment path as a project without AKS hosting. The Bicep template derives names for omitted supporting resources and creates them with the required grants. Successful provisioning returns the standard Project ID, endpoint and registry outputs used by `azd deploy`. Registry outputs are empty when no registry is selected. @@ -247,6 +247,32 @@ BYO AKS uses the standard Foundry resource-group creation/update behavior. The d The provider fills `federatedIdentityRoleDefinitionId` and supplied identities' `hostingPrincipalId` / `workloadPrincipalId` only when the template declares the parameter and the user has not provided it. The same rule applies to `kubeletObjectId` when `includeAcr=true`. Explicit parameter values and envelopes take precedence, with parameter names matched ignoring case, and are forwarded unchanged. User parameters absent from the template are also forwarded so ARM reports the error. Location, resource-name salt and resource-group parameters follow the normal on-disk parameter precedence and ARM validation. +Provision and preview run read-only preflight for every BYO AKS plan, including plans that supply only AKS and create all supporting resources. Checks cover resource existence and identity metadata, subscription owning tenants, supplied-identity grants, AKS hosting prerequisites, the selected existing Agent subnet, and deploying-principal permissions at the actual operation scopes. Ordinary Foundry projects without `agentHosting` do not enter this pipeline. + +| Checks | Required evidence | +|---|---| +| A01-A04 | AKS Base rather than Automatic; actual running Kubernetes version at least 1.31; Azure CNI Overlay or a qualifying Agent pool's Pod Subnet; supported managed cluster identity and kubelet object ID; enabled OIDC with issuer and Workload Identity. | +| A05 | Required System pools' applicable VM sizes provide at least 4 available vCPU and 16 GiB. | +| A06-A10 | One complete User Agent pool has Azure Linux, KataVmIsolation, at least one node, at least 8 available vCPU and 32 GiB, an `AKSAzureLinux-V3katagen2-` image at least `202608.14.0`, `agents.ai.azure.com/sandbox-node=true`, disabled SSH, and successful cluster/required pool ARM provisioning states. Requirements cannot be combined across different pools. | +| A11 | Azure or Calico network policy, or the supported Cilium dataplane. | +| N02 | The selected existing `network.agentSubnet` is private IPv4, at least /27, contained in its VNet, non-overlapping with the selected AKS Node subnet, and delegated to `Microsoft.App/environments`. Its provisioning state is not required. New or unselected Agent subnets are not applicable; unrelated Node subnets are not compared. | +| P02, P04, P06 | Identity lifecycle, Storage writes, and unconditional grant authority follow actual creation/reference decisions and current existence. Existing targets use their resource scopes; future targets use their existing parent scope. | +| P03 | Actual selected VNet reads, subnet writes/joins, private endpoints, private-DNS groups, zones and links use the selected resource or deployment-parent scopes. Existing DNS zones include an explicitly selected external DNS subscription's owning tenant check. | +| P05, P07 | Native registry writes and native pull-grant authority apply only to the registry and grants selected by the actual plan. A plan without a native registry does not query a future registry or force ACR creation. | +| P09 | Actual cluster-grant modules require deployment-write at the AKS resource group even when it is also the Foundry resource group. Cross-resource-group/subscription grant deployments retain their scoped deployment read/write and operation-read checks. | + +Determinate mandatory failures, including nonexistent references, incompatible configuration and missing required permissions, block deployment and ARM what-if. Unreadable or incomplete Azure evidence retains the corresponding mandatory check result. An unresolved local template-plan dependency instead produces a visible `WARNING` naming the resource ID, or its type and template/module position, the check ID and the reason; only that dependent check is recorded as `SKIPPED`, not `PASS`. Later independent checks still run, and other mandatory failures still block deployment. + +Supplied Storage must have a successful provisioning state, StorageV2 kind, Hot tier, HNS explicitly disabled, HTTPS-only transport, TLS 1.2 or later, Shared Key disabled, and anonymous Blob access disabled. Anonymous Blob access is verified as disabled only when Azure returns `allowBlobPublicAccess: false`; `true` fails the security check, while a missing or `null` value is unverified and blocks provisioning. Other missing security properties use the Storage API's documented defaults. Preflight does not change supplied Storage settings. Zone redundancy is recommended but does not block provisioning. The network check requires a valid HTTPS Blob endpoint and explicit network settings: public access with an allow-default firewall passes the static baseline, not a connectivity test. Restricted/private network connectivity is skipped with a warning and does not block provisioning. + +A supplied hosting-management identity needs Reader and Kubernetes Extension Contributor on the cluster, plus federation-management access when the workload identity already exists. A supplied workload identity needs Storage Blob Data Contributor when Storage already exists. Applicable inherited permissions, including management-group assignments returned by the target scope's Azure query, or provably equivalent unconditional permissions are accepted; missing or unsupported evidence blocks provisioning. If the dependency does not yet exist, preflight checks the deployer's creation and unconditional role-assignment permissions and ARM creates the dependency and grant. ARM creates the federation or Storage grant when either corresponding resource selection is omitted; two explicit endpoints require existing grants. + +Preflight uses current Azure state, not a persisted deployment journal. An existing target missing a required supplied-identity grant can block a retry, including after an interrupted deployment. Omitted Storage retains the deterministic `st` name and ordinary ARM creation/update path even if it already exists; current existence does not convert it into a supplied reference or establish ownership. + +For both default and supplied supporting resources, preflight identifies resources from the actual compiled ARM template and merged parameters, including local nested modules, resolvable conditions, names, resource scopes, selected network resources and native ACR connections. It does not compare the template against a standard-template digest or substitute standard names for rewritten resources. + +The local plan adapter is bounded, not a general ARM interpreter. Inner-scoped nested templates use their own `deployment().name`. Ordinary `reference()` returns resource properties; `reference(..., 'Full')` returns the full resource shape, preserving symbolic identity principal IDs for grant checks. Runtime-only reference properties, unavailable linked templates and unsupported expressions remain unresolved; dependent checks warn and skip individually. Supported constant `uniqueString` and `guid` expressions use the existing azd-managed Bicep tool. Explicit deployment bindings retain their normal precedence. + ### Infrastructure layers For projects that use `infra.layers`, declare exactly one layer with diff --git a/cli/azd/extensions/azure.ai.projects/go.mod b/cli/azd/extensions/azure.ai.projects/go.mod index 59d552585fa..7cb0658b49c 100644 --- a/cli/azd/extensions/azure.ai.projects/go.mod +++ b/cli/azd/extensions/azure.ai.projects/go.mod @@ -7,9 +7,12 @@ require ( github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.15.0 github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/authorization/armauthorization/v2 v2.2.0 github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/cognitiveservices/armcognitiveservices/v2 v2.0.0 + github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/compute/armcompute/v7 v7.3.0 github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/containerservice/armcontainerservice/v9 v9.0.0 github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/msi/armmsi v1.3.0 + github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/network/armnetwork/v7 v7.2.0 github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/resources/armresources v1.2.0 + github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/storage/armstorage v1.8.1 github.com/azure/azure-dev/cli/azd v1.37.0 github.com/drone/envsubst v1.0.3 github.com/fatih/color v1.19.0 diff --git a/cli/azd/extensions/azure.ai.projects/go.sum b/cli/azd/extensions/azure.ai.projects/go.sum index 641d0c6fcbe..fdad47bec07 100644 --- a/cli/azd/extensions/azure.ai.projects/go.sum +++ b/cli/azd/extensions/azure.ai.projects/go.sum @@ -17,6 +17,8 @@ github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/authorization/armauthoriza github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/authorization/armauthorization/v2 v2.2.0/go.mod h1:/pz8dyNQe+Ey3yBp/XuYz7oqX8YDNWVpPB0hH3XWfbc= github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/cognitiveservices/armcognitiveservices/v2 v2.0.0 h1:pxphC/uRZKNHNPbZ0duDDgKkefju2F03OkG5xF6byHQ= github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/cognitiveservices/armcognitiveservices/v2 v2.0.0/go.mod h1:twcwRey+l1znKBL5TEzYiZMtiVkWfM7Pq8a9vY04xYc= +github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/compute/armcompute/v7 v7.3.0 h1:nyxugFxG2uhbMeJVCFFuD2j9wu+6KgeabITdINraQsE= +github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/compute/armcompute/v7 v7.3.0/go.mod h1:e4RAYykLIz73CF52KhSooo4whZGXvXrD09m0jkgnWiU= github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/containerservice/armcontainerservice/v9 v9.0.0 h1:z5qKE92WxOYrrU2yWL7fL0XW7vVBjMKyaXOITK+Z3bI= github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/containerservice/armcontainerservice/v9 v9.0.0/go.mod h1:1NzwJtdlA4Qwki8NuLFN2wVS24bOUiWRcNWpmcmHNiU= github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/internal/v2 v2.0.0 h1:PTFGRSlMKCQelWwxUyYVEUqseBJVemLyqWJjvMyt0do= @@ -29,10 +31,14 @@ github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/managementgroups/armmanage github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/managementgroups/armmanagementgroups v1.0.0/go.mod h1:mLfWfj8v3jfWKsL9G4eoBoXVcsqcIUTapmdKy7uGOp0= github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/msi/armmsi v1.3.0 h1:L7G3dExHBgUxsO3qpTGhk/P2dgnYyW48yn7AO33Tbek= github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/msi/armmsi v1.3.0/go.mod h1:Ms6gYEy0+A2knfKrwdatsggTXYA2+ICKug8w7STorFw= +github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/network/armnetwork/v7 v7.2.0 h1:DgqO2jYgDEqmN8W5sPP+ZU7Tfxyn+i9RqXtNsX6Enb8= +github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/network/armnetwork/v7 v7.2.0/go.mod h1:FBChJszHNRdH5AYJ+Y/NgWilJihKa5WcSlFrNnj2eY0= github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/resources/armresources v1.2.0 h1:Dd+RhdJn0OTtVGaeDLZpcumkIVCtA/3/Fo42+eoYvVM= github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/resources/armresources v1.2.0/go.mod h1:5kakwfW5CjC9KK+Q4wjXAg+ShuIm2mBMua0ZFj2C8PE= github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/resources/armsubscriptions v1.3.0 h1:wxQx2Bt4xzPIKvW59WQf1tJNx/ZZKPfN+EhPX3Z6CYY= github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/resources/armsubscriptions v1.3.0/go.mod h1:TpiwjwnW/khS0LKs4vW5UmmT9OWcxaveS8U7+tlknzo= +github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/storage/armstorage v1.8.1 h1:/Zt+cDPnpC3OVDm/JKLOs7M2DKmLRIIp3XIx9pHHiig= +github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/storage/armstorage v1.8.1/go.mod h1:Ng3urmn6dYe8gnbCMoHHVl5APYz2txho3koEkV2o2HA= github.com/Azure/azure-sdk-for-go/sdk/security/keyvault/azsecrets v1.5.0 h1:aMFOzch6ZJo4Ct9hI4A9Y2fPen5YNRTPmkSBhe5m0ZQ= github.com/Azure/azure-sdk-for-go/sdk/security/keyvault/azsecrets v1.5.0/go.mod h1:Oct8bx+g+DXKngU7i/LzFzYt44rmLdMu4uoofIpooVo= github.com/Azure/azure-sdk-for-go/sdk/security/keyvault/internal v1.2.0 h1:nCYfgcSyHZXJI8J0IWE5MsCGlb2xp9fJiXyxWgmOFg4= diff --git a/cli/azd/extensions/azure.ai.projects/internal/onboarding/authorization_evidence.go b/cli/azd/extensions/azure.ai.projects/internal/onboarding/authorization_evidence.go new file mode 100644 index 00000000000..6123893e053 --- /dev/null +++ b/cli/azd/extensions/azure.ai.projects/internal/onboarding/authorization_evidence.go @@ -0,0 +1,76 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +package onboarding + +import ( + "context" + "strings" + "sync" + + "github.com/Azure/azure-sdk-for-go/sdk/azcore" + "github.com/Azure/azure-sdk-for-go/sdk/azcore/arm" + "github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/authorization/armauthorization/v2" +) + +// passReads shares raw evidence, including errors, only within one read-only pass. +// OnceValues also coalesces concurrent misses; callers must not mutate returned evidence. +type passReads[T any] struct { + mu sync.Mutex + values map[string]func() (T, error) +} + +func (c *passReads[T]) get(key string, read func() (T, error)) (T, error) { + c.mu.Lock() + if c.values == nil { + c.values = map[string]func() (T, error){} + } + load, ok := c.values[key] + if !ok { + load = sync.OnceValues(read) + c.values[key] = load + } + c.mu.Unlock() + return load() +} + +// AuthorizationEvidence shares authorization reads within one provisioning preparation. +type AuthorizationEvidence struct { + credential azcore.TokenCredential + options *arm.ClientOptions + assignmentClients passReads[*armauthorization.RoleAssignmentsClient] + definitionClients passReads[*armauthorization.RoleDefinitionsClient] + assignments passReads[[]*armauthorization.RoleAssignment] + definitions passReads[armauthorization.RoleDefinitionsClientGetByIDResponse] + roleNames passReads[string] +} + +// NewAuthorizationEvidence creates a request-local, concurrency-safe evidence cache. +func NewAuthorizationEvidence(credential azcore.TokenCredential, options *arm.ClientOptions) *AuthorizationEvidence { + return &AuthorizationEvidence{credential: credential, options: onboardingARMOptions(options)} +} + +func (a *scopedAuthorizer) evidence() *AuthorizationEvidence { + if a.reads != nil { + return a.reads + } + return NewAuthorizationEvidence(a.credential, a.options) +} + +func (e *AuthorizationEvidence) definitionClient() (*armauthorization.RoleDefinitionsClient, error) { + return e.definitionClients.get("", func() (*armauthorization.RoleDefinitionsClient, error) { + return armauthorization.NewRoleDefinitionsClient(e.credential, e.options) + }) +} + +func (e *AuthorizationEvidence) roleDefinition( + ctx context.Context, id string, +) (armauthorization.RoleDefinitionsClientGetByIDResponse, error) { + return e.definitions.get(strings.ToLower(id), func() (armauthorization.RoleDefinitionsClientGetByIDResponse, error) { + client, err := e.definitionClient() + if err != nil { + return armauthorization.RoleDefinitionsClientGetByIDResponse{}, err + } + return client.GetByID(ctx, id, nil) + }) +} diff --git a/cli/azd/extensions/azure.ai.projects/internal/onboarding/authorization_rules.go b/cli/azd/extensions/azure.ai.projects/internal/onboarding/authorization_rules.go new file mode 100644 index 00000000000..56016ef52c9 --- /dev/null +++ b/cli/azd/extensions/azure.ai.projects/internal/onboarding/authorization_rules.go @@ -0,0 +1,140 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +package onboarding + +import ( + "encoding/base64" + "encoding/json" + "fmt" + "regexp" + "strings" + + "github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/authorization/armauthorization/v2" +) + +const roleAssignmentWrite = "Microsoft.Authorization/roleAssignments/write" + +type grantRequest struct { + checkID, scope, principalID, roleName, roleID string +} + +func grantRequests(p Plan, hosting, workload Identity) []grantRequest { + return []grantRequest{ + {"G01", p.Cluster.ID, hosting.PrincipalID, "Reader", "acdd72a7-3385-48ef-bd42-f606fba81ae7"}, + {"G01", p.Cluster.ID, hosting.PrincipalID, + "Kubernetes Extension Contributor", "85cb6faf-e071-4c9b-8136-154b5a04f717"}, + {"G02", p.Workload.ID, hosting.PrincipalID, + "Managed Identity Federated Identity Credential Contributor", ""}, + {"G03", p.Storage.ID, workload.PrincipalID, + "Storage Blob Data Contributor", "ba92f5b4-2d11-453d-a403-e96b0029c9fe"}, + } +} + +// GrantCheckIDForRole classifies authored assignments using the existing required grant-role definitions. +func GrantCheckIDForRole(roleID string) string { + for _, request := range grantRequests(Plan{}, Identity{}, Identity{}) { + if request.roleID != "" && strings.EqualFold(request.roleID, roleID) { + return request.checkID + } + } + return "" +} + +func createsGrant(plan Plan, id string) bool { + if plan.Grants != nil { + for _, grant := range plan.Grants { + if grant.CheckID == id { + return true + } + } + return false + } + switch id { + case "G01": + return !plan.Hosting.Supplied + case "G02": + return !plan.Hosting.Supplied || !plan.Workload.Supplied + case "G03": + return !plan.Workload.Supplied || !plan.Storage.Supplied + default: + return false + } +} + +func authString(p *string) string { + if p == nil { + return "" + } + return *p +} + +func scopeContains(parent, child string) bool { + parent = strings.ToLower(strings.TrimRight(parent, "/")) + child = strings.ToLower(strings.TrimRight(child, "/")) + return parent != "" && (parent == child || strings.HasPrefix(child, parent+"/")) +} + +// ARM operation wildcards are not filesystem globs: '*' also matches separators. +func actionMatches(pattern, action string) bool { + pattern = regexp.QuoteMeta(strings.ToLower(pattern)) + pattern = strings.ReplaceAll(pattern, `\*`, ".*") + matched, _ := regexp.MatchString("^"+pattern+"$", strings.ToLower(action)) + return matched +} + +func anyAction(patterns []*string, action string) bool { + for _, p := range patterns { + if p != nil && actionMatches(*p, action) { + return true + } + } + return false +} + +func permissionAllows(permissions []*armauthorization.Permission, action string) bool { + for _, p := range permissions { + if p != nil && anyAction(p.Actions, action) && !anyAction(p.NotActions, action) { + return true + } + } + return false +} + +type callerIdentity struct { + objectID string + groups map[string]bool + groupOverage bool +} + +func identityFromToken(token string) (callerIdentity, error) { + parts := strings.Split(token, ".") + if len(parts) != 3 { + return callerIdentity{}, fmt.Errorf("operator token has no supported object-ID evidence") + } + raw, err := base64.RawURLEncoding.DecodeString(parts[1]) + if err != nil { + return callerIdentity{}, fmt.Errorf("decode operator identity evidence: %w", err) + } + var claims struct { + ObjectID string `json:"oid"` + Groups []string `json:"groups"` + HasGroups bool `json:"hasgroups"` + ClaimNames map[string]string `json:"_claim_names"` + } + if err := json.Unmarshal(raw, &claims); err != nil || claims.ObjectID == "" { + return callerIdentity{}, fmt.Errorf("operator token has no supported object-ID evidence") + } + caller := callerIdentity{ + objectID: claims.ObjectID, groups: map[string]bool{}, + groupOverage: claims.HasGroups || claims.ClaimNames["groups"] != "", + } + for _, g := range claims.Groups { + caller.groups[strings.ToLower(g)] = true + } + return caller, nil +} + +func (c callerIdentity) includes(id string) bool { + return strings.EqualFold(c.objectID, id) || c.groups[strings.ToLower(id)] +} diff --git a/cli/azd/extensions/azure.ai.projects/internal/onboarding/authorization_sdk.go b/cli/azd/extensions/azure.ai.projects/internal/onboarding/authorization_sdk.go new file mode 100644 index 00000000000..b4c6037393c --- /dev/null +++ b/cli/azd/extensions/azure.ai.projects/internal/onboarding/authorization_sdk.go @@ -0,0 +1,274 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +package onboarding + +import ( + "context" + "fmt" + "slices" + "strings" + + "github.com/Azure/azure-sdk-for-go/sdk/azcore" + "github.com/Azure/azure-sdk-for-go/sdk/azcore/arm" + "github.com/Azure/azure-sdk-for-go/sdk/azcore/cloud" + "github.com/Azure/azure-sdk-for-go/sdk/azcore/policy" + "github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/authorization/armauthorization/v2" +) + +type scopedAuthorizer struct { + credential azcore.TokenCredential + options *arm.ClientOptions + reads *AuthorizationEvidence +} + +type operation struct { + checkID, action, scope string +} + +// NewAuthorizer checks basic scoped ARM action allow without trial writes, deny reads or Graph calls. +func NewAuthorizer(credential azcore.TokenCredential, options *arm.ClientOptions) Authorizer { + return &scopedAuthorizer{credential: credential, options: onboardingARMOptions(options)} +} + +// NewAuthorizerWithEvidence reuses request-local evidence across preflight and deployment bindings. +func NewAuthorizerWithEvidence( + credential azcore.TokenCredential, options *arm.ClientOptions, evidence *AuthorizationEvidence, +) Authorizer { + return &scopedAuthorizer{credential: credential, options: onboardingARMOptions(options), reads: evidence} +} + +// CheckClusterDeployment checks deployment-write permission in the AKS resource group without checking supporting resources. +func CheckClusterDeployment( + ctx context.Context, clusterID string, credential azcore.TokenCredential, options *arm.ClientOptions, +) (Report, error) { + cluster, err := ParseID(clusterID, "Microsoft.ContainerService/managedClusters") + if err != nil { + return nil, err + } + a := &scopedAuthorizer{credential: credential, options: onboardingARMOptions(options)} + a.reads = NewAuthorizationEvidence(a.credential, a.options) + caller, tokenErr, identityErr := a.operatorIdentity(ctx) + report := a.checkOperations(ctx, []operation{ + {"P09", "Microsoft.Resources/deployments/write", cluster.Parent.String()}, + }, caller, tokenErr, identityErr) + return report, report.Err() +} + +func supportingOperations(plan Plan, facts Facts) ([]operation, error) { + var operations []operation + add := func(id, action, scope string) { + op := operation{id, action, scope} + if !slices.ContainsFunc(operations, func(existing operation) bool { + return existing.checkID == id && strings.EqualFold(existing.action, action) && + strings.EqualFold(existing.scope, scope) + }) { + operations = append(operations, op) + } + } + for _, identity := range []Resource{plan.Hosting, plan.Workload} { + if identity.Missing != "" || identity.LifecycleMissing != "" { + continue + } + actions := []string{"write", "read", "assign/action"} + if identity.Supplied { + actions = []string{"read", "assign/action"} + } + for _, action := range actions { + add("P02", "Microsoft.ManagedIdentity/userAssignedIdentities/"+action, facts.scope(identity)) + } + } + if !plan.Storage.Supplied && plan.Storage.Missing == "" && plan.Storage.LifecycleMissing == "" { + add("P04", "Microsoft.Storage/storageAccounts/write", facts.scope(plan.Storage)) + } + if plan.Registry.ID != "" && plan.Registry.Missing == "" && plan.Registry.LifecycleMissing == "" { + add("P05", "Microsoft.ContainerRegistry/registries/write", facts.scope(plan.Registry)) + } + for _, op := range plan.Operations { + add(op.checkID, op.action, op.scope) + } + target := "/subscriptions/" + plan.SubscriptionID + "/resourceGroups/" + plan.ResourceGroup + if plan.Grants != nil { + for _, grant := range plan.Grants { + if grant.DeploymentMissing != "" || grant.DeploymentScope == "" { + continue + } + scope := grant.DeploymentScope + if plan.TargetScopeMissing != "" { + if grant.CheckID == "G01" { + add("P09", "Microsoft.Resources/deployments/write", scope) + } + continue + } + if strings.EqualFold(scope, target) { + if grant.CheckID == "G01" { + add("P09", "Microsoft.Resources/deployments/write", scope) + } + continue + } + for _, action := range []string{"write", "read", "operations/read"} { + add("P09", "Microsoft.Resources/deployments/"+action, scope) + } + } + return operations, nil + } + for _, item := range []struct { + resource Resource + checkID string + }{ + {plan.Cluster, "G01"}, {plan.Workload, "G02"}, {plan.Storage, "G03"}, + } { + if !createsGrant(plan, item.checkID) { + continue + } + if item.resource.Missing != "" || item.resource.LifecycleMissing != "" { + continue + } + id, err := arm.ParseResourceID(item.resource.ID) + if err != nil { + return nil, fmt.Errorf("resolve cross-resource-group deployment scope: %w", err) + } + scope := id.Parent.String() + if strings.EqualFold(scope, target) { + if item.checkID == "G01" { + add("P09", "Microsoft.Resources/deployments/write", scope) + } + continue + } + for _, action := range []string{"write", "read", "operations/read"} { + add("P09", "Microsoft.Resources/deployments/"+action, scope) + } + } + return operations, nil +} + +func (a *scopedAuthorizer) Check(ctx context.Context, p Plan, f Facts) (Report, error) { + local := *a + local.reads = a.evidence() + a = &local + report := a.suppliedIdentityChecks(ctx, p, f) + report = append(report, permissionPlanSkips(p)...) + caller, tokenErr, identityErr := a.operatorIdentity(ctx) + basicIdentityErr := identityErr + if tokenErr != nil { + basicIdentityErr = tokenErr + } + report = append(report, a.checkP06(ctx, p, f, caller, basicIdentityErr)...) + if p.Registry.ID != "" || p.Registry.Missing != "" { + report = append(report, a.checkP07(ctx, p, f, caller, basicIdentityErr)) + } + operations, err := supportingOperations(p, f) + if err != nil { + return report, err + } + report = append(report, a.checkOperations(ctx, operations, caller, tokenErr, identityErr)...) + return report, report.Err() +} + +func (a *scopedAuthorizer) operatorIdentity( + ctx context.Context, +) (caller callerIdentity, tokenErr, identityErr error) { + audience := "https://management.azure.com/" + if a.options != nil { + if service, ok := a.options.Cloud.Services[cloud.ResourceManager]; ok && service.Audience != "" { + audience = service.Audience + } + } + token, tokenErr := a.credential.GetToken(ctx, policy.TokenRequestOptions{ + Scopes: []string{strings.TrimRight(audience, "/") + "/.default"}, + }) + caller, identityErr = identityFromToken(token.Token) + return caller, tokenErr, identityErr +} + +func (a *scopedAuthorizer) checkOperations( + ctx context.Context, operations []operation, caller callerIdentity, tokenErr, identityErr error, +) Report { + checks := make(Report, len(operations)) + tasks := make([]func(), 0, len(operations)) + for i, op := range operations { + tasks = append(tasks, func() { + check := Check{ + ID: op.checkID, Scope: op.scope, Status: Unverified, Mandatory: true, + Expected: op.action, Missing: op.action, + Remediation: "Ask the scope administrator to provide readable operator role assignments " + + "and definitions permitting " + op.action + " at this scope or an applicable ancestor.", + } + switch { + case op.scope == "": + check.Observed = "current resource existence is unverified; concrete or existing parent scope is unknown" + check.Missing = "verified current resource existence and actual permission scope" + case tokenErr != nil: + check.Observed, check.Cause = "operator token evidence unavailable", tokenErr + case identityErr != nil: + check.Observed, check.Cause = identityErr.Error(), identityErr + default: + check.Status, check.Observed, check.Cause = a.basicAction(ctx, op.scope, caller, op.action, false) + if check.Status == Pass { + check.Observed += "; ARM authorizes the actual operation" + check.Missing, check.Remediation = "", "" + } + } + checks[i] = check + }) + } + parallelReads(tasks...) + return checks +} + +func permissionPlanSkips(plan Plan) Report { + var report Report + for _, resource := range []Resource{plan.Hosting, plan.Workload} { + if missing := resource.Missing + resource.LifecycleMissing; missing != "" { + report = append(report, SkippedCheck("P02", resource.diagnosticScope(), + "managed identity lifecycle/read/assign permissions", missing)) + } + } + if missing := plan.Storage.Missing + plan.Storage.LifecycleMissing; missing != "" { + report = append(report, SkippedCheck("P04", plan.Storage.diagnosticScope(), "Storage creation permission", missing)) + } + if missing := plan.Registry.Missing + plan.Registry.LifecycleMissing; missing != "" { + report = append(report, SkippedCheck("P05", plan.Registry.diagnosticScope(), "native registry creation", missing)) + } + return report +} + +func (e *AuthorizationEvidence) resolveGrantRole( + ctx context.Context, subscription string, grant grantRequest, +) (string, error) { + if scope, err := arm.ParseResourceID(grant.scope); err == nil && scope.SubscriptionID != "" { + subscription = scope.SubscriptionID + } + if grant.roleID != "" { + return "/subscriptions/" + subscription + "/providers/Microsoft.Authorization/roleDefinitions/" + grant.roleID, nil + } + return e.ResolveRoleByName(ctx, subscription, grant.roleName) +} + +// ResolveRoleByName resolves a built-in role, sharing concurrent reads and errors by subscription and exact role name. +func (e *AuthorizationEvidence) ResolveRoleByName( + ctx context.Context, subscription, roleName string, +) (string, error) { + return e.roleNames.get(strings.ToLower(subscription)+"|"+roleName, func() (string, error) { + client, err := e.definitionClient() + if err != nil { + return "", err + } + pager := client.NewListPager("/subscriptions/"+subscription, + &armauthorization.RoleDefinitionsClientListOptions{Filter: new("roleName eq '" + roleName + "'")}) + for pager.More() { + page, err := pager.NextPage(ctx) + if err != nil { + return "", err + } + for _, role := range page.Value { + if role != nil && role.ID != nil && role.Properties != nil && + authString(role.Properties.RoleName) == roleName && + strings.EqualFold(authString(role.Properties.RoleType), "BuiltInRole") { + return *role.ID, nil + } + } + } + return "", fmt.Errorf("required built-in role %q is unavailable", roleName) + }) +} diff --git a/cli/azd/extensions/azure.ai.projects/internal/onboarding/authorization_test.go b/cli/azd/extensions/azure.ai.projects/internal/onboarding/authorization_test.go new file mode 100644 index 00000000000..53344b061d2 --- /dev/null +++ b/cli/azd/extensions/azure.ai.projects/internal/onboarding/authorization_test.go @@ -0,0 +1,380 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +package onboarding + +import ( + "context" + "errors" + "fmt" + "net/http" + "strings" + "testing" + + "github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/authorization/armauthorization/v2" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +const ownerRole = testSubscription + "/providers/Microsoft.Authorization/roleDefinitions/owner" +const federationRoleGUID = "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa" +const assignmentSuffix = "/providers/Microsoft.Authorization/roleAssignments" + +func assignment(scope, principal, role string) map[string]any { + return map[string]any{ + "id": scope + assignmentSuffix + "/assignment-" + principal, + "properties": map[string]any{"scope": scope, "principalId": principal, "roleDefinitionId": role}, + } +} + +func authorizationFake() *readFake { + fake := &readFake{replies: map[string]readReply{}} + fake.handler = func(request *http.Request) (readReply, bool) { + path := request.URL.Path + if _, overridden := fake.replies[strings.ToLower(path)]; overridden { + return readReply{}, false + } + if strings.HasSuffix(strings.ToLower(path), strings.ToLower(assignmentSuffix)) { + scope := path[:len(path)-len(assignmentSuffix)] + return readReply{200, map[string]any{"value": []any{ + assignment(scope, testOperator, ownerRole), + assignment(scope, "hosting-principal", ownerRole), + assignment(scope, "workload-principal", ownerRole), + }}}, true + } + if strings.HasSuffix(strings.ToLower(path), "/roledefinitions") { + return readReply{200, map[string]any{"value": []any{map[string]any{ + "id": path + "/" + federationRoleGUID, + "properties": map[string]any{ + "roleName": "Managed Identity Federated Identity Credential Contributor", "type": "BuiltInRole", + }, + }}}}, true + } + if strings.Contains(strings.ToLower(path), "/roledefinitions/") { + permissions := map[string]any{} + switch path[strings.LastIndex(path, "/")+1:] { + case "owner": + permissions["actions"], permissions["dataActions"] = []string{"*"}, []string{"*"} + case "acdd72a7-3385-48ef-bd42-f606fba81ae7": + permissions["actions"] = []string{"*/read"} + case "85cb6faf-e071-4c9b-8136-154b5a04f717": + permissions["actions"] = []string{"Microsoft.KubernetesConfiguration/extensions/*"} + case federationRoleGUID: + permissions["actions"] = []string{ + "Microsoft.ManagedIdentity/userAssignedIdentities/federatedIdentityCredentials/*", + } + case "ba92f5b4-2d11-453d-a403-e96b0029c9fe": + permissions["dataActions"] = []string{"Microsoft.Storage/storageAccounts/blobServices/containers/blobs/*"} + default: + return readReply{}, false + } + return readReply{200, map[string]any{ + "id": path, "properties": map[string]any{"permissions": []any{permissions}}, + }}, true + } + return readReply{}, false + } + return fake +} + +func suppliedFacts() Facts { + return Facts{ + TenantID: "tenant", + HostingIdentity: Identity{ID: testHosting, PrincipalID: "hosting-principal"}, + WorkloadIdentity: Identity{ID: testWorkload, PrincipalID: "workload-principal"}, + Exists: map[string]bool{ + strings.ToLower(testHosting): true, strings.ToLower(testWorkload): true, strings.ToLower(testStorage): true, + }, + } +} + +func TestSuppliedGrantChecksUseCurrentExistenceWithoutRecoveryExemptions(t *testing.T) { + for _, id := range []string{"G02", "G03"} { + for _, state := range []string{"absent", "existing", "unknown", "supplied"} { + t.Run(id+"/"+state, func(t *testing.T) { + plan, facts := suppliedPlan(t), suppliedFacts() + target := &plan.Workload + if id == "G03" { + target = &plan.Storage + plan.Hosting.Supplied = false + } + target.Supplied = state == "supplied" + delete(facts.Exists, strings.ToLower(target.ID)) + if state == "existing" { + facts.Exists[strings.ToLower(target.ID)] = true + } else if state == "absent" { + facts.Exists[strings.ToLower(target.ID)] = false + } + fake := authorizationFake() + fake.replies[strings.ToLower(target.ID+assignmentSuffix)] = + readReply{200, map[string]any{"value": []any{}}} + authorizer := NewAuthorizer(testCredential{}, fake.options()).(*scopedAuthorizer) + report := authorizer.suppliedIdentityChecks(t.Context(), plan, facts) + check := checkWithID(t, report, id) + if state == "absent" { + assert.Equal(t, NotApplicable, check.Status) + assert.Zero(t, fake.count(target.ID+assignmentSuffix)) + } else { + assert.Equal(t, Fail, check.Status) + assert.NotEmpty(t, check.Remediation) + assert.Contains(t, check.Remediation, "does not repair") + } + }) + } + } +} + +func TestRoleAssignmentPermissionUsesConcreteOrExistingParentScope(t *testing.T) { + for _, existing := range []bool{false, true} { + plan, facts := suppliedPlan(t), suppliedFacts() + plan.Storage.Supplied = false + facts.Exists[strings.ToLower(testStorage)] = existing + fake := authorizationFake() + authorizer := NewAuthorizer(testCredential{}, fake.options()).(*scopedAuthorizer) + report := authorizer.checkP06(t.Context(), plan, facts, callerIdentity{objectID: testOperator}, nil) + require.Len(t, report, 1) + require.NoError(t, report.Err()) + expected := testGroup + if existing { + expected = testStorage + } + assert.Equal(t, expected, report[0].Scope) + assert.Equal(t, 1, fake.count(expected+assignmentSuffix)) + } + for _, allowed := range []bool{true, false} { + t.Run("known ancestor with unresolved resource/"+fmt.Sprint(allowed), func(t *testing.T) { + plan, facts := suppliedPlan(t), suppliedFacts() + plan.Grants = []PlannedGrant{{ + CheckID: "G03", Target: Resource{Origin: "Storage in edited module", Missing: "runtime Storage name"}, + AuthorityScope: testGroup, DeploymentScope: testGroup, + }} + fake := authorizationFake() + if !allowed { + fake.replies[strings.ToLower(testGroup+assignmentSuffix)] = + readReply{200, map[string]any{"value": []any{}}} + } + authorizer := NewAuthorizer(testCredential{}, fake.options()).(*scopedAuthorizer) + report := authorizer.checkP06(t.Context(), plan, facts, callerIdentity{objectID: testOperator}, nil) + require.Len(t, report, 1) + assert.Equal(t, testGroup, report[0].Scope) + if allowed { + assert.Equal(t, Pass, report[0].Status) + } else { + assert.Equal(t, Fail, report[0].Status) + require.Error(t, report.Err()) + } + assert.Equal(t, 1, fake.count(testGroup+assignmentSuffix)) + }) + } + for _, field := range []string{"condition", "conditionVersion", "delegatedManagedIdentityResourceId"} { + t.Run(field, func(t *testing.T) { + plan, facts := suppliedPlan(t), suppliedFacts() + plan.Storage.Supplied = false + fake := authorizationFake() + grant := assignment(testStorage, testOperator, ownerRole) + grant["properties"].(map[string]any)[field] = "unsupported" + fake.replies[strings.ToLower(testStorage+assignmentSuffix)] = + readReply{200, map[string]any{"value": []any{grant}}} + authorizer := NewAuthorizer(testCredential{}, fake.options()).(*scopedAuthorizer) + report := authorizer.checkP06(t.Context(), plan, facts, callerIdentity{objectID: testOperator}, nil) + require.Error(t, report.Err()) + assert.Equal(t, Fail, report[0].Status) + assert.Contains(t, report[0].Missing, "unconditional") + }) + } +} + +func TestSupportingOperationsKeepReferencesReadOnlyAndDefaultWrites(t *testing.T) { + plan, facts := suppliedPlan(t), suppliedFacts() + operations, err := supportingOperations(plan, facts) + require.NoError(t, err) + require.Len(t, operations, 4) + for _, operation := range operations { + assert.Equal(t, "P02", operation.checkID) + assert.NotContains(t, operation.action, "/write") + } + plan.Storage.Supplied = false + operations, err = supportingOperations(plan, facts) + require.NoError(t, err) + assert.Contains(t, operations, operation{"P04", "Microsoft.Storage/storageAccounts/write", testStorage}) + plan.Workload.ID = strings.Replace(testWorkload, "/target-rg/", "/other-rg/", 1) + plan.Hosting.Supplied = false + plan.Cluster.ID = testGroup + "/providers/Microsoft.ContainerService/managedClusters/cluster" + plan.Cluster.Parent = testGroup + operations, err = supportingOperations(plan, facts) + require.NoError(t, err) + assert.Contains(t, operations, operation{"P09", "Microsoft.Resources/deployments/write", testGroup}) + for _, action := range []string{"write", "read", "operations/read"} { + assert.Contains(t, operations, operation{"P09", "Microsoft.Resources/deployments/" + action, + testSubscription + "/resourceGroups/other-rg"}) + } + for _, operation := range operations { + assert.NotContains(t, operation.action, "Microsoft.ContainerRegistry") + assert.NotContains(t, operation.action, "Microsoft.Network") + } +} + +func permission(actions, notActions, data, notData []string) *armauthorization.Permission { + pointers := func(values []string) []*string { + var result []*string + for _, value := range values { + result = append(result, new(value)) + } + return result + } + return &armauthorization.Permission{ + Actions: pointers(actions), NotActions: pointers(notActions), + DataActions: pointers(data), NotDataActions: pointers(notData), + } +} + +func TestEquivalentIdentityPermissionsPreserveActionsAndDataActions(t *testing.T) { + read := "Microsoft.Storage/storageAccounts/read" + blob := "Microsoft.Storage/storageAccounts/blobServices/containers/blobs/*" + required := []*armauthorization.Permission{permission([]string{read}, nil, []string{blob}, nil)} + for _, test := range []struct { + name string + available []*armauthorization.Permission + allowed bool + }{ + {"wildcard owner", []*armauthorization.Permission{permission([]string{"*"}, nil, []string{"*"}, nil)}, true}, + {"control plane does not imply data plane", + []*armauthorization.Permission{permission([]string{"*"}, nil, nil, nil)}, false}, + {"union of applicable roles", []*armauthorization.Permission{ + permission([]string{read}, nil, nil, nil), permission(nil, nil, []string{blob}, nil), + }, true}, + {"NotActions denies required read", + []*armauthorization.Permission{permission([]string{"*"}, []string{read}, []string{"*"}, nil)}, false}, + {"NotDataActions denies required write", + []*armauthorization.Permission{permission([]string{"*"}, nil, []string{"*"}, []string{ + "Microsoft.Storage/storageAccounts/blobServices/containers/blobs/write", + })}, false}, + {"missing permissions", nil, false}, + } { + t.Run(test.name, func(t *testing.T) { assert.Equal(t, test.allowed, roleCovers(test.available, required)) }) + } + assert.True(t, actionMatches("Microsoft.Resources/*", "Microsoft.Resources/deployments/operations/read")) +} + +func TestIdentityEvidenceHonorsInheritanceConditionsAndPaging(t *testing.T) { + grant := grantRequests(suppliedPlan(t), Identity{}, suppliedFacts().WorkloadIdentity)[3] + for _, state := range []string{ + "inherited", "management group", "unrelated", "conditional", "missing", "incomplete", "paged", + } { + t.Run(state, func(t *testing.T) { + fake := authorizationFake() + scope := testGroup + if state == "management group" { + scope = "/providers/Microsoft.Management/managementGroups/parent" + } else if state == "unrelated" { + scope = testSubscription + "/resourceGroups/unrelated" + } + evidence := assignment(scope, "workload-principal", ownerRole) + page := map[string]any{"value": []any{evidence}} + switch state { + case "conditional": + evidence["properties"].(map[string]any)["condition"] = "unsupported" + case "missing": + page["value"] = []any{} + case "incomplete": + delete(evidence["properties"].(map[string]any), "roleDefinitionId") + case "paged": + fake.replies["/next-role-page"] = readReply{200, page} + page = map[string]any{"value": []any{}, "nextLink": "https://management.azure.com/next-role-page"} + } + fake.replies[strings.ToLower(testStorage+assignmentSuffix)] = readReply{200, page} + authorizer := NewAuthorizer(testCredential{}, fake.options()).(*scopedAuthorizer) + status, _, err := authorizer.identityGrantAllows(t.Context(), "target-sub", grant) + require.NoError(t, err) + switch state { + case "inherited", "management group", "paged": + assert.Equal(t, Pass, status) + case "conditional", "incomplete": + assert.Equal(t, Unverified, status) + case "missing", "unrelated": + assert.Equal(t, Fail, status) + } + }) + } +} + +func TestAuthorizationReadsAreCachedOnlyWithinOnePass(t *testing.T) { + fake := authorizationFake() + authorizer := NewAuthorizer(testCredential{}, fake.options()) + for pass := 1; pass <= 2; pass++ { + report, err := authorizer.Check(t.Context(), suppliedPlan(t), suppliedFacts()) + require.NoError(t, err) + require.NoError(t, report.Err()) + assert.Equal(t, pass, fake.count(testCluster+assignmentSuffix), "G01 has two role requirements at the same scope") + assert.Equal(t, pass, fake.count(testWorkload+assignmentSuffix), "G02 and P02 share raw scope evidence") + assert.Equal(t, pass, fake.count(ownerRole), "roles shared by multiple scopes are read once per pass") + } +} + +func TestOperatorGroupEvidenceAndManagementGroupAncestry(t *testing.T) { + fake := authorizationFake() + original := fake.handler + fake.handler = func(request *http.Request) (readReply, bool) { + if strings.HasSuffix(strings.ToLower(request.URL.Path), strings.ToLower(assignmentSuffix)) { + values := []any{} + if strings.HasPrefix(request.URL.Query().Get("$filter"), "assignedTo(") { + scope := "/providers/Microsoft.Management/managementGroups/parent" + values = append(values, assignment(scope, "operator-group", ownerRole)) + } + return readReply{200, map[string]any{"value": values}}, true + } + return original(request) + } + authorizer := NewAuthorizer(testCredential{}, fake.options()).(*scopedAuthorizer) + caller := callerIdentity{objectID: testOperator, groupOverage: true} + status, _, err := authorizer.basicAction(t.Context(), testStorage, caller, roleAssignmentWrite, true) + require.NoError(t, err) + assert.Equal(t, Pass, status) + assert.Equal(t, 2, fake.count(testStorage+assignmentSuffix)) + caller.objectID = "invalid') or ('" + status, _, err = authorizer.basicAction(t.Context(), testStorage, caller, roleAssignmentWrite, true) + require.NoError(t, err) + assert.Equal(t, Unverified, status) + for _, request := range fake.requests { + assert.NotContains(t, request.filter, caller.objectID) + assert.NotContains(t, request.path, "graph") + } +} + +type resourceStub struct { + report Report + err error +} + +func (s resourceStub) Check(context.Context, Plan) (Facts, Report, error) { + if s.err != nil { + return Facts{}, s.report, s.err + } + return Facts{}, s.report, s.report.Err() +} + +type accessStub struct{ called *bool } + +func (s accessStub) Check(context.Context, Plan, Facts) (Report, error) { + *s.called = true + report := Report{{ID: "P06", Status: Unverified, Mandatory: true}} + return report, report.Err() +} + +func TestPipelineCollectsMandatoryFailuresAndPreservesServiceErrors(t *testing.T) { + cause := errors.New("original resource read error") + called := false + pipeline := Pipeline{ + Resources: resourceStub{report: Report{{ID: "S01", Status: Fail, Mandatory: true, Cause: cause}}}, + Access: accessStub{called: &called}, + } + _, report, err := pipeline.Preflight(t.Context(), suppliedPlan(t)) + require.ErrorIs(t, err, cause) + assert.True(t, called) + assert.Len(t, report, 2) + called = false + pipeline.Resources = resourceStub{err: context.Canceled} + _, _, err = pipeline.Preflight(t.Context(), suppliedPlan(t)) + require.ErrorIs(t, err, context.Canceled) + assert.False(t, called) +} diff --git a/cli/azd/extensions/azure.ai.projects/internal/onboarding/deployment_plan.go b/cli/azd/extensions/azure.ai.projects/internal/onboarding/deployment_plan.go new file mode 100644 index 00000000000..79786fc540a --- /dev/null +++ b/cli/azd/extensions/azure.ai.projects/internal/onboarding/deployment_plan.go @@ -0,0 +1,117 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +package onboarding + +import ( + "fmt" + "slices" + "strings" +) + +// LegacyNetwork describes selections of the existing standard-template network module. +type LegacyNetwork struct { + Enabled bool + VNetID string + AgentSubnetName string + PESubnetName string + PESubnetPrefix string + CreateAgentSubnet bool + CreatePESubnet bool + ManagedEgress bool + IsolationMode string + DNSResourceGroup string + DNSSubscription string +} + +// DeploymentSelection contains effective network selections from the standard Foundry template. +type DeploymentSelection struct { + Network *LegacyNetwork +} + +// AddOperation adds one identified action/scope using the preflight's existing deduplication contract. +func (plan *Plan) AddOperation(checkID, action, scope string) { + op := operation{checkID, action, scope} + if !slices.ContainsFunc(plan.Operations, func(existing operation) bool { + return existing.checkID == op.checkID && strings.EqualFold(existing.action, op.action) && + strings.EqualFold(existing.scope, op.scope) + }) { + plan.Operations = append(plan.Operations, op) + } +} + +// WithDeploymentOperations binds selected standard-network resource IDs and permission scopes. +func WithDeploymentOperations(plan Plan, selected DeploymentSelection) (Plan, error) { + plan.Operations = slices.Clone(plan.Operations) + plan.LegacyNetwork = selected.Network + n := selected.Network + if n == nil || !n.Enabled { + return plan, nil + } + if _, err := existingResourceID(n.VNetID, "Microsoft.Network/virtualNetworks"); err != nil { + return plan, fmt.Errorf("P03: selected network VNet scope: %w", err) + } + if !n.ManagedEgress { + id := n.VNetID + "/subnets/" + n.AgentSubnetName + subnet, err := existingResourceID(id, "Microsoft.Network/virtualNetworks/subnets") + if err != nil { + return plan, fmt.Errorf("N02: selected network.agentSubnet: %w", err) + } + plan.AgentSubnet = Resource{ID: id, Parent: subnet.Parent.String(), Supplied: !n.CreateAgentSubnet} + } + if _, err := existingResourceID(n.VNetID+"/subnets/"+n.PESubnetName, + "Microsoft.Network/virtualNetworks/subnets"); err != nil { + return plan, fmt.Errorf("P03: selected private-endpoint subnet scope: %w", err) + } + target := "/subscriptions/" + plan.SubscriptionID + if plan.ResourceGroupSupplied { + target += "/resourceGroups/" + plan.ResourceGroup + } + appendOperation := func(action, scope string) { + plan.AddOperation("P03", action, scope) + } + appendOperation("Microsoft.Network/virtualNetworks/read", n.VNetID) + if n.CreateAgentSubnet || n.CreatePESubnet { + appendOperation("Microsoft.Network/virtualNetworks/subnets/write", n.VNetID) + } + peScope := n.VNetID + "/subnets/" + n.PESubnetName + if n.CreatePESubnet { + peScope = n.VNetID + } + appendOperation("Microsoft.Network/virtualNetworks/subnets/join/action", peScope) + if !n.ManagedEgress { + agentScope := n.VNetID + "/subnets/" + n.AgentSubnetName + if n.CreateAgentSubnet { + agentScope = n.VNetID + } + if !strings.EqualFold(agentScope, peScope) { + appendOperation("Microsoft.Network/virtualNetworks/subnets/join/action", agentScope) + } + } + appendOperation("Microsoft.Network/privateEndpoints/write", target) + appendOperation("Microsoft.Network/privateEndpoints/privateDnsZoneGroups/write", target) + if n.DNSResourceGroup == "" { + appendOperation("Microsoft.Network/virtualNetworks/join/action", n.VNetID) + appendOperation("Microsoft.Network/privateDnsZones/write", target) + appendOperation("Microsoft.Network/privateDnsZones/virtualNetworkLinks/write", target) + appendOperation("Microsoft.Network/privateDnsZones/join/action", target) + } else { + subscription := n.DNSSubscription + if subscription == "" { + subscription = plan.SubscriptionID + } + for _, zone := range []string{ + "privatelink.services.ai.azure.com", + "privatelink.openai.azure.com", + "privatelink.cognitiveservices.azure.com", + } { + scope := "/subscriptions/" + subscription + "/resourceGroups/" + n.DNSResourceGroup + + "/providers/Microsoft.Network/privateDnsZones/" + zone + if _, err := existingResourceID(scope, "Microsoft.Network/privateDnsZones"); err != nil { + return plan, fmt.Errorf("P03: selected DNS zone scope: %w", err) + } + appendOperation("Microsoft.Network/privateDnsZones/join/action", scope) + } + } + return plan, nil +} diff --git a/cli/azd/extensions/azure.ai.projects/internal/onboarding/identity_authorization.go b/cli/azd/extensions/azure.ai.projects/internal/onboarding/identity_authorization.go new file mode 100644 index 00000000000..f11ff5dec02 --- /dev/null +++ b/cli/azd/extensions/azure.ai.projects/internal/onboarding/identity_authorization.go @@ -0,0 +1,214 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +package onboarding + +import ( + "context" + "fmt" + "strings" + + "github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/authorization/armauthorization/v2" +) + +func (a *scopedAuthorizer) suppliedIdentityChecks(ctx context.Context, plan Plan, facts Facts) Report { + var report Report + for _, grant := range grantRequests(plan, facts.HostingIdentity, facts.WorkloadIdentity) { + principal, target := plan.Hosting, plan.Cluster + switch grant.checkID { + case "G02": + target = plan.Workload + case "G03": + principal, target = plan.Workload, plan.Storage + } + if missing := principal.Missing + principal.LifecycleMissing; missing != "" { + report = append(report, SkippedCheck(grant.checkID, principal.diagnosticScope(), + "supplied identity grant requirements", missing)) + continue + } + needed := grant.checkID == "G01" && plan.Hosting.Supplied || + grant.checkID == "G02" && plan.Hosting.Supplied || + grant.checkID == "G03" && plan.Workload.Supplied + if !needed { + continue + } + if missing := target.Missing + target.LifecycleMissing; missing != "" { + report = append(report, SkippedCheck(grant.checkID, target.diagnosticScope(), + "supplied identity access to its dependency", missing)) + continue + } + check := resourceCheck(grant.checkID, grant.scope, + "supplied identity "+grant.principalID+" has "+grant.roleName+" or provably equivalent inherited permissions") + future := grant.checkID == "G02" && !plan.Workload.Supplied && facts.absent(plan.Workload) || + grant.checkID == "G03" && !plan.Storage.Supplied && facts.absent(plan.Storage) + if future { + report = append(report, resourceResult(check, NotApplicable, + "dependency does not yet exist; the same ARM graph will create the dependency and grant")) + continue + } + if grant.principalID == "" { + check.Observed, check.Missing = "supplied identity metadata is unavailable", "verified identity principal ID" + report = append(report, check) + continue + } + check.Status, check.Observed, check.Cause = a.identityGrantAllows(ctx, plan.SubscriptionID, grant) + if check.Status != Pass && check.Status != NotApplicable { + check.Missing = "readable applicable unconditional role permissions for the supplied identity" + check.Remediation = "Have the resource administrator configure the supplied identity's required permissions; " + + "azd does not repair customer-preexisting grants." + } + report = append(report, check) + } + return report +} + +func (a *scopedAuthorizer) identityGrantAllows( + ctx context.Context, subscription string, grant grantRequest, +) (Status, string, error) { + reads := a.evidence() + roleID, err := reads.resolveGrantRole(ctx, subscription, grant) + if err != nil { + return Unverified, "required role definition is unavailable", err + } + required, err := reads.roleDefinition(ctx, roleID) + if err != nil { + return Unverified, "required role definition cannot be read", err + } + if required.Properties == nil || len(required.Properties.Permissions) == 0 { + return Unverified, "required role permission evidence is missing", nil + } + assignments, err := reads.filteredAssignments(ctx, grant.scope, "atScope()") + if err != nil { + return Unverified, "supplied identity role assignments cannot be read", err + } + var uncertain bool + var permissions []*armauthorization.Permission + for _, assignment := range assignments { + if assignment == nil || assignment.Properties == nil { + uncertain = true + continue + } + p := assignment.Properties + if !strings.EqualFold(authString(p.PrincipalID), grant.principalID) { + continue + } + if authString(p.Scope) == "" || authString(p.RoleDefinitionID) == "" { + uncertain = true + continue + } + if !assignmentApplies(assignment, grant.scope) { + continue + } + if authString(p.Condition) != "" || authString(p.ConditionVersion) != "" || + authString(p.DelegatedManagedIdentityResourceID) != "" { + uncertain = true + continue + } + role := required.RoleDefinition + if !strings.EqualFold(*p.RoleDefinitionID, roleID) { + result, readErr := reads.roleDefinition(ctx, *p.RoleDefinitionID) + if readErr != nil { + return Unverified, "supplied identity role definition cannot be read", readErr + } + role = result.RoleDefinition + } + if role.Properties == nil || len(role.Properties.Permissions) == 0 { + uncertain = true + continue + } + permissions = append(permissions, role.Properties.Permissions...) + if roleCovers(permissions, required.Properties.Permissions) { + return Pass, "verified supplied-identity permission allow at an applicable scope", nil + } + } + if uncertain { + return Unverified, + "necessary identity permission evidence is missing or conditional/delegated evidence is unsupported", nil + } + return Fail, fmt.Sprintf("no confirmed permission allow for identity %s and role %s", + grant.principalID, grant.roleName), nil +} + +// coversPattern deliberately recognizes only provable wildcard containment, not condition expressions. +func coversPattern(allow, required string) bool { + allow, required = strings.ToLower(allow), strings.ToLower(required) + if allow == "*" || allow == required { + return true + } + if !strings.Contains(required, "*") { + return actionMatches(allow, required) + } + return strings.HasSuffix(allow, "*") && strings.Count(allow, "*") == 1 && + strings.HasPrefix(required, strings.TrimSuffix(allow, "*")) +} + +func patternsMayOverlap(a, b string) bool { + a, b = strings.ToLower(a), strings.ToLower(b) + if !strings.Contains(a, "*") { + return actionMatches(b, a) + } + if !strings.Contains(b, "*") { + return actionMatches(a, b) + } + ap, bp := strings.Split(a, "*"), strings.Split(b, "*") + return (strings.HasPrefix(ap[0], bp[0]) || strings.HasPrefix(bp[0], ap[0])) && + (strings.HasSuffix(ap[len(ap)-1], bp[len(bp)-1]) || strings.HasSuffix(bp[len(bp)-1], ap[len(ap)-1])) +} + +func roleCovers(available, required []*armauthorization.Permission) bool { + hasPermission := false + for _, permission := range required { + if permission == nil { + return false + } + for _, data := range []bool{false, true} { + actions, except := permission.Actions, permission.NotActions + if data { + actions, except = permission.DataActions, permission.NotDataActions + } + for _, action := range actions { + if action == nil || strings.TrimSpace(*action) == "" { + return false + } + hasPermission = true + covered := false + for _, candidate := range available { + if candidate == nil { + continue + } + allows, exclusions := candidate.Actions, candidate.NotActions + if data { + allows, exclusions = candidate.DataActions, candidate.NotDataActions + } + for _, allow := range allows { + if allow == nil || !coversPattern(*allow, *action) { + continue + } + blocked := false + for _, exclusion := range exclusions { + if exclusion == nil { + blocked = true + break + } + if !patternsMayOverlap(*exclusion, *action) { + continue + } + irrelevant := false + for _, removed := range except { + if removed != nil && coversPattern(*removed, *exclusion) { + irrelevant = true + } + } + blocked = blocked || !irrelevant + } + covered = covered || !blocked + } + } + if !covered { + return false + } + } + } + } + return hasPermission +} diff --git a/cli/azd/extensions/azure.ai.projects/internal/onboarding/managed_node_network.go b/cli/azd/extensions/azure.ai.projects/internal/onboarding/managed_node_network.go new file mode 100644 index 00000000000..f7a90b65a13 --- /dev/null +++ b/cli/azd/extensions/azure.ai.projects/internal/onboarding/managed_node_network.go @@ -0,0 +1,137 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +package onboarding + +import ( + "context" + "fmt" + "slices" + "strings" + + "github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/compute/armcompute/v7" + "github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/containerservice/armcontainerservice/v9" + "github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/resources/armresources" +) + +// Managed-VNet pools may omit vnetSubnetID. Resolve their subnet through the +// cluster-owned node RG and the selected pool's actual VMSS network configuration, +// never through generated VNet/VMSS names or unrelated pools. +func (c *sdkResourceChecker) nodeSubnetIDs( + ctx context.Context, plan Plan, cluster *armcontainerservice.ManagedCluster, pool *armcontainerservice.AgentPool, +) ([]string, Report) { + if pool != nil && pool.Properties != nil && resourceValue(pool.Properties.VnetSubnetID) != "" { + return []string{resourceValue(pool.Properties.VnetSubnetID)}, nil + } + check := resourceCheck("N02", plan.Cluster.ID, + "selected AKS pool Node subnet from cluster-owned node resource group and ARM VMSS network configuration") + check.Missing = "cluster nodeResourceGroup, exact managedBy cluster binding, selected pool VMSS and subnet IDs" + check.Remediation = "allow resourceGroups/read and virtualMachineScaleSets/read on the AKS node resource group, " + + "and virtualNetworks/subnets/read on its node subnet; retain fail-closed validation if evidence is unavailable" + clusterID, err := existingResourceID(plan.Cluster.ID, "Microsoft.ContainerService/managedClusters") + if err != nil || cluster == nil || cluster.Properties == nil || + !strings.EqualFold(resourceValue(cluster.ID), plan.Cluster.ID) || pool == nil || pool.Properties == nil { + check.Observed = "selected cluster or pool metadata is unavailable or mismatched" + return nil, Report{check} + } + poolName := resourceValue(pool.Name) + nodeRG := resourceValue(cluster.Properties.NodeResourceGroup) + if poolName == "" || strings.ContainsAny(poolName, "/\\?#%{} \t\r\n") || + !strings.EqualFold(resourceValue(pool.ID), plan.Cluster.ID+"/agentPools/"+poolName) || + nodeRG == "" || strings.ContainsAny(nodeRG, "/\\?#%{} \t\r\n") { + check.Observed = "selected pool resource ID or cluster nodeResourceGroup is absent or invalid" + return nil, Report{check} + } + scope := "/subscriptions/" + clusterID.SubscriptionID + "/resourceGroups/" + nodeRG + groups, err := armresources.NewResourceGroupsClient(clusterID.SubscriptionID, c.credential, c.options) + if err != nil { + check.Cause = err + return nil, Report{check} + } + group, err := groups.Get(ctx, nodeRG, nil) + if err != nil { + return nil, Report{managedNodeReadFailure(scope, "Microsoft.Resources/subscriptions/resourceGroups/read", err)} + } + check.Scope = scope + if !strings.EqualFold(resourceValue(group.ID), scope) || + !strings.EqualFold(resourceValue(group.ManagedBy), plan.Cluster.ID) { + check.Observed = "node resource group ID/managedBy does not prove association with the selected AKS cluster" + return nil, Report{check} + } + sets, err := armcompute.NewVirtualMachineScaleSetsClient(clusterID.SubscriptionID, c.credential, c.options) + if err != nil { + check.Cause = err + return nil, Report{check} + } + var subnetIDs []string + matches := 0 + pager := sets.NewListPager(nodeRG, nil) + for pager.More() { + page, err := pager.NextPage(ctx) + if err != nil { + return nil, Report{managedNodeReadFailure(scope, "Microsoft.Compute/virtualMachineScaleSets/read", err)} + } + for _, set := range page.Value { + if set == nil || !strings.EqualFold(resourceValue(set.Tags["aks-managed-poolName"]), poolName) { + continue + } + ids, err := managedPoolSubnetIDs(set, scope) + if err != nil { + check.Observed = "selected pool VMSS subnet evidence is incomplete or inconsistent" + check.Cause = err + return nil, Report{check} + } + matches++ + subnetIDs = append(subnetIDs, ids...) + } + } + if matches == 0 { + check.Observed = "no VMSS in the cluster-owned node resource group identifies the selected pool" + return nil, Report{check} + } + slices.Sort(subnetIDs) + subnetIDs = slices.Compact(subnetIDs) + check = resourceResult(check, Pass, fmt.Sprintf( + "node resource group managedBy matches selected cluster; %d selected-pool VMSS network profiles; subnet IDs=%s", + matches, strings.Join(subnetIDs, ", "))) + return subnetIDs, Report{check} +} + +func managedNodeReadFailure(scope, action string, err error) Check { + check := readFailure("N02", scope, action, err) + check.Status = Unverified + return check +} + +func managedPoolSubnetIDs(set *armcompute.VirtualMachineScaleSet, groupID string) ([]string, error) { + id, err := existingResourceID(resourceValue(set.ID), "Microsoft.Compute/virtualMachineScaleSets") + if err != nil || !strings.EqualFold(id.Parent.String(), groupID) { + return nil, fmt.Errorf("selected pool VMSS resource ID does not belong to the cluster-owned node resource group") + } + p := set.Properties + if p == nil || resourceValue(p.ProvisioningState) != "Succeeded" || + p.VirtualMachineProfile == nil || p.VirtualMachineProfile.NetworkProfile == nil { + return nil, fmt.Errorf("selected pool VMSS lacks a succeeded provisioning state or network profile") + } + interfaces := p.VirtualMachineProfile.NetworkProfile.NetworkInterfaceConfigurations + if len(interfaces) == 0 { + return nil, fmt.Errorf("selected pool VMSS has no network interface configurations") + } + var ids []string + for _, nic := range interfaces { + if nic == nil || nic.Properties == nil || len(nic.Properties.IPConfigurations) == 0 { + return nil, fmt.Errorf("selected pool VMSS has incomplete network interface configuration") + } + for _, ip := range nic.Properties.IPConfigurations { + if ip == nil || ip.Properties == nil || ip.Properties.Subnet == nil { + return nil, fmt.Errorf("selected pool VMSS IP configuration has no subnet reference") + } + raw := resourceValue(ip.Properties.Subnet.ID) + if _, err := existingResourceID(raw, "Microsoft.Network/virtualNetworks/subnets"); err != nil { + return nil, fmt.Errorf("selected pool VMSS subnet reference: %w", err) + } + ids = append(ids, raw) + } + } + return ids, nil +} diff --git a/cli/azd/extensions/azure.ai.projects/internal/onboarding/network_configuration.go b/cli/azd/extensions/azure.ai.projects/internal/onboarding/network_configuration.go new file mode 100644 index 00000000000..9efc1ac64b1 --- /dev/null +++ b/cli/azd/extensions/azure.ai.projects/internal/onboarding/network_configuration.go @@ -0,0 +1,204 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +package onboarding + +import ( + "context" + "fmt" + "net/netip" + "slices" + "strings" + + "github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/compute/armcompute/v7" + "github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/containerservice/armcontainerservice/v9" + "github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/network/armnetwork/v7" + "github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/storage/armstorage" +) + +func subnetPrefixes(subnet *armnetwork.Subnet) ([]netip.Prefix, error) { + if subnet == nil || subnet.Properties == nil { + return nil, fmt.Errorf("subnet properties are unavailable") + } + values := slices.Clone(subnet.Properties.AddressPrefixes) + if subnet.Properties.AddressPrefix != nil { + values = append(values, subnet.Properties.AddressPrefix) + } + var prefixes []netip.Prefix + for _, value := range values { + prefix, err := netip.ParsePrefix(resourceValue(value)) + if err != nil { + return nil, fmt.Errorf("invalid subnet prefix: %w", err) + } + if prefix != prefix.Masked() { + return nil, fmt.Errorf("subnet CIDR must use its network address") + } + prefixes = append(prefixes, prefix) + } + if len(prefixes) == 0 { + return nil, fmt.Errorf("subnet CIDR evidence is absent") + } + return prefixes, nil +} + +func delegatedSubnetCheck( + subnet *armnetwork.Subnet, parent *armnetwork.VirtualNetwork, related []netip.Prefix, +) Check { + c := resourceCheck("N02", resourceValue(subnet.ID), + "private non-overlapping delegated subnet, at least /27, contained in its VNet") + prefixes, err := subnetPrefixes(subnet) + if err != nil { + c.Observed, c.Cause = "subnet CIDR evidence could not be validated", err + return c + } + if parent == nil || parent.Properties == nil || parent.Properties.AddressSpace == nil || + len(parent.Properties.AddressSpace.AddressPrefixes) == 0 { + c.Missing = "parent VNet address space" + return c + } + var parents []netip.Prefix + for _, raw := range parent.Properties.AddressSpace.AddressPrefixes { + prefix, err := netip.ParsePrefix(resourceValue(raw)) + if err != nil { + c.Cause = err + c.Missing = "parseable parent address spaces" + return c + } + parents = append(parents, prefix) + } + for _, prefix := range prefixes { + // netip.Addr.IsPrivate includes RFC1918 and IPv6 ULA. Both ends must be private. + last := prefix.Addr().As16() + bits := prefix.Bits() + if prefix.Addr().Is4() { + bits += 96 + } + for bit := bits; bit < 128; bit++ { + last[bit/8] |= 1 << (7 - bit%8) + } + end := netip.AddrFrom16(last).Unmap() + valid := prefix.Addr().Is4() && prefix.Bits() <= 27 && prefix.Addr().IsPrivate() && end.IsPrivate() + contained := slices.ContainsFunc(parents, func(parent netip.Prefix) bool { + return parent.Bits() <= prefix.Bits() && parent.Contains(prefix.Addr()) && parent.Contains(end) + }) + if !valid || !contained { + return resourceResult(c, Fail, "subnet must be a private IPv4 CIDR, /27 or larger, within parent VNet") + } + if slices.ContainsFunc(related, prefix.Overlaps) { + return resourceResult(c, Fail, "delegated subnet overlaps the selected AKS pool Node subnet") + } + } + delegated := slices.ContainsFunc(subnet.Properties.Delegations, func(d *armnetwork.Delegation) bool { + return d != nil && d.Properties != nil && + strings.EqualFold(resourceValue(d.Properties.ServiceName), "Microsoft.App/environments") + }) + if !delegated { + return resourceResult(c, Fail, "subnet is not delegated to Microsoft.App/environments") + } + return resourceResult(c, Pass, "ARM CIDR containment/non-overlap and Microsoft.App/environments delegation verified") +} + +func (c *sdkResourceChecker) networkChecks( + ctx context.Context, plan Plan, cluster *armcontainerservice.ManagedCluster, + pools []*armcontainerservice.AgentPool, skus []*armcompute.ResourceSKU, storage *armstorage.Account, +) Report { + report := Report{} + checkSubnet := plan.AgentSubnet.ID != "" && plan.AgentSubnet.Supplied && + plan.AgentSubnet.Missing == "" && plan.AgentSubnet.LifecycleMissing == "" + var selected *armcontainerservice.AgentPool + if cluster != nil && checkSubnet && skus != nil { + // The SKU-aware choice is made from the same child resources as A06-A10. + // Nil SKUs mean enumeration failed; an empty successful enumeration is non-nil. + selected, _ = selectedAgentPool(*cluster, pools, skus) + } + var related []netip.Prefix + var nodeMetadataKnown bool + var nodeReport, parentReport, subnetReport Report + var parent *armnetwork.VirtualNetwork + var subnet *armnetwork.Subnet + if checkSubnet { + parallelReads(func() { + related, nodeMetadataKnown, nodeReport = c.readNodeSubnetPrefixes(ctx, plan, cluster, selected) + }, func() { + parent, parentReport = c.readParentNetwork(ctx, plan.AgentSubnet.ID) + }, func() { + subnet, subnetReport = c.readNetworkSubnet(ctx, plan.AgentSubnet.ID, "N02") + }) + } + report = append(report, nodeReport...) + if missing := plan.Storage.Missing + plan.Storage.LifecycleMissing; missing != "" { + report = append(report, SkippedCheck("S03", plan.Storage.diagnosticScope(), "Storage network access", missing)) + } else { + report = append(report, storageNetworkChecks(plan, storage)...) + } + if missing := plan.AgentSubnet.Missing + plan.AgentSubnet.LifecycleMissing; missing != "" { + return append(report, SkippedCheck("N02", plan.AgentSubnet.diagnosticScope(), + "existing Agent subnet configuration", missing)) + } + if !checkSubnet { + report = append(report, resourceResult(resourceCheck("N02", plan.AgentSubnet.ID, + "existing standard network.agentSubnet selection"), NotApplicable, + "no existing customer Agent subnet selected; standard network configuration is unchanged")) + return report + } + report = append(report, parentReport...) + report = append(report, subnetReport...) + if subnet == nil { + return report + } + subnetCheck := delegatedSubnetCheck(subnet, parent, related) + if plan.Cluster.Missing != "" { + if subnetCheck.Status == Pass { + subnetCheck.Expected = "private IPv4 CIDR, at least /27, contained in its VNet and delegated" + subnetCheck.Observed = "ARM CIDR containment and Microsoft.App/environments delegation verified" + } + return append(report, subnetCheck, SkippedCheck("N02", plan.AgentSubnet.ID, + "selected AKS Node subnet non-overlap", plan.Cluster.Missing)) + } + if !nodeMetadataKnown && subnetCheck.Status != Fail { + subnetCheck.Status = Unverified + subnetCheck.Missing = "selected AKS pool Node subnet ARM CIDR metadata" + } + report = append(report, subnetCheck) + return report +} + +func (c *sdkResourceChecker) readNodeSubnetPrefixes( + ctx context.Context, plan Plan, cluster *armcontainerservice.ManagedCluster, selected *armcontainerservice.AgentPool, +) ([]netip.Prefix, bool, Report) { + var related []netip.Prefix + var report Report + nodeMetadataKnown := false + if selected != nil && selected.Properties != nil { + subnetIDs, checks := c.nodeSubnetIDs(ctx, plan, cluster, selected) + report = append(report, checks...) + nodeMetadataKnown = len(subnetIDs) > 0 + for _, subnetID := range subnetIDs { + subnet, checks := c.readNetworkSubnet(ctx, subnetID, "N02") + report = append(report, checks...) + if subnet != nil { + state := "" + if subnet.Properties != nil { + state = resourceValue(subnet.Properties.ProvisioningState) + } + readiness := resourceReadiness(subnetID, resourceValue(subnet.ID), state, "subnets/read") + report = append(report, readiness) + prefixes, err := subnetPrefixes(subnet) + if err != nil { + metadata := resourceCheck("N02", subnetID, "parseable selected AKS pool Node subnet CIDRs") + metadata.Missing = "ARM subnet addressPrefix/addressPrefixes" + metadata.Cause = err + report = append(report, metadata) + } + if err == nil && readiness.Status == Pass { + related = append(related, prefixes...) + } else { + nodeMetadataKnown = false + } + } else { + nodeMetadataKnown = false + } + } + } + return related, nodeMetadataKnown, report +} diff --git a/cli/azd/extensions/azure.ai.projects/internal/onboarding/network_sdk.go b/cli/azd/extensions/azure.ai.projects/internal/onboarding/network_sdk.go new file mode 100644 index 00000000000..fa8174e4527 --- /dev/null +++ b/cli/azd/extensions/azure.ai.projects/internal/onboarding/network_sdk.go @@ -0,0 +1,57 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +package onboarding + +import ( + "context" + "strings" + + "github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/network/armnetwork/v7" +) + +func (c *sdkResourceChecker) readNetworkSubnet(ctx context.Context, raw, checkID string) (*armnetwork.Subnet, Report) { + id, err := existingResourceID(raw, "Microsoft.Network/virtualNetworks/subnets") + if err != nil { + check := resourceResult(resourceCheck(checkID, raw, "complete subnet resource ID"), Fail, "invalid subnet ID") + check.Cause = err + return nil, Report{check} + } + client, err := c.clients().subnets.get(strings.ToLower(id.SubscriptionID), + func() (*armnetwork.SubnetsClient, error) { + return armnetwork.NewSubnetsClient(id.SubscriptionID, c.credential, c.options) + }) + if err != nil { + return nil, Report{readFailure(checkID, raw, "Microsoft.Network/virtualNetworks/subnets/read", err)} + } + response, err := client.Get(ctx, id.ResourceGroupName, id.Parent.Name, id.Name, nil) + if err != nil { + return nil, Report{readFailure(checkID, raw, "Microsoft.Network/virtualNetworks/subnets/read", err)} + } + return &response.Subnet, nil +} + +func (c *sdkResourceChecker) readParentNetwork(ctx context.Context, raw string) (*armnetwork.VirtualNetwork, Report) { + id, err := existingResourceID(raw, "Microsoft.Network/virtualNetworks/subnets") + if err != nil { + return nil, nil + } + client, err := c.clients().networks.get(strings.ToLower(id.SubscriptionID), + func() (*armnetwork.VirtualNetworksClient, error) { + return armnetwork.NewVirtualNetworksClient(id.SubscriptionID, c.credential, c.options) + }) + if err != nil { + return nil, Report{readFailure("N02", id.Parent.String(), "Microsoft.Network/virtualNetworks/read", err)} + } + response, err := client.Get(ctx, id.ResourceGroupName, id.Parent.Name, nil) + if err != nil { + return nil, Report{readFailure("N02", id.Parent.String(), "Microsoft.Network/virtualNetworks/read", err)} + } + state := "" + if response.Properties != nil { + state = resourceValue(response.Properties.ProvisioningState) + } + check := resourceReadiness(id.Parent.String(), resourceValue(response.ID), state, "ARM network configuration") + check.ID = "N02" + return &response.VirtualNetwork, Report{check} +} diff --git a/cli/azd/extensions/azure.ai.projects/internal/onboarding/operator_authorization.go b/cli/azd/extensions/azure.ai.projects/internal/onboarding/operator_authorization.go new file mode 100644 index 00000000000..f15d827af57 --- /dev/null +++ b/cli/azd/extensions/azure.ai.projects/internal/onboarding/operator_authorization.go @@ -0,0 +1,238 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +package onboarding + +import ( + "context" + "fmt" + "regexp" + "slices" + "strings" + + "github.com/Azure/azure-sdk-for-go/sdk/azcore/arm" + "github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/authorization/armauthorization/v2" +) + +var operatorObjectIDPattern = regexp.MustCompile(`(?i)^[0-9a-f]{8}(-[0-9a-f]{4}){3}-[0-9a-f]{12}$`) + +func (e *AuthorizationEvidence) filteredAssignments( + ctx context.Context, scope, filter string, +) ([]*armauthorization.RoleAssignment, error) { + return e.assignments.get(strings.ToLower(scope)+"|"+filter, func() ([]*armauthorization.RoleAssignment, error) { + return e.readAssignments(ctx, scope, filter) + }) +} + +func (e *AuthorizationEvidence) readAssignments( + ctx context.Context, scope, filter string, +) ([]*armauthorization.RoleAssignment, error) { + id, err := arm.ParseResourceID(scope) + if err != nil || id.SubscriptionID == "" { + return nil, fmt.Errorf("authorization requires a complete subscription scope: %s", scope) + } + client, err := e.assignmentClients.get(strings.ToLower(id.SubscriptionID), + func() (*armauthorization.RoleAssignmentsClient, error) { + return armauthorization.NewRoleAssignmentsClient(id.SubscriptionID, e.credential, e.options) + }) + if err != nil { + return nil, err + } + pager := client.NewListForScopePager(scope, + &armauthorization.RoleAssignmentsClientListForScopeOptions{Filter: new(filter)}) + var assignments []*armauthorization.RoleAssignment + for pager.More() { + page, err := pager.NextPage(ctx) + if err != nil { + return nil, fmt.Errorf("read applicable role assignments at %s: %w", scope, err) + } + assignments = append(assignments, page.Value...) + } + return assignments, nil +} + +// assignmentApplies consumes only assignments returned by the target's ARM scope query. +// ARM establishes management-group ancestry; its resource IDs are not subscription path prefixes. +func assignmentApplies(assignment *armauthorization.RoleAssignment, scope string) bool { + p := assignment.Properties + if scopeContains(authString(p.Scope), scope) { + return true + } + parent := authString(p.Scope) + parts := strings.Split(parent, "/") + if len(parts) != 5 || parts[0] != "" || !strings.EqualFold(parts[1], "providers") || + !strings.EqualFold(parts[2], "Microsoft.Management") || !strings.EqualFold(parts[3], "managementGroups") || + parts[4] == "" || strings.ContainsAny(parts[4], "?#%\\@:\x00 \t\r\n") { + return false + } + id, err := arm.ParseResourceID(authString(assignment.ID)) + return err == nil && id.Name != "" && strings.EqualFold(id.ResourceType.String(), + "Microsoft.Authorization/roleAssignments") && strings.EqualFold(id.Parent.String(), parent) +} + +func (a *scopedAuthorizer) basicAction( + ctx context.Context, scope string, caller callerIdentity, action string, unconditionalOnly bool, +) (Status, string, error) { + reads := a.evidence() + assignments, err := reads.filteredAssignments(ctx, scope, "atScope()") + if err != nil { + return Unverified, "operator role assignments could not be read", err + } + provenAssignments := map[*armauthorization.RoleAssignment]bool{} + if caller.groupOverage { + if !operatorObjectIDPattern.MatchString(caller.objectID) { + return Unverified, "operator object ID is invalid for ARM group-assignment evidence", nil + } + proven, err := reads.filteredAssignments(ctx, scope, + "assignedTo('"+caller.objectID+"')") + if err != nil { + return Unverified, "ARM operator group-assignment evidence could not be read", err + } + for _, assignment := range proven { + provenAssignments[assignment] = true + } + assignments = append(slices.Clone(assignments), proven...) + } + operator := "operator" + if unconditionalOnly { + operator = "unconditional operator" + } + for _, assignment := range assignments { + if assignment == nil || assignment.Properties == nil { + return Unverified, "operator role assignment evidence is incomplete", nil + } + p := assignment.Properties + if unconditionalOnly && (authString(p.Condition) != "" || authString(p.ConditionVersion) != "" || + authString(p.DelegatedManagedIdentityResourceID) != "") { + continue + } + if authString(p.PrincipalID) == "" { + return Unverified, "operator role assignment evidence is incomplete", nil + } + if !caller.includes(*p.PrincipalID) && !provenAssignments[assignment] { + continue + } + if authString(p.Scope) == "" || authString(p.RoleDefinitionID) == "" { + return Unverified, "operator assignment scope or role definition is unavailable", nil + } + if !assignmentApplies(assignment, scope) { + continue + } + role, err := reads.roleDefinition(ctx, *p.RoleDefinitionID) + if err != nil { + return Unverified, "operator role definition could not be read", err + } + if role.Properties == nil || role.Properties.Permissions == nil { + return Unverified, "operator role definition permissions are unavailable", nil + } + if permissionAllows(role.Properties.Permissions, action) { + return Pass, operator + " Actions/NotActions allow " + action, nil + } + } + return Fail, "no applicable " + operator + " role permits " + action, nil +} + +func (a *scopedAuthorizer) checkP06( + ctx context.Context, plan Plan, facts Facts, caller callerIdentity, identityErr error, +) Report { + var report Report + grants := plan.Grants + if grants == nil { + for _, entry := range []struct { + id string + resource Resource + }{{"G01", plan.Cluster}, {"G02", plan.Workload}, {"G03", plan.Storage}} { + if createsGrant(plan, entry.id) { + grants = append(grants, PlannedGrant{CheckID: entry.id, Target: entry.resource}) + } + } + } + seen := map[string]bool{} + for _, grant := range grants { + resource := grant.Target + missing := grant.Missing + if grant.AuthorityScope == "" { + missing += resource.Missing + resource.LifecycleMissing + } + if missing != "" { + report = append(report, SkippedCheck("P06", resource.diagnosticScope(), + "role-assignment authority for "+grant.CheckID, missing)) + continue + } + scope := grant.AuthorityScope + if scope == "" { + scope = facts.scope(resource) + } + key := strings.ToLower(scope) + if scope == "" { + check := resourceCheck("P06", resource.ID, "unconditional operator "+roleAssignmentWrite+" for "+grant.CheckID) + check.Observed = "current resource existence is unverified; permission scope cannot be selected" + report = append(report, check) + continue + } + if seen[key] { + continue + } + seen[key] = true + check := resourceCheck("P06", scope, "unconditional operator "+roleAssignmentWrite+" for G01-G03") + check.Missing = "readable unconditional operator " + roleAssignmentWrite + check.Remediation = "Ask the scope administrator to provide a readable, unconditional role assignment " + + "permitting " + roleAssignmentWrite + " at this scope or an applicable ancestor." + if identityErr != nil || caller.objectID == "" { + check.Observed, check.Cause = "operator object ID is unavailable", identityErr + check.Missing = "operator object ID" + } else { + check.Status, check.Observed, check.Cause = a.basicAction(ctx, scope, caller, roleAssignmentWrite, true) + if check.Status == Pass { + check.Observed += "; ARM authorizes the actual assignment" + check.Missing, check.Remediation = "", "" + } + } + report = append(report, check) + } + return report +} + +func (a *scopedAuthorizer) checkP07( + ctx context.Context, plan Plan, facts Facts, caller callerIdentity, identityErr error, +) Check { + resource := plan.Registry + if plan.RegistryGrantMissing != "" { + return SkippedCheck("P07", resource.diagnosticScope(), "native pull grant authority", plan.RegistryGrantMissing) + } + if plan.RegistryGrantPlanned != nil && !*plan.RegistryGrantPlanned { + return resourceResult(resourceCheck("P07", resource.diagnosticScope(), "native pull grant authority"), + NotApplicable, "the actual template creates no native registry pull role assignment") + } + if missing := resource.Missing + resource.LifecycleMissing; missing != "" { + return SkippedCheck("P07", resource.diagnosticScope(), "native kubelet pull grant authority", missing) + } + if plan.Cluster.Missing != "" { + return SkippedCheck("P07", resource.diagnosticScope(), "native kubelet pull grant authority", + "actual AKS kubelet metadata cannot be selected: "+plan.Cluster.Missing) + } + scope := facts.scope(resource) + check := resourceCheck("P07", scope, "operator "+roleAssignmentWrite+" for the native kubelet pull grant") + check.Missing = "readable operator " + roleAssignmentWrite + check.Remediation = "Ask the scope administrator to provide readable operator role assignments " + + "permitting the native registry pull assignment at this scope or an applicable ancestor." + if scope == "" { + check.Scope, check.Observed = resource.ID, "current registry existence is unverified" + return check + } + if facts.KubeletObjectID == "" { + check.Missing = "AKS kubelet object ID" + return check + } + if identityErr != nil || caller.objectID == "" { + check.Observed, check.Cause = "operator object ID is unavailable", identityErr + check.Missing = "operator object ID" + return check + } + check.Status, check.Observed, check.Cause = a.basicAction(ctx, scope, caller, roleAssignmentWrite, false) + if check.Status == Pass { + check.Observed += "; ARM authorizes the actual assignment" + check.Missing, check.Remediation = "", "" + } + return check +} diff --git a/cli/azd/extensions/azure.ai.projects/internal/onboarding/parallel_reads.go b/cli/azd/extensions/azure.ai.projects/internal/onboarding/parallel_reads.go new file mode 100644 index 00000000000..1c3be4fc6f5 --- /dev/null +++ b/cli/azd/extensions/azure.ai.projects/internal/onboarding/parallel_reads.go @@ -0,0 +1,25 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +package onboarding + +import "sync" + +// parallelReads bounds independent reads without canceling sibling evidence on failure. +// Each task owns its output; callers assemble reports in their original order after the join. +func parallelReads(tasks ...func()) { + var wg sync.WaitGroup + work := make(chan func(), len(tasks)) + for _, task := range tasks { + work <- task + } + close(work) + for range min(4, len(tasks)) { + wg.Go(func() { + for task := range work { + task() + } + }) + } + wg.Wait() +} diff --git a/cli/azd/extensions/azure.ai.projects/internal/onboarding/pipeline.go b/cli/azd/extensions/azure.ai.projects/internal/onboarding/pipeline.go new file mode 100644 index 00000000000..d52353383ac --- /dev/null +++ b/cli/azd/extensions/azure.ai.projects/internal/onboarding/pipeline.go @@ -0,0 +1,37 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +package onboarding + +import ( + "context" + "errors" + "fmt" +) + +// Pipeline collects read-only AKS, selected-network, resource and permission evidence. +type Pipeline struct { + Resources ResourceChecker + Access Authorizer +} + +// Preflight collects resource and permission evidence before opening the write gate. +func (p Pipeline) Preflight(ctx context.Context, plan Plan) (Facts, Report, error) { + if p.Resources == nil || p.Access == nil { + return Facts{}, nil, fmt.Errorf("BYO-AKS preflight dependencies are unavailable") + } + facts, report, err := p.Resources.Check(ctx, plan) + if err != nil { + if _, blocked := errors.AsType[*BlockedError](err); !blocked { + return facts, report, err + } + } + access, err := p.Access.Check(ctx, plan, facts) + report = append(report, access...) + if err != nil { + if _, blocked := errors.AsType[*BlockedError](err); !blocked { + return facts, report, err + } + } + return facts, report, report.Err() +} diff --git a/cli/azd/extensions/azure.ai.projects/internal/onboarding/plan.go b/cli/azd/extensions/azure.ai.projects/internal/onboarding/plan.go new file mode 100644 index 00000000000..5a69730b7d6 --- /dev/null +++ b/cli/azd/extensions/azure.ai.projects/internal/onboarding/plan.go @@ -0,0 +1,109 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +package onboarding + +import ( + "fmt" + "strings" + + "github.com/Azure/azure-sdk-for-go/sdk/azcore/arm" +) + +// Input combines public selections with the unchanged template's evaluated default resource IDs. +type Input struct { + SubscriptionID string + ResourceGroup string + GroupExists bool + ClusterID string + StorageID string + HostingIdentityID string + WorkloadIdentityID string + DefaultStorageID string + DefaultHostingID string + DefaultWorkloadID string + DefaultRegistryID string +} + +// ParseID validates the exact ARM type and complete subscription/RG hierarchy. +func ParseID(value, resourceType string) (*arm.ResourceID, error) { + if !validResourceIDSyntax(value) { + return nil, fmt.Errorf( + "expected a complete %s resource ID without URL credentials or extra components", resourceType) + } + id, err := arm.ParseResourceID(value) + if err != nil { + return nil, fmt.Errorf("invalid %s resource ID: %w", resourceType, err) + } + segments := 9 + if strings.EqualFold(resourceType, "Microsoft.Network/virtualNetworks/subnets") { + segments = 11 + } + if id.SubscriptionID == "" || id.ResourceGroupName == "" || id.Name == "" || + !strings.EqualFold(id.ResourceType.String(), resourceType) || len(strings.Split(value, "/")) != segments { + return nil, fmt.Errorf("expected a complete %s resource ID", resourceType) + } + return id, nil +} + +func validResourceIDSyntax(value string) bool { + return strings.HasPrefix(strings.ToLower(value), "/subscriptions/") && + !strings.ContainsAny(value, "?#%\\@:\x00 \t\r\n") && !strings.Contains(value, "//") +} + +// BuildPlan independently selects each reference or default without Azure reads or writes. +func BuildPlan(in Input) (Plan, error) { + cluster, err := ParseID(in.ClusterID, "Microsoft.ContainerService/managedClusters") + if err != nil { + return Plan{}, fmt.Errorf("I01 clusterResourceId: %w", err) + } + parent := "/subscriptions/" + in.SubscriptionID + "/resourceGroups/" + in.ResourceGroup + if _, err := arm.ParseResourceID(parent); err != nil || in.SubscriptionID == "" || in.ResourceGroup == "" || + strings.ContainsAny(in.SubscriptionID+in.ResourceGroup, "/\\?#%@:\x00 \t\r\n") { + return Plan{}, fmt.Errorf("I03: target subscription and resource group are required") + } + ancestor := "/subscriptions/" + in.SubscriptionID + if in.GroupExists { + ancestor = parent + } + plan := Plan{ + SubscriptionID: in.SubscriptionID, ResourceGroup: in.ResourceGroup, + ResourceGroupSupplied: in.GroupExists, + Cluster: Resource{ID: in.ClusterID, Parent: cluster.Parent.String(), Supplied: true}, + } + for _, item := range []struct { + supplied, fallback, kind string + out *Resource + }{ + {in.StorageID, in.DefaultStorageID, "Microsoft.Storage/storageAccounts", &plan.Storage}, + {in.HostingIdentityID, in.DefaultHostingID, "Microsoft.ManagedIdentity/userAssignedIdentities", &plan.Hosting}, + {in.WorkloadIdentityID, in.DefaultWorkloadID, "Microsoft.ManagedIdentity/userAssignedIdentities", &plan.Workload}, + } { + value := item.supplied + if value == "" { + value = item.fallback + } + id, err := ParseID(value, item.kind) + if err != nil { + return Plan{}, fmt.Errorf("I02: %w", err) + } + *item.out = Resource{ID: value, Parent: id.Parent.String(), Supplied: item.supplied != ""} + if !item.out.Supplied { + if !strings.EqualFold(id.Parent.String(), parent) { + return Plan{}, fmt.Errorf("I03: default supporting resource differs from the deployment target") + } + item.out.Parent = ancestor + } + } + if in.DefaultRegistryID != "" { + id, err := ParseID(in.DefaultRegistryID, "Microsoft.ContainerRegistry/registries") + if err != nil { + return Plan{}, fmt.Errorf("P05: %w", err) + } + if !strings.EqualFold(id.Parent.String(), parent) { + return Plan{}, fmt.Errorf("P05: native registry differs from the deployment target") + } + plan.Registry = Resource{ID: in.DefaultRegistryID, Parent: ancestor} + } + return plan, nil +} diff --git a/cli/azd/extensions/azure.ai.projects/internal/onboarding/resource_clients.go b/cli/azd/extensions/azure.ai.projects/internal/onboarding/resource_clients.go new file mode 100644 index 00000000000..7c78d8b7b96 --- /dev/null +++ b/cli/azd/extensions/azure.ai.projects/internal/onboarding/resource_clients.go @@ -0,0 +1,22 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +package onboarding + +import ( + "github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/msi/armmsi" + "github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/network/armnetwork/v7" +) + +type resourceClients struct { + identities passReads[*armmsi.UserAssignedIdentitiesClient] + subnets passReads[*armnetwork.SubnetsClient] + networks passReads[*armnetwork.VirtualNetworksClient] +} + +func (c *sdkResourceChecker) clients() *resourceClients { + if c.readClients != nil { + return c.readClients + } + return &resourceClients{} +} diff --git a/cli/azd/extensions/azure.ai.projects/internal/onboarding/resource_rules.go b/cli/azd/extensions/azure.ai.projects/internal/onboarding/resource_rules.go new file mode 100644 index 00000000000..cbd61ba3609 --- /dev/null +++ b/cli/azd/extensions/azure.ai.projects/internal/onboarding/resource_rules.go @@ -0,0 +1,460 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +package onboarding + +import ( + "fmt" + "slices" + "strconv" + "strings" + + "github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/compute/armcompute/v7" + "github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/containerservice/armcontainerservice/v9" + "github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/storage/armstorage" +) + +func resourceCheck(id, scope, expected string) Check { + return Check{ + ID: id, Scope: scope, Status: Unverified, Mandatory: true, Expected: expected, + Observed: "required ARM evidence is unavailable", Missing: "readable, explicit resource properties", + Remediation: "provide the stated evidence or ask the resource administrator to correct the configuration", + } +} + +func resourceResult(c Check, status Status, observed string) Check { + c.Status, c.Observed = status, observed + if status == Pass || status == NotApplicable { + c.Missing, c.Remediation = "", "" + } + return c +} + +func resourceValue[T ~string](p *T) string { + if p == nil { + return "" + } + return string(*p) +} + +func resourceBool(p *bool) bool { + return p != nil && *p +} + +func resourceKnown(c Check, known, valid bool, observed string) Check { + if !known { + return resourceResult(c, Unverified, observed) + } + if !valid { + return resourceResult(c, Fail, observed) + } + return resourceResult(c, Pass, observed) +} + +func numericVersion(raw string, minimum []int) (bool, bool) { + parts := strings.Split(raw, ".") + if len(parts) < len(minimum) || len(parts) > 3 { + return false, false + } + values := make([]int, len(parts)) + for i, part := range parts { + if part == "" || strings.IndexFunc(part, func(r rune) bool { return r < '0' || r > '9' }) >= 0 { + return false, false + } + v, err := strconv.Atoi(part) + if err != nil { + return false, false + } + values[i] = v + } + for i, required := range minimum { + if values[i] != required { + return true, values[i] > required + } + } + return true, true +} + +func agentImage(raw string) (bool, bool) { + const family = "AKSAzureLinux-V3katagen2-" + if !strings.HasPrefix(raw, family) { + return false, false + } + return numericVersion(strings.TrimPrefix(raw, family), []int{202608, 14, 0}) +} + +func vmSizes(pool *armcontainerservice.ManagedClusterAgentPoolProfileProperties) []string { + var result []string + if pool.VMSize != nil { + result = append(result, *pool.VMSize) + } + if pool.VirtualMachinesProfile != nil && pool.VirtualMachinesProfile.Scale != nil { + for _, manual := range pool.VirtualMachinesProfile.Scale.Manual { + if manual == nil { + result = append(result, "") + } else { + result = append(result, resourceValue(manual.Size)) + } + } + } + for _, actual := range pool.VirtualMachineNodesStatus { + if actual == nil { + result = append(result, "") + } else { + result = append(result, resourceValue(actual.Size)) + } + } + slices.Sort(result) + return slices.Compact(result) +} + +func skuCapacity( + skus []*armcompute.ResourceSKU, region, size string, cpu, memory float64, +) (Status, string) { + for _, sku := range skus { + if sku == nil || !strings.EqualFold(resourceValue(sku.Name), size) || + resourceValue(sku.ResourceType) != "virtualMachines" { + continue + } + inRegion := false + for _, location := range sku.Locations { + inRegion = inRegion || strings.EqualFold(resourceValue(location), region) + } + for _, location := range sku.LocationInfo { + inRegion = inRegion || location != nil && strings.EqualFold(resourceValue(location.Location), region) + } + if !inRegion { + continue + } + capabilities := map[string]float64{} + for _, capability := range sku.Capabilities { + if capability == nil { + continue + } + value, err := strconv.ParseFloat(resourceValue(capability.Value), 64) + if err == nil { + capabilities[resourceValue(capability.Name)] = value + } + } + cores, hasCPU := capabilities["vCPUs"] + if available, ok := capabilities["vCPUsAvailable"]; ok { + cores, hasCPU = available, true + } + ram, hasMemory := capabilities["MemoryGB"] + if !hasCPU || !hasMemory { + return Unverified, "SKU lacks vCPUs/vCPUsAvailable or MemoryGB capability" + } + detail := fmt.Sprintf("%s in %s: %g available vCPU / %g GiB", size, region, cores, ram) + if !(cores >= cpu && ram >= memory) { + return Fail, detail + } + return Pass, detail + } + return Unverified, fmt.Sprintf("no exact virtualMachines SKU capability for %s in %s", size, region) +} + +func poolCapacity( + p *armcontainerservice.ManagedClusterAgentPoolProfileProperties, + skus []*armcompute.ResourceSKU, region string, cpu, memory float64, +) (Status, string) { + sizes := vmSizes(p) + if len(sizes) == 0 { + return Unverified, "pool has no explicit VM size evidence" + } + status, details := Pass, []string{} + for _, size := range sizes { + s, detail := skuCapacity(skus, region, size, cpu, memory) + details = append(details, detail) + if s == Fail || s == Unverified && status == Pass { + status = s + } + } + return status, strings.Join(details, "; ") +} + +func agentPoolChecks( + pool *armcontainerservice.AgentPool, skus []*armcompute.ResourceSKU, region string, +) Report { + p := pool.Properties + scope := resourceValue(pool.ID) + if scope == "" { + scope = resourceValue(pool.Name) + } + checks := Report{ + resourceCheck("A06", scope, "one User pool: Azure Linux, KataVmIsolation, count >= 1, >= 8 vCPU / 32 GiB"), + resourceCheck("A07", scope, "compatible AKSAzureLinux-V3katagen2 image >= 202608.14.0"), + resourceCheck("A08", scope, "agents.ai.azure.com/sandbox-node=true"), + resourceCheck("A09", scope, "AgentPool securityProfile.sshAccess=Disabled"), + resourceCheck("A10", scope, "cluster and required AgentPool child resources provisioningState=Succeeded"), + } + knownOS := p.OSSKU != nil + validOS := resourceValue(p.OSSKU) == "AzureLinux" || resourceValue(p.OSSKU) == "AzureLinux3" + capacity, detail := poolCapacity(p, skus, region, 8, 32) + known := knownOS && p.WorkloadRuntime != nil && p.Count != nil && capacity != Unverified + valid := validOS && resourceValue(p.WorkloadRuntime) == "KataVmIsolation" && p.Count != nil && *p.Count >= 1 + checks[0] = resourceKnown(checks[0], known, valid && capacity == Pass, detail) + // A known incompatible field is a failure even when a different property is absent. + if knownOS && !validOS || p.WorkloadRuntime != nil && resourceValue(p.WorkloadRuntime) != "KataVmIsolation" || + p.Count != nil && *p.Count < 1 || capacity == Fail { + checks[0] = resourceResult(checks[0], Fail, "incompatible agent pool configuration; "+detail) + } + imageKnown, imageValid := agentImage(resourceValue(p.NodeImageVersion)) + checks[1] = resourceKnown(checks[1], imageKnown, imageValid, resourceValue(p.NodeImageVersion)) + label, labelExists := p.NodeLabels["agents.ai.azure.com/sandbox-node"] + checks[2] = resourceKnown(checks[2], p.NodeLabels != nil, labelExists && resourceValue(label) == "true", + "persisted pool sandbox-node label="+resourceValue(label)) + if p.SecurityProfile != nil { + ssh := resourceValue(p.SecurityProfile.SSHAccess) + checks[3] = resourceKnown(checks[3], ssh != "", ssh == "Disabled", "sshAccess="+ssh) + } + state := resourceValue(p.ProvisioningState) + checks[4] = resourceKnown(checks[4], state != "", state == "Succeeded", "AgentPool provisioningState="+state) + return checks +} + +func selectedAgentPool( + cluster armcontainerservice.ManagedCluster, pools []*armcontainerservice.AgentPool, skus []*armcompute.ResourceSKU, +) (*armcontainerservice.AgentPool, Report) { + var best *armcontainerservice.AgentPool + var checks Report + bestScore := -1 + for _, pool := range pools { + if pool == nil || pool.Properties == nil || resourceValue(pool.Properties.Mode) != "User" { + continue + } + candidate := agentPoolChecks(pool, skus, resourceValue(cluster.Location)) + score := 0 + for _, check := range candidate { + if check.Status == Pass { + score += 2 + } else if check.Status == Unverified { + score++ + } + } + if cluster.Properties != nil && cluster.Properties.NetworkProfile != nil { + n := cluster.Properties.NetworkProfile + if resourceValue(n.NetworkPlugin) == "azure" && + (resourceValue(n.NetworkPluginMode) == "overlay" || resourceValue(pool.Properties.PodSubnetID) != "") { + score++ + } + } + if score > bestScore { + best, checks, bestScore = pool, candidate, score + } + } + return best, checks +} + +func aksResourceChecks( + cluster armcontainerservice.ManagedCluster, pools []*armcontainerservice.AgentPool, skus []*armcompute.ResourceSKU, +) (string, Report) { + scope := resourceValue(cluster.ID) + report := Report{} + add := func(id, expected string, known, valid bool, observed string) { + report = append(report, resourceKnown(resourceCheck(id, scope, expected), known, valid, observed)) + } + p := cluster.Properties + if p == nil { + for i := 1; i <= 11; i++ { + report = append(report, resourceCheck(fmt.Sprintf("A%02d", i), scope, "readable AKS properties")) + } + return "", report + } + // Base is the documented default SKU name. The billing tier is not the product type. + sku := "Base" + if cluster.SKU != nil && cluster.SKU.Name != nil { + sku = resourceValue(cluster.SKU.Name) + } + version := resourceValue(p.CurrentKubernetesVersion) + knownVersion, validVersion := numericVersion(version, []int{1, 31}) + add("A01", "AKS Base (not Automatic), running Kubernetes >= 1.31", + knownVersion, sku == "Base" && validVersion, "SKU="+sku+"; currentKubernetesVersion="+version) + if sku != "Base" { + report[len(report)-1] = resourceResult(report[len(report)-1], Fail, "unsupported AKS SKU="+sku) + } + n := p.NetworkProfile + selectedPool, selectedChecks := selectedAgentPool(cluster, pools, skus) + if n == nil { + add("A02", "Azure CNI Overlay or Pod Subnet", false, false, "networkProfile is absent") + } else { + plugin, mode := resourceValue(n.NetworkPlugin), resourceValue(n.NetworkPluginMode) + podSubnet := false + if selectedPool != nil { + podSubnet = resourceValue(selectedPool.Properties.PodSubnetID) != "" + } + add("A02", "Azure CNI Overlay or Pod Subnet", plugin != "", + plugin == "azure" && (mode == "overlay" || podSubnet), + "plugin="+plugin+"; mode="+mode+"; User pool podSubnetID evidence="+strconv.FormatBool(podSubnet)) + } + kubelet := "" + if identity := p.IdentityProfile["kubeletidentity"]; identity != nil { + kubelet = resourceValue(identity.ObjectID) + } + identityType := "" + if cluster.Identity != nil { + identityType = resourceValue(cluster.Identity.Type) + } + managed := identityType == "SystemAssigned" || identityType == "UserAssigned" || + identityType == "SystemAssigned, UserAssigned" + add("A03", "managed cluster identity and kubelet principal object ID", + identityType != "" && kubelet != "", managed && kubelet != "", + "identity.type="+identityType+"; kubelet object ID="+kubelet) + if identityType != "" && !managed || identityType == "" && p.ServicePrincipalProfile != nil { + report[len(report)-1] = resourceResult(report[len(report)-1], Fail, "cluster lacks a supported managed identity") + } + oidc, workload, issuer := false, false, "" + knownFederation := p.OidcIssuerProfile != nil && p.SecurityProfile != nil && + p.SecurityProfile.WorkloadIdentity != nil + if p.OidcIssuerProfile != nil { + oidc, issuer = resourceBool(p.OidcIssuerProfile.Enabled), resourceValue(p.OidcIssuerProfile.IssuerURL) + } + if p.SecurityProfile != nil && p.SecurityProfile.WorkloadIdentity != nil { + workload = resourceBool(p.SecurityProfile.WorkloadIdentity.Enabled) + } + add("A04", "OIDC enabled with issuer URL; Workload Identity enabled", + knownFederation, oidc && workload && strings.TrimSpace(issuer) != "", "OIDC and Workload Identity configuration") + systemStatus, systemDetails, hasSystem := Pass, []string{}, false + for _, pool := range pools { + if pool == nil || pool.Properties == nil { + continue + } + switch resourceValue(pool.Properties.Mode) { + case "System": + hasSystem = true + status, detail := poolCapacity(pool.Properties, skus, resourceValue(cluster.Location), 4, 16) + systemDetails = append(systemDetails, resourceValue(pool.Name)+": "+detail) + if status == Fail || status == Unverified && systemStatus == Pass { + systemStatus = status + } + } + } + system := resourceCheck("A05", scope, "System pools, every applicable VM size >= 4 vCPU / 16 GiB") + if !hasSystem { + systemStatus, systemDetails = Fail, []string{"no System child pool"} + } + report = append(report, resourceResult(system, systemStatus, strings.Join(systemDetails, "; "))) + if selectedPool == nil { + for _, id := range []string{"A06", "A07", "A08", "A09", "A10"} { + report = append(report, resourceResult(resourceCheck(id, scope, "one complete qualifying User pool"), + Fail, "no User child pool")) + } + } else { + report = append(report, selectedChecks...) + } + for i := range report { + if report[i].ID != "A10" { + continue + } + for _, pool := range pools { + if pool == nil || pool.Properties == nil || resourceValue(pool.Properties.Mode) != "System" { + continue + } + state := resourceValue(pool.Properties.ProvisioningState) + if state != "Succeeded" { + report[i] = resourceKnown(report[i], state != "", false, "System child pool state="+state) + } + } + state := resourceValue(p.ProvisioningState) + if state != "Succeeded" { + report[i] = resourceKnown(report[i], state != "", false, "cluster provisioningState="+state) + } + } + if n == nil { + add("A11", "supported network policy engine enabled", false, false, "networkProfile is absent") + } else { + policy, dataplane := resourceValue(n.NetworkPolicy), resourceValue(n.NetworkDataplane) + valid := policy == "azure" || policy == "calico" || dataplane == "cilium" && (policy == "" || policy == "cilium") + known := policy != "" || dataplane == "azure" || dataplane == "cilium" + add("A11", "Azure/Calico policy or supported Cilium dataplane", + known, valid, "networkPolicy="+policy+"; networkDataplane="+dataplane) + } + return kubelet, report +} + +func storageResourceChecks(account armstorage.Account) Report { + scope := resourceValue(account.ID) + typeCheck := resourceCheck("S01", scope, "StorageV2; Hot tier; HNS explicitly disabled") + security := resourceCheck("S02", scope, + "HTTPS only; TLS >= 1.2; Shared Key disabled; anonymous Blob explicitly disabled") + p := account.Properties + if p == nil { + return Report{typeCheck, security} + } + known := account.Kind != nil && p.AccessTier != nil && p.IsHnsEnabled != nil + valid := resourceValue(account.Kind) == "StorageV2" && resourceValue(p.AccessTier) == "Hot" && + p.IsHnsEnabled != nil && !*p.IsHnsEnabled + typeCheck = resourceKnown(typeCheck, known, valid, + "kind="+resourceValue(account.Kind)+"; tier="+resourceValue(p.AccessTier)+"; HNS evidence") + if account.Kind != nil && resourceValue(account.Kind) != "StorageV2" || + p.AccessTier != nil && resourceValue(p.AccessTier) != "Hot" || resourceBool(p.IsHnsEnabled) { + typeCheck = resourceResult(typeCheck, Fail, "incompatible Storage kind, tier or HNS configuration") + } + sku := "" + if account.SKU != nil { + sku = resourceValue(account.SKU.Name) + } + typeCheck.Observed += "; SKU=" + sku + switch strings.ToUpper(strings.TrimSpace(sku)) { + case "STANDARD_ZRS", "STANDARD_GZRS", "STANDARD_RAGZRS", "PREMIUM_ZRS": + case "STANDARD_LRS", "STANDARD_GRS", "STANDARD_RAGRS", "PREMIUM_LRS": + typeCheck.Warning = fmt.Sprintf( + "Supplied Storage account uses %s without zone redundancy. "+ + "Zone redundancy is recommended; this recommendation does not block provisioning or change the account.", + sku) + default: + typeCheck.Warning = "The supplied Storage account SKU does not establish zone redundancy. " + + "Zone redundancy is recommended; confirm the account's redundancy setting. " + + "This recommendation does not block provisioning or change the account." + } + // Accounts.GetProperties uses API 2024-01-01. Its documented defaults are: + // HTTPS=true (since 2019-04-01), minimum TLS=1.0, Shared Key=true. + // Anonymous Blob access requires explicit allowBlobPublicAccess=false evidence. + https := p.EnableHTTPSTrafficOnly == nil || *p.EnableHTTPSTrafficOnly + anonymousKnown := p.AllowBlobPublicAccess != nil + anonymous := resourceBool(p.AllowBlobPublicAccess) + anonymousObserved := "unknown" + if anonymousKnown { + anonymousObserved = strconv.FormatBool(anonymous) + } + shared := p.AllowSharedKeyAccess == nil || *p.AllowSharedKeyAccess + tls := resourceValue(p.MinimumTLSVersion) + tlsKnown := tls == "" || tls == "TLS1_0" || tls == "TLS1_1" || tls == "TLS1_2" || tls == "TLS1_3" + tlsValid := tls == "TLS1_2" || tls == "TLS1_3" + security = resourceKnown(security, tlsKnown && anonymousKnown, https && !anonymous && !shared && tlsValid, + fmt.Sprintf("HTTPS=%t; minimum TLS=%s; Shared Key=%t; anonymous Blob=%s", https, tls, shared, anonymousObserved)) + if !https || anonymous || shared || tlsKnown && !tlsValid { + security = resourceResult(security, Fail, security.Observed) + } + return Report{typeCheck, security} +} + +func storageNetworkChecks(plan Plan, storage *armstorage.Account) Report { + check := resourceCheck("S03", plan.Storage.ID, "supplied Storage matches the public static network baseline") + if !plan.Storage.Supplied { + return Report{resourceResult(check, NotApplicable, + "S03 applies only to supplied Storage; new Storage checks are skipped before and after deployment")} + } + if storage == nil || storage.Properties == nil || storage.Properties.PrimaryEndpoints == nil { + return Report{check} + } + p := storage.Properties + endpoint := safeResourceEndpoint(resourceValue(p.PrimaryEndpoints.Blob)) + access := resourceValue(p.PublicNetworkAccess) + action := "" + if p.NetworkRuleSet != nil { + action = resourceValue(p.NetworkRuleSet.DefaultAction) + } + check.Observed = "Blob endpoint=" + endpoint + "; publicNetworkAccess=" + access + "; defaultAction=" + action + if endpoint == "" || access != "Enabled" && access != "Disabled" || action != "Allow" && action != "Deny" { + check.Missing = "valid HTTPS Blob endpoint and explicit public/firewall access settings" + return Report{check} + } + if access == "Enabled" && action == "Allow" { + return Report{resourceResult(check, Pass, check.Observed+"; public static baseline, not actual connectivity")} + } + check = resourceResult(check, NotApplicable, check.Observed+"; restricted/private network path verification skipped") + check.Mandatory = false + check.Warning = "Restricted/private Storage network connectivity was not verified; " + + "S03 is skipped and does not block provisioning." + return Report{check} +} diff --git a/cli/azd/extensions/azure.ai.projects/internal/onboarding/resource_sdk.go b/cli/azd/extensions/azure.ai.projects/internal/onboarding/resource_sdk.go new file mode 100644 index 00000000000..9425e0dd4fc --- /dev/null +++ b/cli/azd/extensions/azure.ai.projects/internal/onboarding/resource_sdk.go @@ -0,0 +1,506 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +package onboarding + +import ( + "context" + "errors" + "fmt" + "maps" + "net/http" + "net/url" + "slices" + "strings" + + "github.com/Azure/azure-sdk-for-go/sdk/azcore" + "github.com/Azure/azure-sdk-for-go/sdk/azcore/arm" + "github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime" + "github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/compute/armcompute/v7" + "github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/containerservice/armcontainerservice/v9" + "github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/msi/armmsi" + "github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/resources/armresources" + "github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/storage/armstorage" +) + +type sdkResourceChecker struct { + credential azcore.TokenCredential + options *arm.ClientOptions + readClients *resourceClients +} + +// NewResourceChecker constructs read-only ARM adapters using an already resolved user-access-tenant credential. +func NewResourceChecker(credential azcore.TokenCredential, options *arm.ClientOptions) ResourceChecker { + return &sdkResourceChecker{credential: credential, options: onboardingARMOptions(options)} +} + +func onboardingARMOptions(options *arm.ClientOptions) *arm.ClientOptions { + var local arm.ClientOptions + if options != nil { + local = *options + } + local.DisableRPRegistration = true + return &local +} + +func existingResourceID(raw, kind string) (*arm.ResourceID, error) { + if !validResourceIDSyntax(raw) { + return nil, fmt.Errorf("expected a complete %s resource ID without URL credentials or extra components", kind) + } + id, err := arm.ParseResourceID(raw) + if err != nil { + return nil, fmt.Errorf("invalid resource ID: %w", err) + } + if raw != strings.TrimSpace(raw) || strings.ContainsAny(raw, "?#%\\") || + id.SubscriptionID == "" || id.ResourceGroupName == "" || id.Name == "" || + !strings.EqualFold(id.ResourceType.String(), kind) { + return nil, fmt.Errorf("expected a complete %s resource ID", kind) + } + parts := strings.Split(strings.TrimPrefix(raw, "/"), "/") + expected := 8 + if strings.EqualFold(kind, "Microsoft.Network/virtualNetworks/subnets") { + expected = 10 + } + if !strings.HasPrefix(raw, "/") || len(parts) != expected || slices.Contains(parts, "") { + return nil, fmt.Errorf("expected a complete %s resource ID without extra segments", kind) + } + return id, nil +} + +func readFailure(id, scope, action string, err error) Check { + c := resourceCheck(id, scope, action) + c.Cause = err + c.Missing = action + " at " + scope + c.Observed = "resource could not be read; inspect the original service error" + c.Remediation = "allow the stated read operation and retry; do not replace an unreadable resource" + if response, ok := errors.AsType[*azcore.ResponseError](err); ok { + if response.StatusCode == http.StatusNotFound { + c.Status, c.Observed = Fail, "referenced resource does not exist (HTTP 404)" + } else { + c.Observed = fmt.Sprintf("resource read failed (HTTP %d; code %s)", response.StatusCode, response.ErrorCode) + } + } + return c +} + +func responseNotFound(err error) bool { + response, ok := errors.AsType[*azcore.ResponseError](err) + return ok && response.StatusCode == http.StatusNotFound +} + +func resourceReadiness(scope, returnedID, state, action string) Check { + c := resourceCheck("I02", scope, "referenced resource ID matches and provisioningState=Succeeded") + if returnedID == "" || state == "" { + return c + } + return resourceKnown(c, true, strings.EqualFold(scope, returnedID) && state == "Succeeded", + "resource ID matches="+fmt.Sprint(strings.EqualFold(scope, returnedID))+"; provisioningState="+state+"; "+action) +} + +func (c *sdkResourceChecker) subscriptionTenants(ctx context.Context, plan Plan) (string, Report) { + report := Report{} + subscriptions := map[string]bool{} + for _, subscription := range plan.OwningSubscriptions { + subscriptions[strings.ToLower(subscription)] = true + } + if plan.SubscriptionID != "" && !strings.ContainsAny(plan.SubscriptionID, "/\\?#% ") { + subscriptions[strings.ToLower(plan.SubscriptionID)] = true + } else { + report = append(report, resourceResult(resourceCheck("I02", plan.SubscriptionID, + "valid target subscription ID"), Fail, "target subscription is absent or malformed")) + } + for _, item := range []struct { + resource Resource + kind string + }{ + {plan.Cluster, "Microsoft.ContainerService/managedClusters"}, + {plan.Storage, "Microsoft.Storage/storageAccounts"}, + {plan.Hosting, "Microsoft.ManagedIdentity/userAssignedIdentities"}, + {plan.Workload, "Microsoft.ManagedIdentity/userAssignedIdentities"}, + } { + if item.resource.Missing != "" { + if parent, err := arm.ParseResourceID(item.resource.Parent); err == nil && parent.SubscriptionID != "" { + subscriptions[strings.ToLower(parent.SubscriptionID)] = true + } else { + report = append(report, SkippedCheck("I02", item.resource.diagnosticScope(), + "resource subscription owning tenant", item.resource.Missing)) + } + continue + } + id, err := ParseID(item.resource.ID, item.kind) + if err != nil { + check := resourceResult(resourceCheck("I02", item.resource.ID, item.kind), Fail, + "malformed supplied resource ID") + check.Cause = err + report = append(report, check) + continue + } + subscriptions[strings.ToLower(id.SubscriptionID)] = true + } + if network := plan.LegacyNetwork; network != nil && network.Enabled { + if network.DNSResourceGroup != "" && network.DNSSubscription != "" { + subscriptions[strings.ToLower(network.DNSSubscription)] = true + } + } + client, err := arm.NewClient("azure.ai.projects.onboarding", "v1.0.0", c.credential, c.options) + if err != nil { + return "", append(report, readFailure("I02", plan.SubscriptionID, "subscriptions/read", err)) + } + keys := slices.Sorted(maps.Keys(subscriptions)) + checks := make(Report, len(keys)) + tenants := make([]string, len(keys)) + tasks := make([]func(), 0, len(keys)) + for i, subscription := range keys { + tasks = append(tasks, func() { + tenants[i], checks[i] = readSubscriptionTenant(ctx, client, subscription) + }) + } + parallelReads(tasks...) + var tenant string + for i, check := range checks { + if tenants[i] != "" { + if tenant == "" { + tenant = tenants[i] + } + check = resourceKnown(check, true, strings.EqualFold(tenant, tenants[i]), + "subscription owning tenantId="+tenants[i]) + } + report = append(report, check) + } + if report.Err() != nil { + tenant = "" + } + return tenant, report +} + +func readSubscriptionTenant(ctx context.Context, client *arm.Client, subscription string) (string, Check) { + scope := "/subscriptions/" + subscription + request, err := runtime.NewRequest(ctx, http.MethodGet, + strings.TrimRight(client.Endpoint(), "/")+scope+"?api-version=2022-12-01") + if err != nil { + return "", readFailure("I02", scope, "Microsoft.Resources/subscriptions/read", err) + } + response, err := client.Pipeline().Do(request) + if err == nil && !runtime.HasStatusCode(response, http.StatusOK) { + err = runtime.NewResponseError(response) + } + if err != nil { + return "", readFailure("I02", scope, "Microsoft.Resources/subscriptions/read", err) + } + var result struct { + TenantID string `json:"tenantId"` + } + err = runtime.UnmarshalAsJSON(response, &result) + if err != nil { + return "", readFailure("I02", scope, "top-level subscription tenantId", err) + } + check := resourceCheck("I02", scope, "all referenced and target subscriptions have the same top-level tenantId") + if result.TenantID == "" { + check.Observed = "subscription response has no top-level tenantId; managedByTenants is not ownership" + } + return result.TenantID, check +} + +func (c *sdkResourceChecker) Check(ctx context.Context, plan Plan) (Facts, Report, error) { + local := *c + local.readClients = &resourceClients{} + c = &local + tenant, report := c.subscriptionTenants(ctx, plan) + report = append(slices.Clone(plan.SkippedChecks), report...) + facts := Facts{TenantID: tenant, Exists: map[string]bool{}} + var identities [2]Identity + var existence [3]*bool + var identityReports [2]Check + var storage *armstorage.Account + var storageReport Report + var cluster *armcontainerservice.ManagedCluster + var pools []*armcontainerservice.AgentPool + var skus []*armcompute.ResourceSKU + var clusterReport Report + var registryExists *bool + var registryReport Report + tasks := []func(){func() { + storage, existence[2], storageReport = c.readStorage(ctx, plan.Storage) + }, func() { + cluster, pools, skus, clusterReport = c.readCluster(ctx, plan.Cluster) + }} + if plan.Registry.ID != "" || plan.Registry.Missing != "" { + tasks = append(tasks, func() { + registryExists, registryReport = c.readRegistry(ctx, plan.Registry) + }) + } + for i, resource := range []Resource{plan.Hosting, plan.Workload} { + tasks = append(tasks, func() { + identities[i], existence[i], identityReports[i] = c.readIdentity(ctx, resource, tenant) + }) + } + parallelReads(tasks...) + facts.HostingIdentity, facts.WorkloadIdentity = identities[0], identities[1] + for i, resource := range []Resource{plan.Hosting, plan.Workload, plan.Storage} { + if existence[i] != nil { + facts.Exists[strings.ToLower(resource.ID)] = *existence[i] + } + if registryExists != nil { + facts.Exists[strings.ToLower(plan.Registry.ID)] = *registryExists + } + } + report = append(report, identityReports[:]...) + report = append(report, storageReport...) + report = append(report, clusterReport...) + report = append(report, registryReport...) + if cluster != nil { + kubelet, checks := aksResourceChecks(*cluster, pools, skus) + facts.KubeletObjectID = kubelet + report = append(report, checks...) + } else { + for i := 1; i <= 11; i++ { + id := fmt.Sprintf("A%02d", i) + if plan.Cluster.Missing != "" { + report = append(report, SkippedCheck(id, plan.Cluster.diagnosticScope(), + "AKS prerequisite configuration", plan.Cluster.Missing)) + } else { + report = append(report, resourceCheck(id, plan.Cluster.ID, "readable AKS metadata")) + } + } + } + if missing := plan.Storage.Missing + plan.Storage.LifecycleMissing; missing != "" { + for _, id := range []string{"S01", "S02"} { + report = append(report, SkippedCheck(id, plan.Storage.diagnosticScope(), "Storage configuration", missing)) + } + } else if plan.Storage.Supplied { + if storage != nil { + report = append(report, storageResourceChecks(*storage)...) + } else { + for _, id := range []string{"S01", "S02"} { + report = append(report, resourceCheck(id, plan.Storage.ID, "readable supplied Storage account")) + } + } + } else { + for _, id := range []string{"S01", "S02"} { + report = append(report, resourceResult(resourceCheck(id, plan.Storage.Parent, "supplied Storage"), + NotApplicable, "Storage uses the default ARM creation/update path")) + } + } + report = append(report, c.networkChecks(ctx, plan, cluster, pools, skus, storage)...) + if err := ctx.Err(); err != nil { + return facts, report, err + } + return facts, report, report.Err() +} + +func (c *sdkResourceChecker) readCluster( + ctx context.Context, resource Resource, +) (*armcontainerservice.ManagedCluster, []*armcontainerservice.AgentPool, []*armcompute.ResourceSKU, Report) { + if resource.Missing != "" { + return nil, nil, nil, Report{SkippedCheck("I02", resource.diagnosticScope(), + "AKS resource existence and identity", resource.Missing)} + } + id, err := existingResourceID(resource.ID, "Microsoft.ContainerService/managedClusters") + if err != nil { + return nil, nil, nil, Report{readFailure("I02", resource.ID, "complete managedClusters resource ID", err)} + } + client, err := armcontainerservice.NewManagedClustersClient(id.SubscriptionID, c.credential, c.options) + if err != nil { + return nil, nil, nil, Report{readFailure("I02", resource.ID, "managedClusters/read", err)} + } + response, err := client.Get(ctx, id.ResourceGroupName, id.Name, nil) + if err != nil { + return nil, nil, nil, Report{readFailure("I02", resource.ID, + "Microsoft.ContainerService/managedClusters/read", err)} + } + state := "" + if response.Properties != nil { + state = resourceValue(response.Properties.ProvisioningState) + } + report := Report{resourceReadiness(resource.ID, resourceValue(response.ID), state, "managedClusters/read")} + var pools []*armcontainerservice.AgentPool + var skus []*armcompute.ResourceSKU + var poolErr, skuErr error + parallelReads(func() { + pools, poolErr = c.readAgentPools(ctx, id) + }, func() { + skus, skuErr = c.readSKUs(ctx, id.SubscriptionID, resourceValue(response.Location)) + }) + if poolErr != nil { + report = append(report, readFailure("A10", resource.ID, + "Microsoft.ContainerService/managedClusters/agentPools/read", poolErr)) + } + if skuErr != nil { + report = append(report, readFailure("A05", "/subscriptions/"+id.SubscriptionID, + "Microsoft.Compute/skus/read", skuErr)) + } + return &response.ManagedCluster, pools, skus, report +} + +func (c *sdkResourceChecker) readAgentPools( + ctx context.Context, id *arm.ResourceID, +) ([]*armcontainerservice.AgentPool, error) { + client, err := armcontainerservice.NewAgentPoolsClient(id.SubscriptionID, c.credential, c.options) + if err != nil { + return nil, err + } + var pools []*armcontainerservice.AgentPool + pager := client.NewListPager(id.ResourceGroupName, id.Name, nil) + for pager.More() { + page, pageErr := pager.NextPage(ctx) + if pageErr != nil { + // Partial enumeration cannot establish the required pool configuration. + return nil, pageErr + } + pools = append(pools, page.Value...) + } + return pools, nil +} + +func (c *sdkResourceChecker) readSKUs( + ctx context.Context, subscription, location string, +) ([]*armcompute.ResourceSKU, error) { + client, err := armcompute.NewResourceSKUsClient(subscription, c.credential, c.options) + if err != nil { + return nil, err + } + filter := "location eq '" + strings.ReplaceAll(location, "'", "''") + "'" + pager := client.NewListPager(&armcompute.ResourceSKUsClientListOptions{Filter: &filter}) + result := make([]*armcompute.ResourceSKU, 0) + for pager.More() { + page, err := pager.NextPage(ctx) + if err != nil { + return nil, err + } + result = append(result, page.Value...) + } + return result, nil +} + +func (c *sdkResourceChecker) readStorage( + ctx context.Context, resource Resource, +) (*armstorage.Account, *bool, Report) { + if resource.Missing != "" { + return nil, nil, Report{SkippedCheck("I02", resource.diagnosticScope(), + "Storage resource existence and identity", resource.Missing)} + } + id, err := ParseID(resource.ID, "Microsoft.Storage/storageAccounts") + if err != nil { + return nil, nil, Report{readFailure("I02", resource.ID, "valid Storage resource ID", err)} + } + client, err := armstorage.NewAccountsClient(id.SubscriptionID, c.credential, c.options) + if err != nil { + return nil, nil, Report{readFailure("I02", resource.ID, "Microsoft.Storage/storageAccounts/read", err)} + } + response, err := client.GetProperties(ctx, id.ResourceGroupName, id.Name, nil) + if responseNotFound(err) && !resource.Supplied { + if resource.LifecycleMissing != "" { + return nil, new(false), Report{SkippedCheck("I02", resource.ID, + "Storage creation/reference existence", resource.LifecycleMissing)} + } + return nil, new(false), Report{resourceResult(resourceCheck("I03", resource.ID, "current default Storage existence"), + Pass, "default Storage does not exist; ARM will create it")} + } + if err != nil { + return nil, nil, Report{readFailure("I02", resource.ID, "Microsoft.Storage/storageAccounts/read", err)} + } + if !resource.Supplied { + check := resourceKnown(resourceCheck("I03", resource.ID, "exact default Storage target"), + response.ID != nil, strings.EqualFold(resourceValue(response.ID), resource.ID), + "default Storage exists; its ordinary ARM creation/update path is retained") + return &response.Account, new(true), Report{check} + } + state := "" + if response.Properties != nil { + state = resourceValue(response.Properties.ProvisioningState) + } + return &response.Account, new(true), Report{ + resourceReadiness(resource.ID, resourceValue(response.ID), state, "storageAccounts/read"), + } +} + +func (c *sdkResourceChecker) readIdentity( + ctx context.Context, resource Resource, tenant string, +) (Identity, *bool, Check) { + if resource.Missing != "" { + return Identity{}, nil, SkippedCheck("I02", resource.diagnosticScope(), + "managed identity existence and metadata", resource.Missing) + } + check := resourceCheck("I02", resource.ID, + "existing user-assigned identity with matching ID, principal, client and tenant") + id, err := ParseID(resource.ID, "Microsoft.ManagedIdentity/userAssignedIdentities") + if err != nil { + check.Status, check.Observed, check.Cause = Fail, "invalid identity resource ID", err + return Identity{}, nil, check + } + client, err := c.clients().identities.get(strings.ToLower(id.SubscriptionID), + func() (*armmsi.UserAssignedIdentitiesClient, error) { + return armmsi.NewUserAssignedIdentitiesClient(id.SubscriptionID, c.credential, c.options) + }) + if err != nil { + check.Cause = err + return Identity{}, nil, check + } + response, err := client.Get(ctx, id.ResourceGroupName, id.Name, nil) + if responseNotFound(err) && !resource.Supplied { + if resource.LifecycleMissing != "" { + return Identity{}, new(false), SkippedCheck("I02", resource.ID, + "managed identity creation/reference existence", resource.LifecycleMissing) + } + return Identity{}, new(false), resourceResult(check, Pass, "default identity does not exist; ARM will create it") + } + + if err != nil { + return Identity{}, nil, readFailure("I02", resource.ID, "Microsoft.ManagedIdentity/userAssignedIdentities/read", err) + } + if !resource.Supplied { + return Identity{}, new(true), resourceKnown(check, response.ID != nil, + strings.EqualFold(authString(response.ID), resource.ID), + "default identity exists; its ordinary ARM creation/update path is retained") + } + properties := response.Properties + if response.ID == nil || properties == nil || authString(properties.PrincipalID) == "" || + authString(properties.ClientID) == "" || authString(properties.TenantID) == "" || tenant == "" { + check.Observed = "supplied identity ID, principal, client or owning tenant evidence is missing" + return Identity{}, new(true), check + } + identity := Identity{ + ID: *response.ID, PrincipalID: *properties.PrincipalID, + ClientID: *properties.ClientID, TenantID: *properties.TenantID, + } + if !strings.EqualFold(resource.ID, identity.ID) || !strings.EqualFold(tenant, identity.TenantID) { + return Identity{}, new(true), resourceResult(check, Fail, "supplied identity ID or owning tenant does not match") + } + return identity, new(true), resourceResult(check, Pass, + "existing identity metadata matches; no identity PUT or owner tag is planned") +} + +func (c *sdkResourceChecker) readRegistry(ctx context.Context, resource Resource) (*bool, Report) { + if resource.Missing != "" { + return nil, Report{SkippedCheck("I02", resource.diagnosticScope(), + "native registry existence", resource.Missing)} + } + id, err := ParseID(resource.ID, "Microsoft.ContainerRegistry/registries") + if err != nil { + return nil, Report{readFailure("I02", resource.ID, "valid native registry ID", err)} + } + client, err := armresources.NewClient(id.SubscriptionID, c.credential, c.options) + if err != nil { + return nil, Report{readFailure("I02", resource.ID, "Microsoft.ContainerRegistry/registries/read", err)} + } + response, err := client.GetByID(ctx, resource.ID, "2023-07-01", nil) + if responseNotFound(err) && !resource.Supplied { + return new(false), nil + } + if err != nil { + return nil, Report{readFailure("I02", resource.ID, "Microsoft.ContainerRegistry/registries/read", err)} + } + check := resourceKnown(resourceCheck("I02", resource.ID, "exact native registry target"), + response.ID != nil, strings.EqualFold(resourceValue(response.ID), resource.ID), + "native registry exists; the ordinary ARM creation/update path is retained") + return new(true), Report{check} +} + +func safeResourceEndpoint(raw string) string { + u, err := url.Parse(raw) + if err != nil || u.Scheme != "https" || u.Host == "" { + return "" + } + u.User, u.RawQuery, u.Fragment = nil, "", "" + return u.String() +} diff --git a/cli/azd/extensions/azure.ai.projects/internal/onboarding/resource_test.go b/cli/azd/extensions/azure.ai.projects/internal/onboarding/resource_test.go new file mode 100644 index 00000000000..2153c53e95e --- /dev/null +++ b/cli/azd/extensions/azure.ai.projects/internal/onboarding/resource_test.go @@ -0,0 +1,540 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +package onboarding + +import ( + "context" + "encoding/base64" + "encoding/json" + "errors" + "fmt" + "io" + "maps" + "net/http" + "strings" + "sync" + "testing" + "time" + + "github.com/Azure/azure-sdk-for-go/sdk/azcore" + "github.com/Azure/azure-sdk-for-go/sdk/azcore/arm" + "github.com/Azure/azure-sdk-for-go/sdk/azcore/policy" + "github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/storage/armstorage" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +const testSubscription = "/subscriptions/target-sub" +const testGroup = testSubscription + "/resourceGroups/target-rg" +const testCluster = "/subscriptions/cluster-sub/resourceGroups/cluster-rg/providers/" + + "Microsoft.ContainerService/managedClusters/cluster" +const testStorage = testGroup + "/providers/Microsoft.Storage/storageAccounts/customer" +const testHosting = testGroup + "/providers/Microsoft.ManagedIdentity/userAssignedIdentities/hosting" +const testWorkload = testGroup + "/providers/Microsoft.ManagedIdentity/userAssignedIdentities/workload" +const testOperator = "11111111-1111-4111-8111-111111111111" + +type testCredential struct { + err error +} + +func (c testCredential) GetToken(context.Context, policy.TokenRequestOptions) (azcore.AccessToken, error) { + raw := base64.RawURLEncoding.EncodeToString([]byte(`{"oid":"` + testOperator + `"}`)) + return azcore.AccessToken{Token: "header." + raw + ".signature", ExpiresOn: time.Now().Add(time.Hour)}, c.err +} + +type readReply struct { + status int + body any +} + +type readRequest struct { + method, path, filter string +} + +type readFake struct { + mu sync.Mutex + replies map[string]readReply + requests []readRequest + handler func(*http.Request) (readReply, bool) +} + +func (f *readFake) Do(request *http.Request) (*http.Response, error) { + f.mu.Lock() + f.requests = append(f.requests, readRequest{ + request.Method, request.URL.Path, request.URL.Query().Get("$filter"), + }) + f.mu.Unlock() + if request.Method != http.MethodGet { + return nil, fmt.Errorf("preflight attempted a mutation: %s %s", request.Method, request.URL.Path) + } + reply, ok := f.replies[strings.ToLower(request.URL.Path)] + if f.handler != nil { + if handled, found := f.handler(request); found { + reply, ok = handled, true + } + } + if !ok { + reply = readReply{404, map[string]any{"error": map[string]string{ + "code": "ResourceNotFound", "message": "offline resource does not exist", + }}} + } + raw, err := json.Marshal(reply.body) + if err != nil { + return nil, err + } + return &http.Response{ + StatusCode: reply.status, Request: request, Header: http.Header{"Content-Type": {"application/json"}}, + Body: io.NopCloser(strings.NewReader(string(raw))), + }, nil +} + +func (f *readFake) options() *arm.ClientOptions { + return &arm.ClientOptions{ClientOptions: policy.ClientOptions{ + Transport: f, Retry: policy.RetryOptions{MaxRetries: -1}, + }} +} + +func (f *readFake) count(path string) int { + f.mu.Lock() + defer f.mu.Unlock() + count := 0 + for _, request := range f.requests { + if strings.EqualFold(request.path, path) { + count++ + } + } + return count +} + +func suppliedPlan(t *testing.T) Plan { + t.Helper() + plan, err := BuildPlan(Input{ + SubscriptionID: "target-sub", ResourceGroup: "target-rg", GroupExists: true, + ClusterID: testCluster, StorageID: testStorage, HostingIdentityID: testHosting, WorkloadIdentityID: testWorkload, + }) + require.NoError(t, err) + return plan +} + +func storageBody() map[string]any { + return map[string]any{ + "id": testStorage, "kind": "StorageV2", "sku": map[string]any{"name": "Standard_ZRS"}, + "properties": map[string]any{ + "provisioningState": "Succeeded", "accessTier": "Hot", "isHnsEnabled": false, + "supportsHttpsTrafficOnly": true, "minimumTlsVersion": "TLS1_2", + "allowSharedKeyAccess": false, "allowBlobPublicAccess": false, + "publicNetworkAccess": "Enabled", "networkAcls": map[string]any{"defaultAction": "Allow"}, + "primaryEndpoints": map[string]any{"blob": "https://customer.blob.core.windows.net/"}, + }, + } +} + +func resourceFake() *readFake { + fake := &readFake{replies: map[string]readReply{}} + for _, subscription := range []string{"target-sub", "cluster-sub"} { + fake.replies["/subscriptions/"+subscription] = readReply{200, map[string]any{"tenantId": "tenant"}} + } + fake.replies[strings.ToLower(testStorage)] = readReply{200, storageBody()} + for _, item := range []struct{ id, principal string }{ + {testHosting, "hosting-principal"}, {testWorkload, "workload-principal"}, + } { + fake.replies[strings.ToLower(item.id)] = readReply{200, map[string]any{ + "id": item.id, "properties": map[string]any{ + "principalId": item.principal, "clientId": item.principal + "-client", "tenantId": "tenant", + }, + }} + } + fake.replies[strings.ToLower(testCluster)] = readReply{200, map[string]any{ + "id": testCluster, "location": "westus", "sku": map[string]any{"name": "Base"}, + "identity": map[string]any{"type": "SystemAssigned"}, + "properties": map[string]any{ + "currentKubernetesVersion": "1.31.0", "provisioningState": "Succeeded", + "networkProfile": map[string]any{ + "networkPlugin": "azure", "networkPluginMode": "overlay", "networkDataplane": "cilium", + }, + "identityProfile": map[string]any{"kubeletidentity": map[string]any{"objectId": testOperator}}, + "oidcIssuerProfile": map[string]any{"enabled": true, "issuerURL": "https://issuer.example.test/"}, + "securityProfile": map[string]any{"workloadIdentity": map[string]any{"enabled": true}}, + }, + }} + fake.replies[strings.ToLower(testCluster+"/agentPools")] = readReply{200, map[string]any{"value": []any{ + map[string]any{"id": testCluster + "/agentPools/system", "name": "system", "properties": map[string]any{ + "mode": "System", "vmSize": "Standard_E4s_v5", "count": 1, "provisioningState": "Succeeded", + }}, + map[string]any{"id": testCluster + "/agentPools/agents", "name": "agents", "properties": map[string]any{ + "mode": "User", "osSKU": "AzureLinux", "vmSize": "Standard_D8as_v5", "count": 1, + "workloadRuntime": "KataVmIsolation", "nodeImageVersion": "AKSAzureLinux-V3katagen2-202608.14.0", + "nodeLabels": map[string]any{"agents.ai.azure.com/sandbox-node": "true"}, + "securityProfile": map[string]any{"sshAccess": "Disabled"}, "provisioningState": "Succeeded", + }}, + }}} + var skus []any + for _, size := range []struct { + name, cpu string + }{{"Standard_E4s_v5", "4"}, {"Standard_D8as_v5", "8"}} { + skus = append(skus, map[string]any{ + "name": size.name, "resourceType": "virtualMachines", "locations": []string{"westus"}, + "capabilities": []any{map[string]any{"name": "vCPUs", "value": size.cpu}, + map[string]any{"name": "MemoryGB", "value": "32"}}, + }) + } + fake.replies["/subscriptions/cluster-sub/providers/microsoft.compute/skus"] = + readReply{200, map[string]any{"value": skus}} + return fake +} + +func checkWithID(t *testing.T, report Report, id string) Check { + t.Helper() + for _, check := range report { + if check.ID == id { + return check + } + } + t.Fatalf("missing check %s in %#v", id, report) + return Check{} +} + +func TestBuildPlanIndependentSelectionsAndExistingParents(t *testing.T) { + for mask := range 8 { + for _, groupExists := range []bool{false, true} { + t.Run(fmt.Sprintf("%d/group=%t", mask, groupExists), func(t *testing.T) { + input := Input{ + SubscriptionID: "target-sub", ResourceGroup: "target-rg", GroupExists: groupExists, + ClusterID: testCluster, DefaultStorageID: testStorage, DefaultHostingID: testHosting, + DefaultWorkloadID: testWorkload, + } + for i, field := range []*string{&input.StorageID, &input.HostingIdentityID, &input.WorkloadIdentityID} { + if mask&(1< 0 { + return &BlockedError{Checks: blocked} + } + return nil +} + +// Write renders evidence to the caller's writer. +func (r Report) Write(w io.Writer) error { + for _, c := range r { + if _, err := fmt.Fprintf(w, "%s %s [%s]\n Expected: %s\n Observed: %s\n"+ + " Missing evidence/permission: %s\n Remediation: %s\n", + c.ID, c.Status, RedactDiagnostic(c.Scope), RedactDiagnostic(c.Expected), RedactDiagnostic(c.Observed), + RedactDiagnostic(c.Missing), RedactDiagnostic(c.Remediation)); err != nil { + return err + } + } + for _, c := range r { + if c.Warning != "" { + if _, err := fmt.Fprintf(w, "WARNING: %s [%s]: %s\n", + c.ID, RedactDiagnostic(c.Scope), RedactDiagnostic(c.Warning)); err != nil { + return err + } + } + } + return nil +} + +// BlockedError identifies the exact checks that prevent Azure writes. +type BlockedError struct { + Checks []Check +} + +func (e *BlockedError) Error() string { + var message strings.Builder + message.WriteString("BYO-AKS preflight blocked") + for _, c := range e.Checks { + message.WriteString(fmt.Sprintf("; %s %s: %s (%s)", + c.ID, c.Status, RedactDiagnostic(c.Observed), RedactDiagnostic(c.Remediation))) + } + return message.String() +} + +// Unwrap preserves independent service error chains. +func (e *BlockedError) Unwrap() []error { + var causes []error + for _, c := range e.Checks { + if c.Cause != nil { + causes = append(causes, c.Cause) + } + } + return causes +} + +var diagnosticURLPattern = regexp.MustCompile(`(?i)https?://[^\s<>"]+`) + +// RedactDiagnostic removes URL credentials without discarding service error codes or safe endpoint paths. +func RedactDiagnostic(message string) string { + return diagnosticURLPattern.ReplaceAllStringFunc(message, func(raw string) string { + endpoint, err := url.Parse(raw) + if err != nil { + return "[redacted URL]" + } + endpoint.User, endpoint.RawQuery, endpoint.Fragment = nil, "", "" + return endpoint.String() + }) +} + +// Resource distinguishes an explicit reference from an ordinary ARM deployment target. +type Resource struct { + ID string + Parent string + Supplied bool + Origin string + Missing string + LifecycleMissing string +} + +// SkippedCheck reports an unresolved template-plan dependency, not successful evidence. +func SkippedCheck(id, scope, requirement, reason string) Check { + return Check{ + ID: id, Scope: scope, Status: Skipped, Expected: requirement, + Observed: "check skipped: " + reason, Missing: reason, + Warning: "skipped " + requirement + ": " + reason, + } +} + +func (r Resource) diagnosticScope() string { + if r.ID != "" { + return r.ID + } + return r.Origin +} + +// PlannedGrant identifies an actual role-assignment creation in the compiled template. +type PlannedGrant struct { + CheckID string + Target Resource + AuthorityScope string + DeploymentScope string + Missing string + DeploymentMissing string +} + +// Plan contains this deployment's resolved resource and standard-network selections. +type Plan struct { + SubscriptionID string + ResourceGroup string + ResourceGroupSupplied bool + TargetScopeMissing string + LegacyNetwork *LegacyNetwork + AgentSubnet Resource + Operations []operation + SkippedChecks Report + OwningSubscriptions []string + Grants []PlannedGrant + Cluster Resource + Storage Resource + Hosting Resource + Workload Resource + Registry Resource + RegistryGrantPlanned *bool + RegistryGrantMissing string +} + +// Facts contains current read-only evidence; an absent Exists key means existence is unverified. +type Facts struct { + KubeletObjectID string + TenantID string + HostingIdentity Identity + WorkloadIdentity Identity + Exists map[string]bool +} + +func (f Facts) exists(resource Resource) bool { + return f.Exists[strings.ToLower(resource.ID)] +} + +func (f Facts) absent(resource Resource) bool { + exists, known := f.Exists[strings.ToLower(resource.ID)] + return known && !exists +} + +func (f Facts) scope(resource Resource) string { + if resource.Supplied || f.exists(resource) { + return resource.ID + } + if f.absent(resource) { + return resource.Parent + } + return "" +} + +// Identity records distinct ARM, object, client and tenant identifiers. +type Identity struct { + ID string + PrincipalID string + ClientID string + TenantID string +} + +// ResourceChecker performs read-only AKS, selected-network and supporting-resource checks. +type ResourceChecker interface { + Check(context.Context, Plan) (Facts, Report, error) +} + +// Authorizer checks scoped operations and supplied-identity grants without writing. +type Authorizer interface { + Check(context.Context, Plan, Facts) (Report, error) +} diff --git a/cli/azd/extensions/azure.ai.projects/internal/provisioning/aks_hosting.go b/cli/azd/extensions/azure.ai.projects/internal/provisioning/aks_hosting.go index c66195fd5df..b73949f62bc 100644 --- a/cli/azd/extensions/azure.ai.projects/internal/provisioning/aks_hosting.go +++ b/cli/azd/extensions/azure.ai.projects/internal/provisioning/aks_hosting.go @@ -9,18 +9,18 @@ package provisioning import ( "context" "encoding/json" + "errors" "fmt" "os" "strconv" "strings" - "sync" "azure.ai.projects/internal/exterrors" + "azure.ai.projects/internal/onboarding" "azure.ai.projects/internal/synthesis" "github.com/Azure/azure-sdk-for-go/sdk/azcore/arm" - "github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/authorization/armauthorization/v2" - "github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/containerservice/armcontainerservice/v9" + "github.com/azure/azure-dev/cli/azd/pkg/grpcbroker" "github.com/google/uuid" ) @@ -31,7 +31,9 @@ type byoHostingParameters struct { WorkloadIdentityResourceID json.RawMessage `json:"workloadIdentityResourceId"` } -func (p *FoundryProvisioningProvider) prepareBYO(ctx context.Context, source *templateSource) error { +func (p *FoundryProvisioningProvider) prepareBYO( + ctx context.Context, source *templateSource, progress grpcbroker.ProgressFunc, +) error { hosting, err := byoHostingParameter(source) if err != nil { return err @@ -72,14 +74,15 @@ func (p *FoundryProvisioningProvider) prepareBYO(ctx context.Context, source *te } _, resolveFederationRole := hostBindings["federatedIdentityRoleDefinitionId"] identities := []struct { - field, binding, id, principal string - value json.RawMessage + field, binding, id string + value json.RawMessage + fact onboarding.Identity }{ {field: "hostingManagementIdentityResourceId", binding: "hostingPrincipalId", value: hosting.HostingManagementIdentityResourceID}, {field: "workloadIdentityResourceId", binding: "workloadPrincipalId", value: hosting.WorkloadIdentityResourceID}, } - workloadID := "" + roleSubscription := p.subID for i := range identities { identity := &identities[i] if _, needed := hostBindings[identity.binding]; !needed && @@ -98,11 +101,12 @@ func (p *FoundryProvisioningProvider) prepareBYO(ctx context.Context, source *te delete(hostBindings, identity.binding) continue } - if _, err := parseBYOResourceID(identity.id, "Microsoft.ManagedIdentity/userAssignedIdentities"); err != nil { + id, err := parseBYOResourceID(identity.id, "Microsoft.ManagedIdentity/userAssignedIdentities") + if err != nil { return err } if identity.binding == "workloadPrincipalId" { - workloadID = identity.id + roleSubscription = id.SubscriptionID } } enabled, err := strconv.ParseBool(os.Getenv(synthesis.BYOAKSEnvVar)) @@ -116,54 +120,59 @@ func (p *FoundryProvisioningProvider) prepareBYO(ctx context.Context, source *te if err := p.ensureCredential(ctx); err != nil { return err } - readCtx, cancelReads := context.WithCancelCause(ctx) - defer cancelReads(nil) - - var reads sync.WaitGroup - var kubeletObjectID, federationRoleID string - reads.Go(func() { - var err error - kubeletObjectID, err = p.readBYOKubeletObjectID(readCtx, clusterID) - if err != nil { - cancelReads(err) + plan, err := p.identifyBYOTemplatePlan(ctx, source) + if err != nil { + return err + } + evidence := onboarding.NewAuthorizationEvidence(p.credential, p.armClientOptions) + pipeline := onboarding.Pipeline{ + Resources: onboarding.NewResourceChecker(p.credential, p.armClientOptions), + Access: onboarding.NewAuthorizerWithEvidence(p.credential, p.armClientOptions, evidence), + } + facts, report, preflightErr := pipeline.Preflight(ctx, plan) + if err := writeBYOPreflight(report, preflightErr, progress); err != nil { + return err + } + _, needsKubelet := hostBindings["kubeletObjectId"] + if facts.KubeletObjectID == "" && (plan.Cluster.Missing == "" || needsKubelet) { + return fmt.Errorf("BYO AKS requires a kubelet principal object ID from preflight") + } + if facts.KubeletObjectID != "" { + if _, err := uuid.Parse(facts.KubeletObjectID); err != nil { + return fmt.Errorf("AKS kubelet principal object ID must be a GUID: %w", err) } - }) - if resolveFederationRole { - reads.Go(func() { - var err error - federationRoleID, err = p.resolveBYOFederationRoleID(readCtx, workloadID) - if err != nil { - cancelReads(err) - } - }) } + workloadScope := plan.Workload.ID + if workloadScope == "" { + workloadScope = plan.Workload.Parent + } + if resolveFederationRole && workloadScope != "" { + if scope, err := arm.ParseResourceID(workloadScope); err == nil && scope.SubscriptionID != "" { + roleSubscription = scope.SubscriptionID + } + } + identities[0].fact, identities[1].fact = facts.HostingIdentity, facts.WorkloadIdentity + resolvedBindings := map[string]string{"kubeletObjectId": facts.KubeletObjectID} for i := range identities { identity := &identities[i] if _, needed := hostBindings[identity.binding]; !needed { continue } - reads.Go(func() { - var err error - identity.principal, err = p.readBYOIdentityPrincipal(readCtx, identity.id) - if err != nil { - cancelReads(err) - } - }) - } - reads.Wait() - - if err := context.Cause(readCtx); err != nil { - return err - } - resolvedBindings := map[string]string{ - "federatedIdentityRoleDefinitionId": federationRoleID, - "kubeletObjectId": kubeletObjectID, + if !strings.EqualFold(identity.id, identity.fact.ID) || identity.fact.PrincipalID == "" { + return fmt.Errorf("%s deployment binding requires matching resource ID and principal ID from preflight", + identity.field) + } + if _, err := uuid.Parse(identity.fact.PrincipalID); err != nil { + return fmt.Errorf("supplied identity principal ID must be a GUID: %w", err) + } + resolvedBindings[identity.binding] = identity.fact.PrincipalID } - for _, identity := range identities { - if _, needed := hostBindings[identity.binding]; !needed { - continue + if resolveFederationRole { + roleID, err := resolveBYOFederationRoleID(ctx, roleSubscription, evidence) + if err != nil { + return err } - resolvedBindings[identity.binding] = identity.principal + resolvedBindings["federatedIdentityRoleDefinitionId"] = roleID } for name := range hostBindings { hostBindings[name] = map[string]any{"value": resolvedBindings[name]} @@ -172,6 +181,17 @@ func (p *FoundryProvisioningProvider) prepareBYO(ctx context.Context, source *te return nil } +func writeBYOPreflight(report onboarding.Report, preflightErr error, progress grpcbroker.ProgressFunc) error { + var output strings.Builder + if err := report.Write(&output); err != nil { + return errors.Join(preflightErr, err) + } + if output.Len() > 0 { + progress(output.String()) + } + return preflightErr +} + func byoARMOptions(options *arm.ClientOptions) *arm.ClientOptions { var result arm.ClientOptions if options != nil { @@ -197,77 +217,23 @@ func parseBYOResourceID(value, kind string) (*arm.ResourceID, error) { return id, nil } -func (p *FoundryProvisioningProvider) readBYOKubeletObjectID( - ctx context.Context, clusterID string, -) (string, error) { - id, err := parseBYOResourceID(clusterID, "Microsoft.ContainerService/managedClusters") - if err != nil { - return "", err - } - options := byoARMOptions(p.armClientOptions) - clusters, err := armcontainerservice.NewManagedClustersClient(id.SubscriptionID, p.credential, options) - if err != nil { - return "", fmt.Errorf("create AKS client: %w", err) - } - cluster, err := clusters.Get(ctx, id.ResourceGroupName, id.Name, nil) - if err != nil { - return "", fmt.Errorf("read AKS kubelet identity: %w", err) - } - if cluster.Properties == nil || cluster.Properties.IdentityProfile["kubeletidentity"] == nil || - cluster.Properties.IdentityProfile["kubeletidentity"].ObjectID == nil { - return "", fmt.Errorf("BYO AKS requires a kubelet principal object ID") - } - kubeletObjectID := *cluster.Properties.IdentityProfile["kubeletidentity"].ObjectID - if _, err := uuid.Parse(kubeletObjectID); err != nil { - return "", fmt.Errorf("AKS kubelet principal object ID must be a GUID: %w", err) - } - return kubeletObjectID, nil -} - -func (p *FoundryProvisioningProvider) resolveBYOFederationRoleID( - ctx context.Context, workloadID string, +func resolveBYOFederationRoleID( + ctx context.Context, roleSubscription string, evidence *onboarding.AuthorizationEvidence, ) (string, error) { const roleName = "Managed Identity Federated Identity Credential Contributor" - roleSubscription := p.subID - if workloadID != "" { - workload, err := parseBYOResourceID(workloadID, "Microsoft.ManagedIdentity/userAssignedIdentities") - if err != nil { - return "", err - } - roleSubscription = workload.SubscriptionID - } - roles, err := armauthorization.NewRoleDefinitionsClient(p.credential, byoARMOptions(p.armClientOptions)) + id, err := evidence.ResolveRoleByName(ctx, roleSubscription, roleName) if err != nil { - return "", fmt.Errorf("create role definitions client: %w", err) + return "", fmt.Errorf("resolve built-in federation role: %w", err) } - pager := roles.NewListPager("/subscriptions/"+roleSubscription, - &armauthorization.RoleDefinitionsClientListOptions{Filter: new("roleName eq '" + roleName + "'")}) - for pager.More() { - page, err := pager.NextPage(ctx) - if err != nil { - return "", fmt.Errorf("resolve built-in federation role: %w", err) - } - for _, role := range page.Value { - if role == nil || role.Properties == nil || role.Properties.RoleName == nil || - *role.Properties.RoleName != roleName || role.Properties.RoleType == nil || - !strings.EqualFold(*role.Properties.RoleType, "BuiltInRole") { - continue - } - if role.ID == nil { - return "", fmt.Errorf("built-in federation role has an invalid ID") - } - roleID, err := arm.ParseResourceID(*role.ID) - if err != nil || !strings.EqualFold(roleID.ResourceType.String(), "Microsoft.Authorization/roleDefinitions") { - return "", fmt.Errorf("built-in federation role has an invalid ID") - } - if _, err := uuid.Parse(roleID.Name); err != nil { - return "", fmt.Errorf("BYO AKS federation role requires a role definition GUID: %w", err) - } - return "/subscriptions/" + roleSubscription + - "/providers/Microsoft.Authorization/roleDefinitions/" + roleID.Name, nil - } + roleID, err := arm.ParseResourceID(id) + if err != nil || !strings.EqualFold(roleID.ResourceType.String(), "Microsoft.Authorization/roleDefinitions") { + return "", fmt.Errorf("built-in federation role has an invalid ID") + } + if _, err := uuid.Parse(roleID.Name); err != nil { + return "", fmt.Errorf("BYO AKS federation role requires a role definition GUID: %w", err) } - return "", fmt.Errorf("required built-in role %q is unavailable", roleName) + return "/subscriptions/" + roleSubscription + + "/providers/Microsoft.Authorization/roleDefinitions/" + roleID.Name, nil } func byoHostingParameter(source *templateSource) (byoHostingParameters, error) { diff --git a/cli/azd/extensions/azure.ai.projects/internal/provisioning/aks_hosting_compatibility_test.go b/cli/azd/extensions/azure.ai.projects/internal/provisioning/aks_hosting_compatibility_test.go new file mode 100644 index 00000000000..3e705812362 --- /dev/null +++ b/cli/azd/extensions/azure.ai.projects/internal/provisioning/aks_hosting_compatibility_test.go @@ -0,0 +1,271 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +package provisioning + +import ( + "fmt" + "strings" + "testing" + + "azure.ai.projects/internal/synthesis" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +func TestBYONonBooleanIncludeAcrReachesARM(t *testing.T) { + for _, preview := range []bool{false, true} { + for _, value := range []any{"true", "false", "[equals(1, 1)]", 1.0, nil} { + t.Run(fmt.Sprintf("preview=%t/value=%v", preview, value), func(t *testing.T) { + p, fake := offlineBYOProvider(t) + p.onDiskSource = &templateSource{ + mode: templateModeBicep, armTemplate: p.armTemplate, + parameters: map[string]any{"includeAcr": map[string]any{"value": value}}, + } + var err error + if preview { + _, err = p.Preview(t.Context(), func(string) {}) + } else { + _, err = p.Deploy(t.Context(), func(string) {}) + } + require.ErrorContains(t, err, "InvalidTemplate") + require.ErrorContains(t, err, "offline ARM includeAcr type validation") + assert.Equal(t, map[string]any{"value": value}, fake.parameters["includeAcr"]) + assert.NotContains(t, fake.parameters, "kubeletObjectId") + assert.Contains(t, fake.requests, "GET "+offlineClusterID) + assert.Zero(t, fake.writes) + assert.Zero(t, fake.previews) + }) + } + } +} + +func TestBYOPreparePreservesSuppliedKubeletRegardlessOfACR(t *testing.T) { + for _, includeAcr := range []bool{false, true} { + for _, test := range []struct { + name string + parameter any + duplicate bool + }{ + {"empty", map[string]any{"value": ""}, false}, + {"whitespace", map[string]any{"value": " "}, false}, + {"non-string", map[string]any{"value": 1}, false}, + {"null", map[string]any{"value": nil}, false}, + {"reference", map[string]any{"reference": map[string]any{"secretName": "kubelet"}}, false}, + {"duplicate", map[string]any{"value": offlineKubeletObjectID}, true}, + {"different", map[string]any{"value": suppliedHostingPrincipal}, false}, + {"matching", map[string]any{"value": strings.ToUpper(offlineKubeletObjectID)}, false}, + } { + t.Run(fmt.Sprintf("includeAcr=%t/%s", includeAcr, test.name), func(t *testing.T) { + p, fake := offlineBYOProvider(t) + source, err := p.resolveTemplate(t.Context(), func(string) {}) + require.NoError(t, err) + delete(source.parameters, "includeAcr") + source.parameters["INCLUDEACR"] = map[string]any{"value": includeAcr} + source.parameters["kubeletObjectId"] = test.parameter + if test.duplicate { + source.parameters["KUBELETOBJECTID"] = test.parameter + } + require.NoError(t, p.prepareBYO(t.Context(), source, func(string) {})) + assert.Contains(t, fake.requests, "GET "+offlineClusterID) + assert.Equal(t, test.parameter, source.parameters["kubeletObjectId"]) + if test.duplicate { + assert.Equal(t, test.parameter, source.parameters["KUBELETOBJECTID"]) + } + assert.Equal(t, map[string]any{"value": includeAcr}, source.parameters["INCLUDEACR"]) + }) + } + } +} + +func TestBYOSuppliedBindingsPreserveExplicitValues(t *testing.T) { + for _, binding := range []string{"hostingPrincipalId", "workloadPrincipalId", "federatedIdentityRoleDefinitionId"} { + for _, declared := range []bool{false, true} { + for _, preview := range []bool{false, true} { + for _, parameter := range []any{ + map[string]any{"value": "explicit-" + binding}, + map[string]any{"value": ""}, + map[string]any{"reference": map[string]any{"secretName": binding}}, + } { + t.Run(fmt.Sprintf("%s/declared=%t/preview=%t/%v", binding, declared, preview, parameter), + func(t *testing.T) { + p, fake, _ := suppliedProvider(t, 7) + source, err := p.resolveTemplate(t.Context(), func(string) {}) + require.NoError(t, err) + parameterName := strings.ToUpper(binding) + source.parameters[parameterName] = parameter + if !declared { + definitions, ok := source.armTemplate["parameters"].(map[string]any) + require.True(t, ok) + delete(definitions, binding) + } + p.onDiskSource = source + if preview { + _, err = p.Preview(t.Context(), func(string) {}) + } else { + _, err = p.Deploy(t.Context(), func(string) {}) + } + if declared { + require.NoError(t, err) + } else { + require.ErrorContains(t, err, "InvalidTemplate") + require.ErrorContains(t, err, "offline ARM undeclared parameter: "+parameterName) + assert.Zero(t, fake.base.writes) + assert.Zero(t, fake.base.previews) + } + assert.Equal(t, parameter, fake.base.parameters[parameterName]) + assert.NotContains(t, fake.base.parameters, binding) + assert.Contains(t, fake.requests, "GET "+offlineClusterID) + }) + } + } + } + } +} + +func TestBYOSuppliedUsesEffectiveDeploymentInputs(t *testing.T) { + for _, name := range []string{"parameters", "template default", "without YAML", "disabled", "undeclared"} { + t.Run(name, func(t *testing.T) { + p, fake, _ := suppliedProvider(t, 7) + source, err := p.resolveTemplate(t.Context(), func(string) {}) + require.NoError(t, err) + clusterID := offlineClusterID + "-effective" + fake.base.clusterID = clusterID + hosting := map[string]any{ + "enabled": true, "hostingType": "ManagedCluster", "name": "custom", + "clusterResourceId": clusterID, "storageAccountResourceId": suppliedStorageID, + "hostingManagementIdentityResourceId": suppliedHostingID, + "workloadIdentityResourceId": suppliedWorkloadID, + "customSetting": true, + } + delete(source.parameters, "agentHosting") + source.parameters["AGENTHOSTING"] = map[string]any{"value": hosting} + definitions, ok := source.armTemplate["parameters"].(map[string]any) + require.True(t, ok) + switch name { + case "template default", "without YAML": + result, err := synthesis.Synthesize(synthesis.Input{ServiceName: "project", + RawAzureYAML: []byte("services:\n project:\n host: azure.ai.project\n")}) + require.NoError(t, err) + p.synthResult = result + if name == "template default" { + delete(source.parameters, "AGENTHOSTING") + definition, ok := definitions["agentHosting"].(map[string]any) + require.True(t, ok) + definition["defaultValue"] = hosting + } + case "disabled": + hosting["enabled"], hosting["clusterResourceId"] = false, 42 + case "undeclared": + delete(definitions, "agentHosting") + source.parameters["AGENTHOSTING"] = map[string]any{"value": "leave undeclared parameters to ARM"} + } + require.NoError(t, p.prepareBYO(t.Context(), source, func(string) {})) + assert.NotContains(t, source.parameters, "agentHosting") + if name == "disabled" || name == "undeclared" { + assert.Empty(t, fake.requests) + assert.NotContains(t, source.parameters, "federatedIdentityRoleDefinitionId") + } else { + assert.Contains(t, fake.requests, "GET "+clusterID) + assert.NotContains(t, fake.requests, "GET "+offlineClusterID) + assert.Equal(t, map[string]any{"value": suppliedHostingPrincipal}, source.parameters["hostingPrincipalId"]) + assert.Equal(t, map[string]any{"value": suppliedWorkloadPrincipal}, source.parameters["workloadPrincipalId"]) + if name == "template default" { + assert.NotContains(t, source.parameters, "AGENTHOSTING") + } else { + assert.Equal(t, map[string]any{"value": hosting}, source.parameters["AGENTHOSTING"]) + } + } + }) + } +} + +func TestBYOPlanOptionalSupportingSelections(t *testing.T) { + for _, value := range []any{nil, "", suppliedStorageID} { + t.Run(fmt.Sprint(value), func(t *testing.T) { + p, _ := offlineBYOProvider(t) + env := (&templatePlanResolver{}).environment(p.armTemplate, map[string]func() any{ + "agenthosting": func() any { return map[string]any{"storageAccountResourceId": value} }, + }, planScope{}, "root", "test") + result := env.value("[coalesce(tryGet(parameters('agentHosting'), 'storageAccountResourceId'), '')]") + expected := value + if expected == nil { + expected = "" + } + assert.Equal(t, expected, result) + assert.Equal(t, "", env.value("[coalesce(tryGet(parameters('agentHosting'), 'missing'), '')]")) + assert.IsType(t, planUnknown{}, env.value("[coalesce(parameters('unknown'), 'fallback')]")) + assert.Equal(t, "known", env.value("[coalesce('known', parameters('unknown'))]")) + }) + } +} + +func TestBYOSuppliedAllowsIdenticalIdentities(t *testing.T) { + p, fake, _ := suppliedProvider(t, 7) + result, err := synthesis.Synthesize(synthesis.Input{ServiceName: "project", + RawAzureYAML: []byte(strings.Replace(suppliedYAML(7), suppliedWorkloadID, suppliedHostingID, 1))}) + require.NoError(t, err) + p.synthResult = result + _, err = p.Deploy(t.Context(), func(string) {}) + require.NoError(t, err) + assert.Equal(t, 1, fake.base.writes) + for _, binding := range []string{"hostingPrincipalId", "workloadPrincipalId"} { + assert.Equal(t, map[string]any{"value": suppliedHostingPrincipal}, fake.base.parameters[binding]) + } +} + +func TestBYOSuppliedPreflightUsesCustomizedCompiledPlan(t *testing.T) { + p, fake, compiler := suppliedProvider(t, 7) + p.armTemplate["resources"] = map[string]any{"customDeployment": map[string]any{ + "type": "Microsoft.Resources/deployments", "name": "custom-deployment", "resourceGroup": "custom-rg", + "properties": map[string]any{ + "expressionEvaluationOptions": map[string]any{"scope": "inner"}, + "template": map[string]any{"resources": map[string]any{"customAccount": map[string]any{ + "type": "Microsoft.CognitiveServices/accounts", "name": "custom-account", + "properties": map[string]any{"agentHostingConfigurations": []any{map[string]any{ + "hostingType": "ManagedCluster", "clusterResourceId": offlineClusterID, + "storageAccountResourceId": suppliedStorageID, + "hostingManagementIdentityResourceId": suppliedHostingID, + "workloadIdentityResourceId": suppliedWorkloadID, + }}}, + }}}, + }, + }} + p.onDiskSource = &templateSource{ + mode: templateModeBicep, armTemplate: p.armTemplate, + parameters: map[string]any{"includeAcr": map[string]any{"value": true}}, + } + source, err := p.resolveTemplate(t.Context(), func(string) {}) + require.NoError(t, err) + plan, err := p.identifyBYOTemplatePlan(t.Context(), source) + require.NoError(t, err) + assert.Equal(t, "custom-rg", plan.ResourceGroup) + assert.Equal(t, suppliedStorageID, plan.Storage.ID) + assert.True(t, plan.Storage.Supplied) + var progress strings.Builder + _, err = p.Deploy(t.Context(), func(message string) { progress.WriteString(message) }) + require.NoError(t, err, progress.String()) + assert.Equal(t, 1, fake.base.writes) + assert.Equal(t, p.armTemplate, fake.base.template) + assert.Empty(t, compiler.buildParamCalls, "the customized compiled plan has no resource-token expression") + assert.Contains(t, progress.String(), "S03 PASS") + assert.NotContains(t, progress.String(), "P05 ") + assert.NotContains(t, progress.String(), "P07 ") + assert.Equal(t, map[string]any{"value": offlineKubeletObjectID}, fake.base.parameters["kubeletObjectId"]) +} + +func TestBYOSuppliedIdentityRequiresRealPrincipalBinding(t *testing.T) { + for _, body := range []map[string]any{ + {"id": suppliedHostingID}, + {"id": suppliedHostingID, "properties": map[string]any{"principalId": "invalid"}}, + {"id": suppliedWorkloadID, "properties": map[string]any{"principalId": suppliedHostingPrincipal}}, + } { + p, fake, _ := suppliedProvider(t, 2) + fake.overrides[strings.ToLower(suppliedHostingID)] = suppliedReply{status: 200, body: body} + _, err := p.Deploy(t.Context(), func(string) {}) + require.Error(t, err) + assert.Zero(t, fake.base.writes) + assert.Zero(t, fake.base.previews) + } +} diff --git a/cli/azd/extensions/azure.ai.projects/internal/provisioning/aks_hosting_test.go b/cli/azd/extensions/azure.ai.projects/internal/provisioning/aks_hosting_test.go index 55e9e248ef9..c4748946359 100644 --- a/cli/azd/extensions/azure.ai.projects/internal/provisioning/aks_hosting_test.go +++ b/cli/azd/extensions/azure.ai.projects/internal/provisioning/aks_hosting_test.go @@ -6,21 +6,25 @@ package provisioning import ( + "bytes" "context" "encoding/json" "errors" "fmt" "io" + "io/fs" "net/http" "os" + "os/exec" "path/filepath" "strings" "sync" "testing" - "time" "azure.ai.projects/internal/exterrors" + "azure.ai.projects/internal/onboarding" "azure.ai.projects/internal/synthesis" + "azure.ai.projects/internal/testutil" "github.com/Azure/azure-sdk-for-go/sdk/azcore" "github.com/Azure/azure-sdk-for-go/sdk/azcore/arm" @@ -31,9 +35,13 @@ import ( "github.com/stretchr/testify/require" ) -const offlineClusterID = "/subscriptions/cluster-sub/resourceGroups/aks-rg/providers/" + +const offlineClusterGroupID = "/subscriptions/cluster-sub/resourceGroups/aks-rg" +const offlineVNetID = "/subscriptions/network-sub/resourceGroups/net-rg/providers/Microsoft.Network/virtualNetworks/net" +const offlineClusterID = offlineClusterGroupID + "/providers/" + "Microsoft.ContainerService/managedClusters/aks" const offlineKubeletObjectID = "11111111-aaaa-4111-8111-111111111111" +const offlineDeploymentRoleID = "/subscriptions/cluster-sub/providers/Microsoft.Authorization/" + + "roleDefinitions/bbbbbbbb-bbbb-4bbb-8bbb-bbbbbbbbbbbb" const offlineProjectID = "/subscriptions/target-sub/resourceGroups/target-rg/providers/" + "Microsoft.CognitiveServices/accounts/cog-native/projects/project" const offlineProjectEndpoint = "https://cog-native.services.ai.azure.com/api/projects/project" @@ -48,21 +56,25 @@ services: type byoTestCredential struct{} -func (byoTestCredential) GetToken(context.Context, policy.TokenRequestOptions) (azcore.AccessToken, error) { - return azcore.AccessToken{Token: "offline-only", ExpiresOn: time.Now().Add(time.Hour)}, nil +func (byoTestCredential) GetToken(ctx context.Context, options policy.TokenRequestOptions) (azcore.AccessToken, error) { + return suppliedCredential{}.GetToken(ctx, options) } type byoARMFake struct { - mu sync.Mutex - t *testing.T - provider *FoundryProvisioningProvider - requests []string - parameters map[string]any - template map[string]any - clusterProperties map[string]any - writes int - previews int - failWrite bool + mu sync.Mutex + t *testing.T + provider *FoundryProvisioningProvider + requests []string + parameters map[string]any + template map[string]any + writes int + previews int + failWrite bool + clusterVersion string + clusterID string + kubeletID string + nodeSubnetID string + nodePrefix string } func (f *byoARMFake) response(request *http.Request, status int, body any) *http.Response { @@ -76,18 +88,116 @@ func (f *byoARMFake) Do(request *http.Request) (*http.Response, error) { f.mu.Lock() defer f.mu.Unlock() f.requests = append(f.requests, request.Method+" "+request.URL.Path) - if request.Method == http.MethodGet && request.URL.Path == offlineClusterID { - return f.response(request, 200, map[string]any{"properties": f.clusterProperties}), nil + if request.Method == http.MethodGet && request.URL.Path == f.clusterID { + return f.response(request, 200, map[string]any{ + "id": f.clusterID, "location": "westus", "sku": map[string]any{"name": "Base"}, + "identity": map[string]any{"type": "SystemAssigned"}, + "properties": map[string]any{ + "currentKubernetesVersion": f.clusterVersion, "provisioningState": "Succeeded", + "networkProfile": map[string]any{ + "networkPlugin": "azure", "networkPluginMode": "overlay", "networkDataplane": "cilium", + }, + "identityProfile": map[string]any{ + "kubeletidentity": map[string]any{"objectId": f.kubeletID}, + }, + "oidcIssuerProfile": map[string]any{"enabled": true, "issuerURL": "https://issuer.example.test/"}, + "securityProfile": map[string]any{"workloadIdentity": map[string]any{"enabled": true}}, + }, + }), nil + } + if request.Method == http.MethodGet && request.URL.Path == f.clusterID+"/agentPools" { + return f.response(request, 200, map[string]any{"value": []any{ + map[string]any{"id": f.clusterID + "/agentPools/system", "name": "system", + "properties": map[string]any{ + "mode": "System", "vmSize": "Standard_E4s_v5", "count": 1, "provisioningState": "Succeeded", + }}, + map[string]any{"id": f.clusterID + "/agentPools/agents", "name": "agents", + "properties": map[string]any{ + "mode": "User", "osSKU": "AzureLinux", "vmSize": "Standard_D8as_v5", "count": 1, + "workloadRuntime": "KataVmIsolation", "nodeImageVersion": "AKSAzureLinux-V3katagen2-202608.14.0", + "nodeLabels": map[string]any{"agents.ai.azure.com/sandbox-node": "true"}, + "securityProfile": map[string]any{"sshAccess": "Disabled"}, "provisioningState": "Succeeded", + "vnetSubnetID": f.nodeSubnetID, + }}, + }}), nil } if request.Method == http.MethodGet && - request.URL.Path == "/subscriptions/target-sub/providers/Microsoft.Authorization/roleDefinitions" { + strings.EqualFold(request.URL.Path, "/subscriptions/cluster-sub/providers/Microsoft.Compute/skus") { + var values []any + for _, size := range []struct{ name, cpu string }{{"Standard_E4s_v5", "4"}, {"Standard_D8as_v5", "8"}} { + values = append(values, map[string]any{ + "name": size.name, "resourceType": "virtualMachines", "locations": []string{"westus"}, + "capabilities": []any{map[string]any{"name": "vCPUs", "value": size.cpu}, + map[string]any{"name": "MemoryGB", "value": "32"}}, + }) + } + return f.response(request, 200, map[string]any{"value": values}), nil + } + if request.Method == http.MethodGet && request.URL.Path == offlineVNetID { + return f.response(request, 200, map[string]any{ + "id": offlineVNetID, "location": "westus", "properties": map[string]any{ + "provisioningState": "Succeeded", "addressSpace": map[string]any{ + "addressPrefixes": []string{"172.16.0.0/16"}, + }, + }, + }), nil + } + if request.Method == http.MethodGet && (request.URL.Path == offlineVNetID+"/subnets/agents" || + request.URL.Path == offlineVNetID+"/subnets/nodes" || request.URL.Path == offlineVNetID+"/subnets/pe") { + prefix := "172.16.0.0/23" + var delegations []any + if strings.HasSuffix(request.URL.Path, "/nodes") { + prefix = f.nodePrefix + } else if strings.HasSuffix(request.URL.Path, "/pe") { + prefix = "172.16.3.0/24" + } else { + delegations = []any{map[string]any{ + "name": "agents", "properties": map[string]any{"serviceName": "Microsoft.App/environments"}, + }} + } + return f.response(request, 200, map[string]any{"id": request.URL.Path, "properties": map[string]any{ + "provisioningState": "Succeeded", "addressPrefix": prefix, "delegations": delegations, + }}), nil + } + if request.Method == http.MethodGet && strings.HasPrefix(request.URL.Path, "/subscriptions/") && + len(strings.Split(request.URL.Path, "/")) == 3 { + return f.response(request, 200, map[string]any{"id": request.URL.Path, "tenantId": "tenant"}), nil + } + if request.Method == http.MethodGet && strings.Contains(strings.ToLower(request.URL.Path), "/resourcegroups/") && + len(strings.Split(request.URL.Path, "/")) == 5 { + return f.response(request, 200, map[string]any{ + "id": strings.Replace(request.URL.Path, "/resourcegroups/", "/resourceGroups/", 1), "location": "westus", + }), nil + } + if request.Method == http.MethodGet && (strings.Contains(strings.ToLower(request.URL.Path), "/microsoft.storage/") || + strings.Contains(strings.ToLower(request.URL.Path), "/microsoft.managedidentity/") || + strings.Contains(strings.ToLower(request.URL.Path), "/microsoft.containerregistry/")) { + return f.response(request, 404, map[string]any{"error": map[string]any{"code": "ResourceNotFound"}}), nil + } + if request.Method == http.MethodGet && + strings.HasSuffix(strings.ToLower(request.URL.Path), "/providers/microsoft.authorization/roledefinitions") { return f.response(request, 200, map[string]any{"value": []any{map[string]any{ - "id": "/subscriptions/target-sub/providers/Microsoft.Authorization/" + - "roleDefinitions/aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa", + "id": request.URL.Path + "/aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa", "properties": map[string]any{"roleName": "Managed Identity Federated Identity Credential Contributor", "type": "BuiltInRole"}, }}}), nil } + if request.Method == http.MethodGet && + strings.HasSuffix(strings.ToLower(request.URL.Path), "/providers/microsoft.authorization/roleassignments") { + scope := request.URL.Path[:len(request.URL.Path)-len("/providers/Microsoft.Authorization/roleAssignments")] + return f.response(request, 200, map[string]any{"value": []any{map[string]any{ + "id": offlineClusterGroupID + "/providers/Microsoft.Authorization/roleAssignments/deployment", + "properties": map[string]any{ + "scope": scope, "principalId": suppliedOperator, + "roleDefinitionId": offlineDeploymentRoleID, + }, + }}}), nil + } + if request.Method == http.MethodGet && request.URL.Path == offlineDeploymentRoleID { + return f.response(request, 200, map[string]any{"id": offlineDeploymentRoleID, "properties": map[string]any{ + "permissions": []any{map[string]any{"actions": []string{"*"}}}, + }}), nil + } path := "/subscriptions/target-sub/providers/Microsoft.Resources/deployments/" + f.provider.deploymentName() if request.Method == http.MethodGet && request.URL.Path == path { return f.response(request, http.StatusNotFound, map[string]any{ @@ -127,7 +237,11 @@ func (f *byoARMFake) Do(request *http.Request) (*http.Response, error) { } } acrParameter, ok := f.parameters["includeAcr"].(map[string]any) - require.True(f.t, ok) + if !ok { + return f.response(request, 400, map[string]any{ + "error": map[string]string{"code": "InvalidTemplate", "message": "offline ARM includeAcr type validation"}, + }), nil + } includeAcr, ok := acrParameter["value"].(bool) if !ok { return f.response(request, 400, map[string]any{ @@ -191,11 +305,13 @@ func offlineBYOProvider(t *testing.T) (*FoundryProvisioningProvider, *byoARMFake tenantID: "user-access-tenant", virtualEnv: map[string]string{}, } fake := &byoARMFake{ - t: t, provider: p, - clusterProperties: map[string]any{"identityProfile": map[string]any{ - "kubeletidentity": map[string]any{"objectId": offlineKubeletObjectID}, - }}, + t: t, provider: p, clusterVersion: "1.31.0", clusterID: offlineClusterID, + kubeletID: offlineKubeletObjectID, nodePrefix: "172.16.2.0/24", } + p.bicepCliInstance = &stubCompiler{buildParamResult: bicep.BuildResult{ + Compiled: `{"parametersJson":"{\"parameters\":{\"result\":{\"value\":\"abcdefghijklm\"},` + + `\"resourceToken\":{\"value\":\"abcdefghijklm\"}}}"}`, + }} p.armClientOptions = &arm.ClientOptions{ClientOptions: policy.ClientOptions{ Transport: fake, Retry: policy.RetryOptions{MaxRetries: -1}, }} @@ -239,7 +355,7 @@ func TestBYOPrepareFeatureGateUsesProcessEnvironment(t *testing.T) { {"True", "True", false, true}, } { t.Run(test.name, func(t *testing.T) { - p, fake := suppliedProvider(t, 7) + p, fake, _ := suppliedProvider(t, 7) t.Setenv(synthesis.BYOAKSEnvVar, test.value) if test.unset { require.NoError(t, os.Unsetenv(synthesis.BYOAKSEnvVar)) @@ -253,12 +369,12 @@ func TestBYOPrepareFeatureGateUsesProcessEnvironment(t *testing.T) { p.credential = nil } - err = p.prepareBYO(t.Context(), source) + err = p.prepareBYO(t.Context(), source, func(string) {}) if test.enabled { require.NoError(t, err) - assert.ElementsMatch(t, []string{ + assert.Subset(t, fake.reads, []string{ offlineClusterID, - "/subscriptions/workload-sub/providers/Microsoft.Authorization/roleDefinitions", + "/subscriptions/target-sub/providers/Microsoft.Authorization/roleDefinitions", suppliedHostingID, suppliedWorkloadID, }, fake.reads) @@ -377,8 +493,7 @@ func TestBYODeployReturnsStandardProjectOutputs(t *testing.T) { p, fake := offlineBYOProvider(t) server := &resolveEnvStubEnvServer{get: map[string]string{"AZURE_CONTAINER_REGISTRY_RESOURCE_ID": offlineRegistryID}} p.azdClient = newResolveEnvTestClient(t, server, &resolveEnvStubPromptServer{}) - compiler := &stubCompiler{} - p.bicepCliInstance = compiler + compiler := p.bicepCliInstance.(*stubCompiler) result, err := p.Deploy(t.Context(), func(string) {}) require.NoError(t, err) require.NotNil(t, result.Deployment) @@ -393,18 +508,16 @@ func TestBYODeployReturnsStandardProjectOutputs(t *testing.T) { assert.Equal(t, value, result.Deployment.Outputs[name].Value) } assert.Equal(t, 1, fake.writes) - require.Len(t, fake.requests, 4) - assert.ElementsMatch(t, []string{ - "GET " + offlineClusterID, - "GET /subscriptions/target-sub/providers/Microsoft.Authorization/roleDefinitions", - }, fake.requests[:2]) - assert.Equal(t, "GET /subscriptions/target-sub/providers/Microsoft.Resources/deployments/"+p.deploymentName(), - fake.requests[2]) - assert.Equal(t, "PUT /subscriptions/target-sub/providers/Microsoft.Resources/deployments/"+p.deploymentName(), - fake.requests[3]) + assert.Contains(t, fake.requests, "GET "+offlineClusterID) + assert.Contains(t, fake.requests, "GET "+offlineClusterID+"/agentPools") + assert.Contains(t, fake.requests, "GET "+offlineClusterGroupID+"/providers/Microsoft.Authorization/roleAssignments") + assert.Contains(t, fake.requests, "GET "+suppliedGroupID+"/providers/Microsoft.Authorization/roleAssignments") + for _, request := range fake.requests[:len(fake.requests)-1] { + assert.True(t, strings.HasPrefix(request, "GET "), request) + } assert.Empty(t, server.set) assert.Empty(t, compiler.buildCalls) - assert.Empty(t, compiler.buildParamCalls, "embedded provisioning must not invoke a Bicep naming probe") + assert.NotEmpty(t, compiler.buildParamCalls, "actual template naming is evaluated with the existing Bicep tool") assert.NotContains(t, fake.parameters, "kubeletObjectId") assert.Contains(t, fake.parameters, "federatedIdentityRoleDefinitionId") } @@ -511,11 +624,12 @@ func TestBYOFailedDeploymentCanBeRetried(t *testing.T) { fake.failWrite = true _, err := p.Deploy(t.Context(), func(string) {}) require.ErrorContains(t, err, "offline deployment failure") + firstRequests := len(fake.requests) fake.failWrite = false _, err = p.Deploy(t.Context(), func(string) {}) require.NoError(t, err) assert.Equal(t, 2, fake.writes) - assert.Len(t, fake.requests, 8) + assert.Len(t, fake.requests, firstRequests*2) assert.Empty(t, p.virtualEnv) } @@ -535,8 +649,7 @@ func TestBYOAllowsCustomizedOnDiskTemplateAndParameters(t *testing.T) { assert.Equal(t, "2.0.0.0", fake.template["contentVersion"]) assert.Equal(t, map[string]any{"value": "custom"}, fake.parameters["resourceTokenSalt"]) assert.Equal(t, map[string]any{"value": true}, fake.parameters["includeAcr"]) - assert.Equal(t, map[string]any{"value": offlineKubeletObjectID}, fake.parameters["kubeletObjectId"]) - assert.Contains(t, fake.requests, "GET "+offlineClusterID) + assert.Contains(t, fake.parameters, "kubeletObjectId") p.onDiskSource.parameters["resourceTokenSalt"] = map[string]any{"value": "updated"} _, err = p.Deploy(t.Context(), func(string) {}) require.ErrorContains(t, err, "retain the deployed resourceTokenSalt") @@ -548,54 +661,402 @@ func TestBYOAllowsCustomizedOnDiskTemplateAndParameters(t *testing.T) { assert.Equal(t, map[string]any{"value": "custom"}, fake.parameters["resourceTokenSalt"]) assert.Equal(t, map[string]any{"value": map[string]any{"team": "updated"}}, fake.parameters["tags"]) assert.Equal(t, 2, fake.writes) -} -func TestBYONonBooleanIncludeAcrReachesARM(t *testing.T) { - for _, operation := range []string{"deploy", "preview"} { - for _, test := range []struct { - name string - value any - }{ - {"string true", "true"}, - {"string false", "false"}, - {"expression", "[equals(1, 1)]"}, - {"number", 1.0}, - {"null", nil}, - } { - t.Run(operation+"/"+test.name, func(t *testing.T) { - p, fake := offlineBYOProvider(t) - p.onDiskSource = &templateSource{ - mode: templateModeBicep, armTemplate: p.armTemplate, - parameters: map[string]any{"includeAcr": map[string]any{"value": test.value}}, + for _, scenario := range []string{ + "renamed resources", "cross subscription", "cross subscription Foundry", "unknown Storage name", + "unknown Storage and known AKS failure", "known missing Storage reference", + "unknown workload name", "actual cluster selection", "no native pull assignments", + "cross subscription unknown workload", + "native registry without connection", + "selected existing network", "new selected subnets", "external DNS", "overlapping selected Node subnet", + "unknown network injections", "unknown network connections", "unknown network DNS configs", + "empty network selections", "unrelated network endpoint", + } { + t.Run(scenario, func(t *testing.T) { + tool := testutil.LookupBicep(t) + unknownField := map[string]string{ + "unknown network injections": "networkInjections", + "unknown network connections": "privateLinkServiceConnections", + "unknown network DNS configs": "privateDnsZoneConfigs", + }[scenario] + noNetworkSelection := scenario == "empty network selections" || scenario == "unrelated network endpoint" + provider, requests := offlineBYOProvider(t) + root := t.TempDir() + require.NoError(t, fs.WalkDir(synthesis.TemplatesFS(), "templates", + func(path string, entry fs.DirEntry, err error) error { + if err != nil || entry.IsDir() { + return err + } + if !strings.HasSuffix(path, ".bicep") && !strings.HasSuffix(path, "abbreviations.json") { + return nil + } + raw, err := synthesis.TemplatesFS().ReadFile(path) + if err != nil { + return err + } + target := filepath.Join(root, filepath.FromSlash(strings.TrimPrefix(path, "templates/"))) + if err := os.MkdirAll(filepath.Dir(target), 0o750); err != nil { + return err + } + //nolint:gosec // G703: embedded fixture path under t.TempDir. + return os.WriteFile(target, raw, 0o600) + })) + resourcePath := filepath.Join(root, "modules", "resources.bicep") + //nolint:gosec // Embedded fixture copied to t.TempDir. + body, err := os.ReadFile(resourcePath) + require.NoError(t, err) + edited := string(body) + for _, replacement := range []struct{ from, to string }{ + {"var foundryAccountName = '${abbrs.cognitiveServicesAccounts}${resourceToken}'", + "var foundryAccountName = 'edited-account'"}, + {"var storageAccountName = 'st${resourceToken}'", "var storageAccountName = 'editedstore'"}, + {"var hostingIdentityName = '${foundryAccountName}-hosting-mi'", + "var hostingIdentityName = 'edited-hosting'"}, + {"var workloadIdentityName = '${foundryAccountName}-workload-mi'", + "var workloadIdentityName = 'edited-workload'"}, + {"name: '${abbrs.containerRegistryRegistries}${resourceToken}'", "name: 'editedregistry'"}, + } { + require.Contains(t, edited, replacement.from) + edited = strings.Replace(edited, replacement.from, replacement.to, 1) + } + subscription, group := "target-sub", "target-rg" + if scenario == "cross subscription" || scenario == "cross subscription unknown workload" { + subscription, group = "support-sub", "support-rg" + edited = strings.Replace(edited, "name: 'byo-prerequisites'", + "name: 'byo-prerequisites'\n scope: resourceGroup('support-sub', 'support-rg')", 1) + for _, kind := range []string{"Microsoft.Storage/storageAccounts", + "Microsoft.ManagedIdentity/userAssignedIdentities"} { + edited = strings.ReplaceAll(edited, "resourceId('"+kind+"', ", + "resourceId('support-sub', 'support-rg', '"+kind+"', ") } - method, suffix := http.MethodPut, "" - var err error - if operation == "preview" { - method, suffix = http.MethodPost, "/whatIf" - _, err = p.Preview(t.Context(), func(string) {}) + } + if scenario == "cross subscription Foundry" { + subscription, group = "foundry-sub", "foundry-rg" + path := filepath.Join(root, "main.bicep") + //nolint:gosec // Embedded fixture copied to t.TempDir. + raw, err := os.ReadFile(path) + require.NoError(t, err) + require.Contains(t, string(raw), "scope: resourceGroup") + raw = []byte(strings.Replace(string(raw), "scope: resourceGroup", + "scope: az.resourceGroup('foundry-sub', 'foundry-rg')", 1)) + //nolint:gosec // G703: test-owned main.bicep under t.TempDir. + require.NoError(t, os.WriteFile(path, raw, 0o600)) + } + if strings.HasPrefix(scenario, "unknown Storage") { + edited = strings.Replace(edited, "param resourceTokenSalt string = ''", + "param resourceTokenSalt string = ''\nparam actualStorageName string = 'st${utcNow('MMddHHmmss')}'", 1) + edited = strings.Replace(edited, "var storageAccountName = 'editedstore'", + "var storageAccountName = actualStorageName", 1) + } + if scenario == "unknown Storage and known AKS failure" { + requests.clusterVersion = "1.30.9" + } + if scenario == "unknown workload name" || scenario == "cross subscription unknown workload" { + edited = strings.Replace(edited, "param resourceTokenSalt string = ''", + "param resourceTokenSalt string = ''\nparam actualWorkloadName string = 'mi${utcNow('MMddHHmmss')}'", 1) + edited = strings.Replace(edited, "var workloadIdentityName = 'edited-workload'", + "var workloadIdentityName = actualWorkloadName", 1) + } + if scenario == "actual cluster selection" { + requests.clusterID = offlineClusterID + "-edited" + requests.kubeletID = "22222222-2222-4222-8222-222222222222" + edited = strings.ReplaceAll(edited, "clusterResourceId: agentHosting.clusterResourceId", + "clusterResourceId: '"+requests.clusterID+"'") + } + if scenario == "no native pull assignments" { + path := filepath.Join(root, "modules", "acr.bicep") + //nolint:gosec // Embedded fixture copied to t.TempDir. + raw, err := os.ReadFile(path) + require.NoError(t, err) + lines := strings.Split(string(raw), "\n") + disabled := 0 + for i, line := range lines { + if strings.Contains(line, "'Microsoft.Authorization/roleAssignments@") { + equal := strings.Index(line, " = ") + require.Greater(t, equal, 0) + lines[i] = line[:equal] + " = if (false) {" + disabled++ + } + } + require.Equal(t, 2, disabled) + //nolint:gosec // G703: test-owned acr.bicep under t.TempDir. + require.NoError(t, os.WriteFile(path, []byte(strings.Join(lines, "\n")), 0o600)) + } + if scenario == "native registry without connection" { + path := filepath.Join(root, "modules", "acr.bicep") + //nolint:gosec // Embedded fixture copied to t.TempDir. + raw, err := os.ReadFile(path) + require.NoError(t, err) + text := string(raw) + start, end := strings.Index(text, "// Existing parent references"), strings.Index(text, "// Outputs") + require.Greater(t, start, 0) + require.Greater(t, end, start) + text = text[:start] + text[end:] + require.Contains(t, text, "output connectionName string = foundryAccount::project::acrConnection.name") + text = strings.Replace(text, "output connectionName string = foundryAccount::project::acrConnection.name", + "output connectionName string = ''", 1) + //nolint:gosec // G703: test-owned acr.bicep under t.TempDir. + require.NoError(t, os.WriteFile(path, []byte(text), 0o600)) + } + missingStorage := suppliedGroupID + "/providers/Microsoft.Storage/storageAccounts/missingstore" + if scenario == "known missing Storage reference" { + start := strings.Index(edited, "var storageAccountResourceId = ") + end := strings.Index(edited[start:], "\n\n") + require.Greater(t, start, 0) + require.Greater(t, end, 0) + edited = edited[:start] + "var storageAccountResourceId = '" + missingStorage + "'" + edited[start+end:] + } + runtimeNetwork := "\nresource runtimeNetworkConfig " + + "'Microsoft.Storage/storageAccounts@2023-05-01' existing = {\n" + + " name: 'runtime-network-config'\n}\n" + if unknownField == "networkInjections" || noNetworkSelection { + require.Contains(t, edited, "networkInjections: agentNetworkInjections") + value := "[]" + if unknownField != "" { + value = "any(runtimeNetworkConfig.properties)." + unknownField + edited += runtimeNetwork + } + edited = strings.Replace(edited, "networkInjections: agentNetworkInjections", + "networkInjections: "+value, 1) + } + if unknownField == "privateLinkServiceConnections" || unknownField == "privateDnsZoneConfigs" || + noNetworkSelection { + path := filepath.Join(root, "modules", "private-endpoint-dns.bicep") + //nolint:gosec // Embedded fixture copied to t.TempDir. + raw, err := os.ReadFile(path) + require.NoError(t, err) + text := string(raw) + if scenario == "unrelated network endpoint" { + require.Contains(t, text, "privateLinkServiceId: aiAccount.id") + text = strings.Replace(text, "privateLinkServiceId: aiAccount.id", + "privateLinkServiceId: '${aiAccount.id}-unrelated'", 1) } else { - _, err = p.Deploy(t.Context(), func(string) {}) + fields := []string{unknownField} + if noNetworkSelection { + fields = []string{"privateLinkServiceConnections", "privateDnsZoneConfigs"} + } + for _, field := range fields { + start := strings.Index(text, " "+field+": [") + require.Greater(t, start, 0) + end := strings.Index(text[start:], "\n ]") + require.Greater(t, end, 0) + end += start + len("\n ]") + value := "[]" + if unknownField != "" { + value = "any(runtimeNetworkConfig.properties)." + field + } + text = text[:start] + " " + field + ": " + value + text[end:] + } + if unknownField != "" { + text += runtimeNetwork + } } - require.ErrorContains(t, err, "InvalidTemplate") - require.ErrorContains(t, err, "offline ARM includeAcr type validation") - assert.Equal(t, map[string]any{"value": test.value}, fake.parameters["includeAcr"]) - assert.NotContains(t, fake.parameters, "kubeletObjectId") - expectedRequests := []string{} - if operation == "deploy" { - expectedRequests = append(expectedRequests, - "GET /subscriptions/target-sub/providers/Microsoft.Resources/deployments/"+p.deploymentName()) + //nolint:gosec // G703: test-owned endpoint template under t.TempDir. + require.NoError(t, os.WriteFile(path, []byte(text), 0o600)) + } + //nolint:gosec // G703: test-owned resources.bicep under t.TempDir. + require.NoError(t, os.WriteFile(resourcePath, []byte(edited), 0o600)) + main := filepath.Join(root, "main.bicep") + //nolint:gosec // Explicit existing offline Bicep tool; fixture paths are under t.TempDir. + command := exec.CommandContext(t.Context(), tool, "build", main, "--stdout") + var diagnostics bytes.Buffer + command.Stderr = &diagnostics + compiled, err := command.Output() + require.NoError(t, err, diagnostics.String()) + actual, err := unmarshalARMTemplate(string(compiled), main) + require.NoError(t, err) + parameters := map[string]any{ + "includeAcr": map[string]any{"value": true}, "resourceTokenSalt": map[string]any{"value": "edited"}, + "tags": map[string]any{"value": map[string]any{"team": "test"}}, + } + network := scenario == "selected existing network" || scenario == "new selected subnets" || + scenario == "external DNS" || scenario == "overlapping selected Node subnet" || + unknownField != "" || noNetworkSelection + if network { + requests.nodeSubnetID = offlineVNetID + "/subnets/nodes" + parameters["includeAcr"] = map[string]any{"value": false} + for key, value := range map[string]any{ + "enableNetworkIsolation": true, "vnetId": offlineVNetID, + "agentSubnetName": "agents", "peSubnetName": "pe", + } { + parameters[key] = map[string]any{"value": value} } - expectedRequests = append(expectedRequests, - method+" /subscriptions/target-sub/providers/Microsoft.Resources/deployments/"+ - p.deploymentName()+suffix) - require.Len(t, fake.requests, len(expectedRequests)+2) - assert.ElementsMatch(t, []string{ - "GET " + offlineClusterID, - "GET /subscriptions/target-sub/providers/Microsoft.Authorization/roleDefinitions", - }, fake.requests[:2]) - assert.Equal(t, expectedRequests, fake.requests[2:]) - }) - } + if scenario == "new selected subnets" { + parameters["createAgentSubnet"] = map[string]any{"value": true} + parameters["createPESubnet"] = map[string]any{"value": true} + } else if scenario == "external DNS" { + parameters["dnsZonesResourceGroup"] = map[string]any{"value": "dns-rg"} + parameters["dnsZonesSubscription"] = map[string]any{"value": "dns-sub"} + } else if scenario == "overlapping selected Node subnet" { + requests.nodePrefix = "172.16.0.0/24" + } + } + provider.onDiskSource = &templateSource{ + mode: templateModeBicep, sourcePath: main, armTemplate: actual, + parameters: parameters, + } + provider.bicepCliInstance = &stubCompiler{buildParam: func( + ctx context.Context, path string, env []string, + ) (bicep.BuildResult, error) { + //nolint:gosec // The provider's local expression probe uses the explicit offline tool. + command := exec.CommandContext(ctx, tool, "build-params", path, "--stdout") + command.Env = append(os.Environ(), env...) + raw, err := command.Output() + return bicep.BuildResult{Compiled: string(raw)}, err + }} + var progress strings.Builder + if scenario == "cross subscription Foundry" { + source, err := provider.resolveTemplate(t.Context(), func(string) {}) + require.NoError(t, err) + identified, err := provider.identifyBYOTemplatePlan(t.Context(), source) + require.NoError(t, err) + assert.Equal(t, subscription, identified.SubscriptionID) + assert.Equal(t, group, identified.ResourceGroup) + assert.Contains(t, identified.OwningSubscriptions, "target-sub") + assert.Contains(t, identified.OwningSubscriptions, subscription) + } + if network { + source, err := provider.resolveTemplate(t.Context(), func(string) {}) + require.NoError(t, err) + identified, err := provider.identifyBYOTemplatePlan(t.Context(), source) + require.NoError(t, err) + selection, err := byoDeploymentSelection(source) + require.NoError(t, err) + base := identified + base.Operations = nil + expected, err := onboarding.WithDeploymentOperations(base, selection) + require.NoError(t, err) + if unknownField == "" && !noNetworkSelection || unknownField == "privateLinkServiceConnections" { + assert.ElementsMatch(t, expected.Operations, identified.Operations, + "actual standard-network operations must match the reused review planner") + } + if unknownField == "networkInjections" || noNetworkSelection { + assert.Empty(t, identified.AgentSubnet.ID) + } else { + assert.Equal(t, expected.AgentSubnet.ID, identified.AgentSubnet.ID) + assert.Equal(t, expected.AgentSubnet.Supplied, identified.AgentSubnet.Supplied) + } + if unknownField != "" { + var warnings onboarding.Report + for _, check := range identified.SkippedChecks { + if check.ID == "P03" && strings.Contains(check.Missing, unknownField) { + assert.Equal(t, onboarding.Skipped, check.Status) + assert.False(t, check.Mandatory) + assert.Contains(t, check.Scope, "/providers/") + assert.Contains(t, check.Scope, " at ") + assert.NotEmpty(t, check.Warning) + warnings = append(warnings, check) + } + } + if unknownField == "privateDnsZoneConfigs" { + require.Len(t, warnings, 2) + assert.Equal(t, "Microsoft.Network/privateDnsZones/write", warnings[0].Expected) + assert.Equal(t, "Microsoft.Network/privateDnsZones/join/action", warnings[1].Expected) + } else { + require.Len(t, warnings, 1) + } + known := onboarding.Plan{} + for _, action := range []string{"Microsoft.Network/privateEndpoints/write", + "Microsoft.Network/privateEndpoints/privateDnsZoneGroups/write"} { + known.AddOperation("P03", action, suppliedGroupID) + } + assert.Subset(t, identified.Operations, known.Operations, + "unknown array selections must preserve independently known network writes") + } + } + _, err = provider.Deploy(t.Context(), func(message string) { progress.WriteString(message) }) + if scenario == "unknown Storage and known AKS failure" { + require.ErrorContains(t, err, "A01 FAIL") + assert.Zero(t, requests.writes) + } else if scenario == "known missing Storage reference" { + require.ErrorContains(t, err, "does not exist") + assert.Contains(t, requests.requests, "GET "+missingStorage) + assert.Zero(t, requests.writes) + assert.NotContains(t, progress.String(), "S01 SKIPPED") + } else if scenario == "overlapping selected Node subnet" { + require.ErrorContains(t, err, "N02 FAIL") + assert.Zero(t, requests.writes) + } else { + require.NoError(t, err, progress.String()) + assert.Equal(t, 1, requests.writes) + base := "/subscriptions/" + subscription + "/resourceGroups/" + group + "/providers/" + assert.Contains(t, requests.requests, + "GET "+base+"Microsoft.ManagedIdentity/userAssignedIdentities/edited-hosting") + if scenario != "unknown workload name" && scenario != "cross subscription unknown workload" { + assert.Contains(t, requests.requests, + "GET "+base+"Microsoft.ManagedIdentity/userAssignedIdentities/edited-workload") + } + if !network { + registryGroup := suppliedGroupID + if scenario == "cross subscription Foundry" { + registryGroup = "/subscriptions/" + subscription + "/resourceGroups/" + group + } + assert.Contains(t, requests.requests, + "GET "+registryGroup+"/providers/Microsoft.ContainerRegistry/registries/editedregistry") + assert.Contains(t, progress.String(), "P05 PASS") + if scenario == "no native pull assignments" { + assert.Contains(t, progress.String(), "P07 NOT_APPLICABLE") + } else { + assert.Contains(t, progress.String(), "P07 PASS") + } + } + assert.Contains(t, progress.String(), "P02 PASS") + } + if strings.HasPrefix(scenario, "unknown Storage") { + assert.Contains(t, progress.String(), "WARNING: P04 [Microsoft.Storage/storageAccounts at ") + assert.Contains(t, progress.String(), "utcNow") + assert.Contains(t, progress.String(), "P04 SKIPPED") + assert.NotContains(t, progress.String(), "P04 PASS") + assert.Contains(t, progress.String(), "P02 PASS") + } + if scenario == "unknown workload name" || scenario == "cross subscription unknown workload" { + assert.Contains(t, progress.String(), "P02 SKIPPED") + assert.Contains(t, progress.String(), "utcNow") + assert.Equal(t, map[string]any{"value": "/subscriptions/" + subscription + "/providers/" + + "Microsoft.Authorization/roleDefinitions/aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa"}, + requests.parameters["federatedIdentityRoleDefinitionId"]) + assert.Contains(t, progress.String(), "I02 PASS [/subscriptions/target-sub]") + assert.Contains(t, progress.String(), "P04 PASS") + } + if scenario == "actual cluster selection" { + assert.Contains(t, requests.requests, "GET "+requests.clusterID) + assert.NotContains(t, requests.requests, "GET "+offlineClusterID, + "the account's actual cluster controls AKS evidence, not the YAML selector") + assert.Equal(t, map[string]any{"value": requests.kubeletID}, requests.parameters["kubeletObjectId"]) + } + if network { + assert.NotContains(t, progress.String(), "P05 ") + assert.NotContains(t, progress.String(), "P07 ") + if noNetworkSelection { + assert.NotContains(t, progress.String(), "P03 ") + assert.Contains(t, progress.String(), "N02 NOT_APPLICABLE") + } else { + assert.Contains(t, progress.String(), "P03 PASS") + if unknownField != "" { + assert.Contains(t, progress.String(), "P03 SKIPPED") + assert.Contains(t, progress.String(), "WARNING: P03 [") + assert.Contains(t, progress.String(), unknownField) + for _, action := range []string{"Microsoft.Network/privateEndpoints/write", + "Microsoft.Network/privateEndpoints/privateDnsZoneGroups/write"} { + assert.Contains(t, progress.String(), "P03 PASS ["+suppliedGroupID+"]\n Expected: "+action) + } + } else { + assert.NotContains(t, progress.String(), "P03 SKIPPED") + } + } + if scenario == "new selected subnets" { + assert.Contains(t, progress.String(), "N02 NOT_APPLICABLE") + assert.NotContains(t, requests.requests, "GET "+offlineVNetID+"/subnets/agents") + } else if unknownField == "networkInjections" { + assert.Contains(t, progress.String(), "N02 SKIPPED") + } else if scenario != "overlapping selected Node subnet" && !noNetworkSelection { + assert.Contains(t, progress.String(), "N02 PASS") + } + } + assert.NotContains(t, requests.requests, "GET "+defaultStorageID, + "an edited or unknown name must not fall back to the standard naming formula") + assert.NotContains(t, requests.requests, "GET "+defaultWorkloadID) + }) } } @@ -618,7 +1079,8 @@ func TestNativeFoundryDeployKeepsStandardOutputsWithoutAKSReads(t *testing.T) { } func TestBYOPrepareOnlySuppliesDeclaredHostBindings(t *testing.T) { - p, fake := suppliedProvider(t, 7) + p, supplied, _ := suppliedProvider(t, 7) + fake := supplied.base source, err := p.resolveTemplate(t.Context(), func(string) {}) require.NoError(t, err) definitions, ok := source.armTemplate["parameters"].(map[string]any) @@ -628,12 +1090,16 @@ func TestBYOPrepareOnlySuppliesDeclaredHostBindings(t *testing.T) { } { delete(definitions, name) } - require.NoError(t, p.prepareBYO(t.Context(), source)) + require.NoError(t, p.prepareBYO(t.Context(), source, func(string) {})) assert.NotContains(t, source.parameters, "kubeletObjectId") assert.NotContains(t, source.parameters, "federatedIdentityRoleDefinitionId") assert.NotContains(t, source.parameters, "hostingPrincipalId") assert.NotContains(t, source.parameters, "workloadPrincipalId") - assert.Equal(t, []string{offlineClusterID}, fake.reads, "the actual kubelet identity is always checked") + for _, request := range fake.requests { + assert.True(t, strings.HasPrefix(request, "GET "), request) + } + assert.Zero(t, fake.writes) + assert.Zero(t, fake.previews) } func TestBYORejectsExternalRegistryBeforeAuthentication(t *testing.T) { @@ -653,98 +1119,12 @@ func TestBYORejectsExternalRegistryBeforeAuthentication(t *testing.T) { p.acrBinding = acrTestBinding() } p.credential = nil - require.ErrorIs(t, p.prepareBYO(t.Context(), source), synthesis.ErrACRReuseAgentHosting) + require.ErrorIs(t, p.prepareBYO(t.Context(), source, func(string) {}), synthesis.ErrACRReuseAgentHosting) assert.Empty(t, fake.requests) }) } } -func TestBYOPrepareValidatesKubeletIdentityRegardlessOfACR(t *testing.T) { - for _, includeAcr := range []bool{false, true} { - for _, supplied := range []bool{false, true} { - for _, test := range []struct { - name string - properties map[string]any - expected string - }{ - {"missing properties", nil, "requires a kubelet principal object ID"}, - {"missing kubelet identity", map[string]any{}, "requires a kubelet principal object ID"}, - {"missing object ID", map[string]any{"identityProfile": map[string]any{ - "kubeletidentity": map[string]any{}, - }}, "requires a kubelet principal object ID"}, - {"empty object ID", map[string]any{"identityProfile": map[string]any{ - "kubeletidentity": map[string]any{"objectId": ""}, - }}, "must be a GUID"}, - {"invalid object ID", map[string]any{"identityProfile": map[string]any{ - "kubeletidentity": map[string]any{"objectId": "not-a-guid"}, - }}, "must be a GUID"}, - } { - t.Run(fmt.Sprintf("includeAcr=%t/supplied=%t/%s", includeAcr, supplied, test.name), func(t *testing.T) { - p, fake := offlineBYOProvider(t) - fake.clusterProperties = test.properties - source, err := p.resolveTemplate(t.Context(), func(string) {}) - require.NoError(t, err) - source.parameters["includeAcr"] = map[string]any{"value": includeAcr} - if supplied { - source.parameters["kubeletObjectId"] = map[string]any{"value": suppliedHostingPrincipal} - } - require.ErrorContains(t, p.prepareBYO(t.Context(), source), test.expected) - assert.Contains(t, fake.requests, "GET "+offlineClusterID) - assert.Subset(t, []string{ - "GET " + offlineClusterID, - "GET /subscriptions/target-sub/providers/Microsoft.Authorization/roleDefinitions", - }, fake.requests) - }) - } - } - } -} - -func TestBYOPreparePreservesSuppliedKubeletRegardlessOfACR(t *testing.T) { - for _, includeAcr := range []bool{false, true} { - for _, test := range []struct { - name string - value any - duplicate bool - }{ - {"empty", "", false}, - {"whitespace", " ", false}, - {"non-string", 1, false}, - {"null", nil, false}, - {"reference", map[string]any{"reference": map[string]any{"secretName": "kubelet"}}, false}, - {"duplicate", offlineKubeletObjectID, true}, - {"different", "22222222-2222-4222-8222-222222222222", false}, - {"matching", strings.ToUpper(offlineKubeletObjectID), false}, - } { - t.Run(fmt.Sprintf("includeAcr=%t/%s", includeAcr, test.name), func(t *testing.T) { - p, fake := offlineBYOProvider(t) - source, err := p.resolveTemplate(t.Context(), func(string) {}) - require.NoError(t, err) - delete(source.parameters, "includeAcr") - source.parameters["INCLUDEACR"] = map[string]any{"value": includeAcr} - parameter := any(map[string]any{"value": test.value}) - if test.name == "reference" { - parameter = test.value - } - source.parameters["kubeletObjectId"] = parameter - if test.duplicate { - source.parameters["KUBELETOBJECTID"] = map[string]any{"value": test.value} - } - require.NoError(t, p.prepareBYO(t.Context(), source)) - assert.ElementsMatch(t, []string{ - "GET " + offlineClusterID, - "GET /subscriptions/target-sub/providers/Microsoft.Authorization/roleDefinitions", - }, fake.requests) - assert.Equal(t, parameter, source.parameters["kubeletObjectId"]) - if test.duplicate { - assert.Equal(t, map[string]any{"value": test.value}, source.parameters["KUBELETOBJECTID"]) - } - assert.Equal(t, map[string]any{"value": includeAcr}, source.parameters["INCLUDEACR"]) - }) - } - } -} - func TestBYORejectsLocalConflictsBeforeAuthentication(t *testing.T) { for _, test := range []struct { key, expected string @@ -773,8 +1153,70 @@ func TestBYORejectsLocalConflictsBeforeAuthentication(t *testing.T) { source, err := p.resolveTemplate(t.Context(), func(string) {}) require.NoError(t, err) source.parameters[test.key] = map[string]any{"value": test.value} - require.ErrorContains(t, p.prepareBYO(t.Context(), source), test.expected) + require.ErrorContains(t, p.prepareBYO(t.Context(), source, func(string) {}), test.expected) assert.Empty(t, fake.requests) }) } + +} + +func byoDeploymentSelection(source *templateSource) (onboarding.DeploymentSelection, error) { + selection := onboarding.DeploymentSelection{} + var network onboarding.LegacyNetwork + for key, out := range map[string]*bool{ + "enableNetworkIsolation": &network.Enabled, "createAgentSubnet": &network.CreateAgentSubnet, + "createPESubnet": &network.CreatePESubnet, "useManagedEgress": &network.ManagedEgress, + } { + if raw := byoParameterValue(source, key); raw != nil { + value, ok := raw.(bool) + if !ok { + return selection, fmt.Errorf("effective %s must be a boolean", key) + } + *out = value + } + } + for key, out := range map[string]*string{ + "vnetId": &network.VNetID, "agentSubnetName": &network.AgentSubnetName, "peSubnetName": &network.PESubnetName, + "peSubnetPrefix": &network.PESubnetPrefix, + "managedIsolationMode": &network.IsolationMode, "dnsZonesResourceGroup": &network.DNSResourceGroup, + "dnsZonesSubscription": &network.DNSSubscription, + } { + if raw := byoParameterValue(source, key); raw != nil { + value, ok := raw.(string) + if !ok { + return selection, fmt.Errorf("effective %s must be a string", key) + } + *out = value + } + } + if network.Enabled { + selection.Network = &network + } + return selection, nil +} +func TestBYOInitializeRejectsInvalidInputsBeforePromptsOrPersistence(t *testing.T) { + for _, onDisk := range []bool{false, true} { + root := t.TempDir() + config := "services:\n project:\n host: azure.ai.project\n" + + " agentHosting: {clusterResourceId: " + offlineClusterID + ", storageAccountResourceId: []}\n" + path := filepath.Join(root, "azure.yaml") + require.NoError(t, os.WriteFile(path, []byte(config), 0o600)) + if onDisk { + require.NoError(t, os.Mkdir(filepath.Join(root, "infra"), 0o750)) + require.NoError(t, os.WriteFile(filepath.Join(root, "infra", "main.bicep"), []byte("// generated"), 0o600)) + } + env := &resolveEnvStubEnvServer{envName: "test"} + prompt := &resolveEnvStubPromptServer{} + p := &FoundryProvisioningProvider{azdClient: newResolveEnvTestClient(t, env, prompt)} + err := p.Initialize(t.Context(), root, &azdext.ProvisioningOptions{Provider: FoundryProviderName}) + require.ErrorContains(t, err, "storageAccountResourceId: expected a nonempty resource ID") + assert.Empty(t, env.set) + assert.Zero(t, prompt.subscriptionN) + assert.Zero(t, prompt.locationN) + assert.Nil(t, p.credential) + //nolint:gosec // Test-owned azure.yaml under t.TempDir. + raw, err := os.ReadFile(path) + require.NoError(t, err) + assert.Equal(t, config, string(raw)) + } } diff --git a/cli/azd/extensions/azure.ai.projects/internal/provisioning/byo_plan_bindings.go b/cli/azd/extensions/azure.ai.projects/internal/provisioning/byo_plan_bindings.go new file mode 100644 index 00000000000..c3205153e47 --- /dev/null +++ b/cli/azd/extensions/azure.ai.projects/internal/provisioning/byo_plan_bindings.go @@ -0,0 +1,569 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +package provisioning + +import ( + "errors" + "fmt" + "slices" + "strings" + + "azure.ai.projects/internal/onboarding" + + "github.com/Azure/azure-sdk-for-go/sdk/azcore/arm" +) + +func (r *templatePlanResolver) bindResources(plan *onboarding.Plan, root *planEnvironment) { + type accountSelection struct { + node *planResource + config any + } + var accounts []accountSelection + var unresolved string + for _, node := range r.nodes { + if node.inactive || !strings.EqualFold(node.kind, "Microsoft.CognitiveServices/accounts") { + continue + } + properties := node.env.value(node.raw["properties"]) + configs := planField(properties, "agentHostingConfigurations") + if values, ok := configs.([]any); ok { + for _, config := range values { + id, missing := planString(planField(config, "clusterResourceId")) + typeValue := planField(config, "hostingType") + hostingType, typeMissing := planString(typeValue) + if missing == "" && id != "" { + if _, unknown := typeValue.(planUnknown); !unknown && + (typeMissing != "" || hostingType != "ManagedCluster") { + r.err = errors.Join(r.err, fmt.Errorf( + "I03: actual Foundry hostingType must be ManagedCluster at %s", node.path)) + continue + } + } + if missing == "" && id != "" || strings.EqualFold(hostingType, "ManagedCluster") { + accounts = append(accounts, accountSelection{node, config}) + } + } + } else if unknown, ok := configs.(planUnknown); ok { + unresolved = node.path + ": " + unknown.reason + } + } + if len(accounts) > 1 { + outputs := root.value(root.template["outputs"]) + accountName, _ := planString(planField(planField(outputs, "AZURE_AI_ACCOUNT_NAME"), "value")) + matching := slices.DeleteFunc(slices.Clone(accounts), func(account accountSelection) bool { + name, missing := planString(account.node.env.value(account.node.raw["name"])) + return missing != "" || !strings.EqualFold(name, accountName) + }) + if len(matching) == 1 { + accounts = matching + } + } + if len(accounts) != 1 { + reason := "no unique local Foundry account/agentHostingConfigurations association" + if unresolved != "" { + reason += ": " + unresolved + } + for _, entry := range []struct { + kind string + out *onboarding.Resource + }{ + {"Microsoft.ContainerService/managedClusters", &plan.Cluster}, + {"Microsoft.Storage/storageAccounts", &plan.Storage}, + {"Microsoft.ManagedIdentity/userAssignedIdentities (hosting)", &plan.Hosting}, + {"Microsoft.ManagedIdentity/userAssignedIdentities (workload)", &plan.Workload}, + } { + *entry.out = onboarding.Resource{Origin: entry.kind + " at " + root.path, Missing: reason} + } + plan.SkippedChecks = append(plan.SkippedChecks, + onboarding.SkippedCheck("N02", root.path, "selected Agent subnet configuration", reason), + onboarding.SkippedCheck("P03", root.path, "selected network operations", reason), + onboarding.SkippedCheck("P05", root.path, "native registry creation", reason), + onboarding.SkippedCheck("P07", root.path, "kubelet pull grant authority", reason)) + return + } + account := accounts[0] + r.account = account.node + for _, entry := range []struct { + field, kind string + out *onboarding.Resource + }{ + {"clusterResourceId", "Microsoft.ContainerService/managedClusters", &plan.Cluster}, + {"storageAccountResourceId", "Microsoft.Storage/storageAccounts", &plan.Storage}, + {"hostingManagementIdentityResourceId", "Microsoft.ManagedIdentity/userAssignedIdentities", &plan.Hosting}, + {"workloadIdentityResourceId", "Microsoft.ManagedIdentity/userAssignedIdentities", &plan.Workload}, + } { + value := planField(account.config, entry.field) + if config, known := account.config.(map[string]any); known { + present := false + for name := range config { + present = present || strings.EqualFold(name, entry.field) + } + if !present { + r.err = errors.Join(r.err, fmt.Errorf("I02: actual Foundry hosting configuration is missing %s at %s", + entry.field, account.node.path)) + continue + } + } + *entry.out = r.resource(value, entry.kind, + account.node.path+"/properties.agentHostingConfigurations."+entry.field) + if account.node.missing != "" { + entry.out.Missing = strings.Trim(entry.out.Missing+"; Foundry account selection: "+ + account.node.missing, "; ") + } + } + r.bindRegistry(plan, account.node) +} + +func (r *templatePlanResolver) bindRegistry(plan *onboarding.Plan, account *planResource) { + accountID, accountMissing := account.id() + var selections []onboarding.Resource + for _, node := range r.nodes { + if node.inactive || + !strings.EqualFold(node.kind, "Microsoft.CognitiveServices/accounts/projects/connections") { + continue + } + properties := node.env.value(node.raw["properties"]) + id, missing := node.id() + if missing == "" && accountMissing == "" && + !strings.HasPrefix(strings.ToLower(id), strings.ToLower(accountID)+"/projects/") { + continue + } + category, categoryMissing := planString(planField(properties, "category")) + if categoryMissing != "" { + plan.SkippedChecks = append(plan.SkippedChecks, + onboarding.SkippedCheck("P05", node.path, "native registry selection", categoryMissing), + onboarding.SkippedCheck("P07", node.path, "native pull grant selection", categoryMissing)) + continue + } + if category != "ContainerRegistry" { + continue + } + credentials := planField(properties, "credentials") + registry := r.resource(planField(credentials, "resourceId"), "Microsoft.ContainerRegistry/registries", + node.path+"/properties.credentials.resourceId") + if missing != "" || accountMissing != "" { + registry.LifecycleMissing = "registry-to-Foundry association cannot be resolved" + } else if node.missing != "" { + registry.LifecycleMissing = node.path + ": " + node.missing + } + if !slices.ContainsFunc(selections, func(existing onboarding.Resource) bool { + return registry.ID != "" && strings.EqualFold(existing.ID, registry.ID) + }) { + selections = append(selections, registry) + } + } + if len(selections) == 0 { + for _, node := range r.nodes { + if node.inactive || !strings.EqualFold(node.kind, "Microsoft.ContainerRegistry/registries") { + continue + } + id, missing := node.id() + var value any = id + if missing != "" { + value = planUnknown{node.path + ": " + missing} + } + registry := r.resource(value, node.kind, node.path) + registry.LifecycleMissing = strings.Trim(registry.LifecycleMissing+"; "+node.missing, "; ") + selections = append(selections, registry) + } + } + if len(selections) == 1 { + plan.Registry = selections[0] + plan.RegistryGrantPlanned = new(false) + if plan.Registry.Supplied && plan.Registry.LifecycleMissing == "" { + reason := "the compiled template references an external registry; only native registry creation is covered" + plan.SkippedChecks = append(plan.SkippedChecks, + onboarding.SkippedCheck("P05", plan.Registry.ID, "native registry creation", reason), + onboarding.SkippedCheck("P07", plan.Registry.ID, "native kubelet pull grant authority", reason)) + plan.Registry = onboarding.Resource{} + } + } else if len(selections) > 1 { + reason := "multiple registry connections or creations prevent a unique native registry selection" + plan.SkippedChecks = append(plan.SkippedChecks, + onboarding.SkippedCheck("P05", account.path, "native registry creation", reason), + onboarding.SkippedCheck("P07", account.path, "kubelet pull grant authority", reason)) + } else if len(r.unresolved) != 0 { + for _, missing := range r.unresolved { + plan.SkippedChecks = append(plan.SkippedChecks, + onboarding.SkippedCheck("P05", account.path, "native registry selection", missing), + onboarding.SkippedCheck("P07", account.path, "native pull grant selection", missing)) + } + } +} + +func (n *planResource) assignmentScope() (string, string) { + if raw, present := n.raw["scope"]; present { + scope, missing := planString(n.env.value(raw)) + if missing != "" { + return "", missing + } + if strings.HasPrefix(scope, "/") { + return scope, "" + } + if resource := n.env.resources[scope]; resource != nil { + return resource.id() + } + if strings.Contains(scope, "/") && n.env.scope.groupARMID != "" { + return n.env.scope.groupARMID + "/providers/" + scope, "" + } + return "", "role-assignment scope is not a resolved resource ID or local symbol" + } + if n.env.scope.groupARMID != "" { + return n.env.scope.groupARMID, "" + } + if n.env.scope.group == "" && n.env.scope.missing == "" && n.env.scope.subscriptionARMID != "" { + return n.env.scope.subscriptionARMID, "" + } + return "", "role-assignment deployment scope is not resolved" +} + +func (r *templatePlanResolver) bindGrants(plan *onboarding.Plan) { + for _, node := range r.nodes { + if node.inactive || !strings.EqualFold(node.kind, "Microsoft.Authorization/roleAssignments") { + continue + } + scope, missing := node.assignmentScope() + properties := node.env.value(node.raw["properties"]) + principal, symbolic := planField(properties, "principalId").(planPrincipal) + role, _ := planString(planField(properties, "roleDefinitionId")) + roleID, _ := arm.ParseResourceID(role) + checkID := "" + if roleID != nil { + checkID = onboarding.GrantCheckIDForRole(roleID.Name) + } + associated := symbolic && (plan.Hosting.ID != "" && strings.EqualFold(principal.resourceID, plan.Hosting.ID) || + plan.Workload.ID != "" && strings.EqualFold(principal.resourceID, plan.Workload.ID) || + strings.EqualFold(principal.binding, "kubeletObjectId")) + if symbolic && strings.EqualFold(principal.binding, "kubeletObjectId") { + checkID = "P07" + } + var target onboarding.Resource + for _, entry := range []struct { + id string + resource onboarding.Resource + }{ + {"G01", plan.Cluster}, {"G02", plan.Workload}, {"G03", plan.Storage}, {"P07", plan.Registry}, + } { + if entry.resource.ID == "" || !strings.EqualFold(scope, entry.resource.ID) { + continue + } + associated, checkID, target = true, entry.id, entry.resource + break + } + if !associated && node.missing == "" && missing == "" { + continue + } + if checkID == "" && symbolic && plan.Hosting.ID != "" && + strings.EqualFold(principal.resourceID, plan.Hosting.ID) { + checkID = "G02" + } + if checkID == "" { + checkID = "authored grant" + } + if target.ID == "" { + for _, entry := range []struct { + id string + resource onboarding.Resource + }{{"G01", plan.Cluster}, {"G02", plan.Workload}, {"G03", plan.Storage}, {"P07", plan.Registry}} { + if entry.id == checkID { + target = entry.resource + break + } + } + } + deploymentMissing := strings.Trim(node.missing+"; "+node.env.scope.missing, "; ") + deploymentScope := node.env.scope.groupARMID + if deploymentScope == "" && node.env.scope.group == "" && node.env.scope.missing == "" { + deploymentScope = node.env.scope.subscriptionARMID + } + if deploymentScope == "" && deploymentMissing == "" { + deploymentMissing = "actual nested grant deployment scope is not resolved" + } + grant := onboarding.PlannedGrant{ + CheckID: checkID, Target: target, DeploymentScope: deploymentScope, + Missing: strings.Trim(node.missing+"; "+missing, "; "), DeploymentMissing: deploymentMissing, + } + if missing == "" && !strings.EqualFold(scope, target.ID) { + parsed, err := arm.ParseResourceID(scope) + if err != nil || parsed.SubscriptionID == "" || strings.ContainsAny(scope, "?#%\\@:\x00 \t\r\n") { + r.err = errors.Join(r.err, fmt.Errorf("P06: actual role-assignment scope is not a valid ARM ID")) + continue + } + switch strings.ToLower(parsed.ResourceType.String()) { + case "microsoft.resources/subscriptions", "microsoft.resources/resourcegroups": + current := r.scope(parsed.SubscriptionID, parsed.ResourceGroupName) + grant.AuthorityScope = current.subscriptionARMID + if current.groupExists { + grant.AuthorityScope = current.groupARMID + } + default: + grant.Target = r.resource(scope, parsed.ResourceType.String(), node.path+"/scope") + if !grant.Target.Supplied && grant.Target.ID != plan.Cluster.ID && + grant.Target.ID != plan.Storage.ID && grant.Target.ID != plan.Workload.ID && + grant.Target.ID != plan.Registry.ID { + grant.Missing = "current state of an additional authored grant target is not available to this check" + } + } + } + if checkID == "P07" { + if grant.Missing != "" { + plan.RegistryGrantMissing = grant.Missing + } else { + plan.RegistryGrantPlanned = new(true) + } + if deploymentMissing != "" || grant.DeploymentScope == "" { + plan.SkippedChecks = append(plan.SkippedChecks, + onboarding.SkippedCheck("P09", node.kind+" at "+node.path, + "native registry grant deployment permissions", deploymentMissing)) + } else if plan.TargetScopeMissing != "" { + plan.SkippedChecks = append(plan.SkippedChecks, + onboarding.SkippedCheck("P09", grant.DeploymentScope, + "cross-resource-group native registry grant deployment permissions", plan.TargetScopeMissing)) + } else { + targetGroup := "/subscriptions/" + plan.SubscriptionID + "/resourceGroups/" + plan.ResourceGroup + if !strings.EqualFold(grant.DeploymentScope, targetGroup) { + for _, action := range []string{"write", "read", "operations/read"} { + plan.AddOperation("P09", "Microsoft.Resources/deployments/"+action, grant.DeploymentScope) + } + } + } + continue + } + if deploymentMissing != "" || grant.DeploymentScope == "" { + plan.SkippedChecks = append(plan.SkippedChecks, + onboarding.SkippedCheck("P09", node.kind+" at "+node.path, + "nested grant deployment permissions", deploymentMissing)) + } else if plan.TargetScopeMissing != "" { + plan.SkippedChecks = append(plan.SkippedChecks, + onboarding.SkippedCheck("P09", grant.DeploymentScope, + "cross-resource-group grant deployment permissions", plan.TargetScopeMissing)) + } + plan.Grants = append(plan.Grants, grant) + } +} + +func (r *templatePlanResolver) ancestor(node *planResource) string { + if node.env.scope.groupExists { + return node.env.scope.groupARMID + } + return node.env.scope.subscriptionARMID +} + +func (r *templatePlanResolver) networkOperation( + plan *onboarding.Plan, node *planResource, action, scope, missing string, +) { + if missing == "" { + missing = node.missing + } + if missing == "" && scope == "" { + missing = "actual network deployment scope is not resolved" + } + if missing != "" { + diagnostic := node.kind + " at " + node.path + if id, reason := node.id(); reason == "" && id != "" { + diagnostic = id + " at " + node.path + } + plan.SkippedChecks = append(plan.SkippedChecks, + onboarding.SkippedCheck("P03", diagnostic, action, missing)) + return + } + plan.AddOperation("P03", action, scope) +} + +func (r *templatePlanResolver) selectedSubnet( + plan *onboarding.Plan, node *planResource, value any, agent bool, +) onboarding.Resource { + resource := r.resource(value, "Microsoft.Network/virtualNetworks/subnets", node.path) + if node.missing != "" { + resource.Missing = strings.Trim(resource.Missing+"; selected network condition: "+node.missing, "; ") + } + if resource.Missing != "" { + id := "P03" + if agent { + id = "N02" + } + plan.SkippedChecks = append(plan.SkippedChecks, + onboarding.SkippedCheck(id, resource.Origin, "selected subnet configuration", resource.Missing)) + if parent, err := arm.ParseResourceID(resource.Parent); err == nil && + strings.EqualFold(parent.ResourceType.String(), "Microsoft.Network/virtualNetworks") { + plan.AddOperation("P03", "Microsoft.Network/virtualNetworks/read", resource.Parent) + } + plan.SkippedChecks = append(plan.SkippedChecks, onboarding.SkippedCheck("P03", resource.Origin, + "selected subnet creation/join permissions", resource.Missing)) + return resource + } + subnet, err := arm.ParseResourceID(resource.ID) + if err != nil || !strings.EqualFold(subnet.ResourceType.String(), "Microsoft.Network/virtualNetworks/subnets") { + r.err = errors.Join(r.err, fmt.Errorf("N02/P03: actual selected subnet ID is malformed")) + return resource + } + vnet := subnet.Parent.String() + plan.AddOperation("P03", "Microsoft.Network/virtualNetworks/read", vnet) + joinScope := resource.ID + if !resource.Supplied { + joinScope = vnet + if resource.LifecycleMissing == "" { + plan.AddOperation("P03", "Microsoft.Network/virtualNetworks/subnets/write", vnet) + } + } + r.networkOperation(plan, node, "Microsoft.Network/virtualNetworks/subnets/join/action", + joinScope, resource.LifecycleMissing) + return resource +} + +func (r *templatePlanResolver) bindNetwork(plan *onboarding.Plan) { + accountID := "" + vnets := map[string]bool{} + var privateEndpoints []string + optionalField := func(properties any, key string) any { + if properties == nil { + return nil + } + if object, ok := properties.(map[string]any); ok { + for name, value := range object { + if strings.EqualFold(name, key) { + return value + } + } + return nil + } + return planField(properties, key) + } + for _, node := range []*planResource{r.account} { + if node == nil || node.inactive { + continue + } + properties := node.env.value(node.raw["properties"]) + accountID, _ = node.id() + injections := optionalField(properties, "networkInjections") + if injections == nil { + continue + } + if values, ok := injections.([]any); ok { + for _, injection := range values { + scenario, scenarioMissing := planString(planField(injection, "scenario")) + managed, missing := planBool(planField(injection, "useMicrosoftManagedNetwork")) + if scenarioMissing != "" || scenario == "agent" && missing != "" { + reason := strings.Trim(scenarioMissing+"; "+missing, "; ") + plan.AgentSubnet = onboarding.Resource{Origin: node.path + "/networkInjections", Missing: reason} + plan.SkippedChecks = append(plan.SkippedChecks, onboarding.SkippedCheck("P03", + plan.AgentSubnet.Origin, "Agent subnet operation selection", reason)) + continue + } + if scenario != "agent" || managed { + continue + } + plan.AgentSubnet = r.selectedSubnet(plan, node, planField(injection, "subnetArmId"), true) + if id, err := arm.ParseResourceID(plan.AgentSubnet.ID); err == nil { + vnets[strings.ToLower(id.Parent.String())] = true + } + } + } else if unknown, ok := injections.(planUnknown); ok { + reason := "networkInjections cannot be resolved: " + unknown.reason + plan.AgentSubnet = onboarding.Resource{Origin: node.path + "/networkInjections", Missing: reason} + r.networkOperation(plan, node, "Agent subnet read/create/join operation selection", "", reason) + } + } + for _, node := range r.nodes { + if node.inactive || !strings.EqualFold(node.kind, "Microsoft.Network/privateEndpoints") { + continue + } + properties := node.env.value(node.raw["properties"]) + connectionList := optionalField(properties, "privateLinkServiceConnections") + connections, _ := connectionList.([]any) + var associationMissing string + if unknown, ok := connectionList.(planUnknown); ok { + associationMissing = unknown.reason + } + if len(connections) == 0 && associationMissing == "" { + continue + } + associated := slices.ContainsFunc(connections, func(connection any) bool { + target, missing := planString(planField(planField(connection, "properties"), "privateLinkServiceId")) + if missing != "" { + associationMissing = missing + } + return accountID != "" && strings.EqualFold(target, accountID) + }) + if !associated { + if accountID != "" && associationMissing == "" { + continue + } + reason := "privateLinkServiceConnections cannot resolve the private endpoint's Foundry account association" + if accountID == "" { + reason += ": actual Foundry account ID is not resolved" + } + if associationMissing != "" { + reason += ": " + associationMissing + } + r.networkOperation(plan, node, "private endpoint Foundry association", "", reason) + } + r.networkOperation(plan, node, "Microsoft.Network/privateEndpoints/write", r.ancestor(node), "") + subnet := r.selectedSubnet(plan, node, planField(planField(properties, "subnet"), "id"), false) + if id, err := arm.ParseResourceID(subnet.ID); err == nil { + vnets[strings.ToLower(id.Parent.String())] = true + } + if id, missing := node.id(); missing == "" { + privateEndpoints = append(privateEndpoints, id) + } + } + for _, node := range r.nodes { + if node.inactive { + continue + } + switch strings.ToLower(node.kind) { + case "microsoft.network/privateendpoints/privatednszonegroups": + id, missing := node.id() + parsed, err := arm.ParseResourceID(id) + if missing != "" || err != nil { + r.networkOperation(plan, node, "Microsoft.Network/privateEndpoints/privateDnsZoneGroups/write", "", + "private-DNS group/endpoint association is not resolved") + continue + } + if !slices.ContainsFunc(privateEndpoints, func(endpoint string) bool { + return strings.EqualFold(endpoint, parsed.Parent.String()) + }) { + continue + } + r.networkOperation(plan, node, "Microsoft.Network/privateEndpoints/privateDnsZoneGroups/write", + r.ancestor(node), "") + configList := optionalField(node.env.value(node.raw["properties"]), "privateDnsZoneConfigs") + if unknown, ok := configList.(planUnknown); ok { + reason := "privateDnsZoneConfigs cannot resolve the selected DNS zones: " + unknown.reason + for _, action := range []string{"write", "join/action"} { + r.networkOperation(plan, node, "Microsoft.Network/privateDnsZones/"+action, "", reason) + } + continue + } + configs, _ := configList.([]any) + for _, config := range configs { + zone := r.resource(planField(planField(config, "properties"), "privateDnsZoneId"), + "Microsoft.Network/privateDnsZones", node.path+"/privateDnsZoneConfigs") + joinScope := zone.ID + if !zone.Supplied { + joinScope = zone.Parent + r.networkOperation(plan, node, "Microsoft.Network/privateDnsZones/write", + zone.Parent, zone.Missing+zone.LifecycleMissing) + } + r.networkOperation(plan, node, "Microsoft.Network/privateDnsZones/join/action", + joinScope, zone.Missing+zone.LifecycleMissing) + if parsed, err := arm.ParseResourceID(zone.ID); err == nil { + plan.OwningSubscriptions = append(plan.OwningSubscriptions, parsed.SubscriptionID) + } + } + case "microsoft.network/privatednszones/virtualnetworklinks": + vnet, missing := planString(planField(planField(node.env.value(node.raw["properties"]), + "virtualNetwork"), "id")) + if missing != "" { + r.networkOperation(plan, node, "Microsoft.Network/privateDnsZones/virtualNetworkLinks/write", "", missing) + continue + } + if vnets[strings.ToLower(vnet)] { + r.networkOperation(plan, node, "Microsoft.Network/privateDnsZones/virtualNetworkLinks/write", + r.ancestor(node), "") + r.networkOperation(plan, node, "Microsoft.Network/virtualNetworks/join/action", vnet, "") + } + } + } +} diff --git a/cli/azd/extensions/azure.ai.projects/internal/provisioning/byo_plan_expression.go b/cli/azd/extensions/azure.ai.projects/internal/provisioning/byo_plan_expression.go new file mode 100644 index 00000000000..2063f22e7c5 --- /dev/null +++ b/cli/azd/extensions/azure.ai.projects/internal/provisioning/byo_plan_expression.go @@ -0,0 +1,745 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +package provisioning + +import ( + "encoding/json" + "fmt" + "maps" + "math" + "slices" + "strconv" + "strings" + "unicode" + + "github.com/Azure/azure-sdk-for-go/sdk/azcore/arm" +) + +// This adapter evaluates only the local expressions needed to identify a deployment plan. +// Runtime reference properties and unsupported ARM constructs remain explicit unknown values. +type planUnknown struct{ reason string } + +type planPartialID struct { + kind, scope, missing string + components []any +} + +type planPrincipal struct { + resourceID string + binding string +} + +type planExpression struct { + name string + literal any + args []planExpression + selector *planExpression + base *planExpression +} + +type planExpressionParser struct { + text string + pos int +} + +func (p *planExpressionParser) space() { + for p.pos < len(p.text) && unicode.IsSpace(rune(p.text[p.pos])) { + p.pos++ + } +} + +func (p *planExpressionParser) consume(c byte) bool { + p.space() + if p.pos < len(p.text) && p.text[p.pos] == c { + p.pos++ + return true + } + return false +} + +func (p *planExpressionParser) identifier() string { + p.space() + start := p.pos + for p.pos < len(p.text) { + c := rune(p.text[p.pos]) + if !unicode.IsLetter(c) && !unicode.IsDigit(c) && c != '_' && c != '$' && c != '#' { + break + } + p.pos++ + } + return p.text[start:p.pos] +} + +func (p *planExpressionParser) expression(depth int) (planExpression, error) { + if depth > 64 { + return planExpression{}, fmt.Errorf("ARM expression exceeds the local plan adapter's nesting limit") + } + p.space() + var result planExpression + if p.consume('\'') { + var literal strings.Builder + closed := false + for p.pos < len(p.text) { + c := p.text[p.pos] + p.pos++ + if c == '\'' { + if p.pos < len(p.text) && p.text[p.pos] == '\'' { + p.pos++ + literal.WriteByte('\'') + continue + } + closed = true + break + } + literal.WriteByte(c) + } + if !closed { + return result, fmt.Errorf("ARM expression has an unterminated string") + } + result.literal = literal.String() + } else if p.pos < len(p.text) && (p.text[p.pos] == '-' || unicode.IsDigit(rune(p.text[p.pos]))) { + start := p.pos + for p.pos < len(p.text) && strings.ContainsRune("-0123456789.", rune(p.text[p.pos])) { + p.pos++ + } + value, err := strconv.ParseFloat(p.text[start:p.pos], 64) + if err != nil { + return result, fmt.Errorf("ARM expression has an unsupported numeric literal") + } + result.literal = value + } else { + result.name = p.identifier() + if result.name == "" || !p.consume('(') { + return result, fmt.Errorf("ARM expression syntax is not supported by the local plan adapter") + } + if !p.consume(')') { + for { + arg, err := p.expression(depth + 1) + if err != nil { + return result, err + } + result.args = append(result.args, arg) + if p.consume(')') { + break + } + if !p.consume(',') { + return result, fmt.Errorf("ARM expression has an unsupported argument list") + } + } + } + } + for { + switch { + case p.consume('.'): + name := p.identifier() + if name == "" { + return result, fmt.Errorf("ARM expression property selection is incomplete") + } + base := result + result = planExpression{base: &base, selector: &planExpression{literal: name}} + case p.consume('['): + index, err := p.expression(depth + 1) + if err != nil || !p.consume(']') { + return result, fmt.Errorf("ARM expression index selection is unsupported") + } + base := result + result = planExpression{base: &base, selector: &index} + default: + return result, nil + } + } +} + +type planScope struct { + subscriptionID, subscriptionARMID, group, groupARMID, location string + groupExists bool + missing string +} + +type planEnvironment struct { + template map[string]any + parameters map[string]func() any + scope planScope + path string + deployment any + resources map[string]*planResource + variables map[string]any + active map[string]bool + copyIndex int + function func(string, []string) any + walked bool +} + +func planField(value any, key string) any { + if unknown, ok := value.(planUnknown); ok { + return unknown + } + object, ok := value.(map[string]any) + if !ok { + return planUnknown{"expected an object when selecting " + key} + } + for name, field := range object { + if strings.EqualFold(name, key) { + return field + } + } + return planUnknown{"object has no " + key + " property"} +} + +func planString(value any) (string, string) { + if unknown, ok := value.(planUnknown); ok { + return "", unknown.reason + } + if result, ok := value.(string); ok { + return result, "" + } + if partial, ok := value.(planPartialID); ok { + return "", partial.missing + } + return "", "value is not a locally resolved string" +} + +func planBool(value any) (bool, string) { + if unknown, ok := value.(planUnknown); ok { + return false, unknown.reason + } + if result, ok := value.(bool); ok { + return result, "" + } + return false, "condition is not a locally resolved boolean" +} + +func planScalar(value any) bool { + switch value.(type) { + case nil, string, bool, float64: + return true + default: + return false + } +} + +func (e *planEnvironment) value(raw any) any { + switch value := raw.(type) { + case string: + if strings.HasPrefix(value, "[[") { + return value[1:] + } + if !strings.HasPrefix(value, "[") || !strings.HasSuffix(value, "]") { + return value + } + parser := planExpressionParser{text: value[1 : len(value)-1]} + expression, err := parser.expression(0) + parser.space() + if err != nil { + return planUnknown{err.Error()} + } + if parser.pos != len(parser.text) { + return planUnknown{"ARM expression contains unsupported trailing syntax"} + } + return e.evaluate(expression, 0) + case map[string]any: + result := make(map[string]any, len(value)) + for _, key := range slices.Sorted(maps.Keys(value)) { + name, missing := planString(e.value(key)) + if missing != "" { + return planUnknown{"object key cannot be resolved: " + missing} + } + result[name] = e.value(value[key]) + } + return result + case []any: + result := make([]any, len(value)) + for i, entry := range value { + result[i] = e.value(entry) + } + return result + default: + return value + } +} + +func (e *planEnvironment) variable(name string) any { + key := strings.ToLower(name) + if result, found := e.variables[key]; found { + return result + } + if e.active[key] || len(e.active) >= 64 { + return planUnknown{"variable " + name + " is cyclic or exceeds the local dependency limit"} + } + definitions, _ := e.template["variables"].(map[string]any) + e.active[key] = true + defer delete(e.active, key) + for variable, raw := range definitions { + if strings.EqualFold(variable, name) { + result := e.value(raw) + e.variables[key] = result + return result + } + } + return planUnknown{"variable " + name + " is not defined in " + e.path} +} + +func (e *planEnvironment) parameter(name string) any { + key := strings.ToLower(name) + if e.active["parameter:"+key] { + return planUnknown{"parameter " + name + " has a cyclic default expression"} + } + e.active["parameter:"+key] = true + defer delete(e.active, "parameter:"+key) + if parameter, found := e.parameters[key]; found { + return parameter() + } + definitions, _ := e.template["parameters"].(map[string]any) + for parameter, definition := range definitions { + if strings.EqualFold(parameter, name) { + if definition, ok := definition.(map[string]any); ok { + if value, present := definition["defaultValue"]; present { + return e.value(value) + } + } + } + } + return planUnknown{"parameter " + name + " has no resolved value/default in " + e.path} +} + +func (e *planEnvironment) evaluate(expression planExpression, depth int) any { + if depth > 64 { + return planUnknown{"ARM expression evaluation exceeds the local nesting limit"} + } + eval := func(arg planExpression) any { return e.evaluate(arg, depth+1) } + if expression.base != nil { + base, selector := eval(*expression.base), eval(*expression.selector) + if index, ok := selector.(float64); ok { + if unknown, ok := base.(planUnknown); ok { + return unknown + } + values, ok := base.([]any) + if !ok || index < 0 || index != math.Trunc(index) || index >= float64(len(values)) { + return planUnknown{"array index cannot be resolved from the actual plan"} + } + return values[int(index)] + } + key, missing := planString(selector) + if missing != "" { + return planUnknown{missing} + } + return planField(base, key) + } + if expression.name == "" { + return expression.literal + } + name := strings.ToLower(expression.name) + if name == "coalesce" && len(expression.args) > 0 { + for _, arg := range expression.args { + if value := eval(arg); value != nil { + return value + } + } + return nil + } + if name == "tryget" && len(expression.args) >= 2 { + value := eval(expression.args[0]) + for _, arg := range expression.args[1:] { + if value == nil { + return nil + } + if unknown, ok := value.(planUnknown); ok { + return unknown + } + selector := eval(arg) + switch current := value.(type) { + case map[string]any: + key, missing := planString(selector) + if missing != "" { + return planUnknown{missing} + } + value = nil + for name, item := range current { + if strings.EqualFold(name, key) { + value = item + break + } + } + case []any: + index, ok := selector.(float64) + if !ok || index != math.Trunc(index) { + return planUnknown{"tryGet array index is not a resolved integer"} + } + value = nil + if index >= 0 && index < float64(len(current)) { + value = current[int(index)] + } + default: + return planUnknown{"tryGet input is not a locally resolved object or array"} + } + } + return value + } + if name == "if" && len(expression.args) == 3 { + condition, missing := planBool(eval(expression.args[0])) + if missing != "" { + return planUnknown{"if condition cannot be resolved: " + missing} + } + if condition { + return eval(expression.args[1]) + } + return eval(expression.args[2]) + } + if (name == "and" || name == "or") && len(expression.args) >= 2 { + var unresolved string + for _, arg := range expression.args { + value, missing := planBool(eval(arg)) + if missing != "" { + unresolved = missing + } else if value == (name == "or") { + return value + } + } + if unresolved != "" { + return planUnknown{unresolved} + } + return name == "and" + } + args := make([]any, len(expression.args)) + for i, arg := range expression.args { + args[i] = eval(arg) + } + if name == "createarray" { + return args + } + if name == "createobject" && len(args)%2 == 0 { + result := map[string]any{} + for i := 0; i < len(args); i += 2 { + key, ok := args[i].(string) + if !ok { + return planUnknown{"createObject key is not a resolved string"} + } + result[key] = args[i+1] + } + return result + } + if name == "resourceid" || name == "subscriptionresourceid" || name == "extensionresourceid" { + return e.resourceID(name, args) + } + if name == "split" && len(args) == 2 && args[1] == "/" { + if partial, ok := args[0].(planPartialID); ok && partial.components != nil { + return slices.Clone(partial.components) + } + } + for _, arg := range args { + if unknown, ok := arg.(planUnknown); ok { + return unknown + } + if partial, ok := arg.(planPartialID); ok { + return planUnknown{partial.missing} + } + } + if len(args) == 0 { + switch name { + case "true": + return true + case "false": + return false + case "null": + return nil + case "subscription": + if e.scope.missing != "" { + return planUnknown{e.scope.missing} + } + return map[string]any{"id": e.scope.subscriptionARMID, "subscriptionId": e.scope.subscriptionID} + case "resourcegroup": + if e.scope.missing != "" || e.scope.groupARMID == "" { + return planUnknown{"resource-group context is not resolved in " + e.path} + } + var location any = e.scope.location + if e.scope.location == "" { + location = planUnknown{"actual resource-group location has not been resolved"} + } + return map[string]any{"id": e.scope.groupARMID, "name": e.scope.group, "location": location} + case "deployment": + return map[string]any{"name": e.deployment} + case "copyindex": + return float64(e.copyIndex) + } + } + if len(args) == 1 { + switch name { + case "parameters": + if key, ok := args[0].(string); ok { + return e.parameter(key) + } + case "variables": + if key, ok := args[0].(string); ok { + return e.variable(key) + } + case "not": + if value, ok := args[0].(bool); ok { + return !value + } + case "empty", "length": + length := -1 + switch value := args[0].(type) { + case string: + if name == "length" && strings.ContainsFunc(value, func(r rune) bool { return r > 127 }) { + return planUnknown{"non-ASCII string length is not evaluated by the local plan adapter"} + } + length = len(value) + case []any: + length = len(value) + case map[string]any: + length = len(value) + case nil: + if name == "empty" { + return true + } + } + if length >= 0 { + if name == "empty" { + return length == 0 + } + return float64(length) + } + case "first", "last": + if values, ok := args[0].([]any); ok && len(values) > 0 { + if name == "first" { + return values[0] + } + return values[len(values)-1] + } + case "tolower", "toupper", "trim": + if value, ok := args[0].(string); ok { + switch name { + case "tolower": + return strings.ToLower(value) + case "toupper": + return strings.ToUpper(value) + default: + return strings.TrimSpace(value) + } + } + case "json": + if value, ok := args[0].(string); ok { + var decoded any + if err := json.Unmarshal([]byte(value), &decoded); err == nil { + return decoded + } + } + case "shallowmerge": + if values, ok := args[0].([]any); ok { + result := map[string]any{} + for _, value := range values { + object, ok := value.(map[string]any) + if !ok { + return planUnknown{"shallowMerge input is not a resolved object"} + } + maps.Copy(result, object) + } + return result + } + } + } + if name == "reference" || name == "resourceinfo" { + full := name == "resourceinfo" + if name == "reference" && len(args) == 3 { + mode, ok := args[2].(string) + if !ok || !strings.EqualFold(mode, "Full") { + return planUnknown{"reference mode is not supported by the local plan adapter"} + } + full = true + } + if len(args) >= 1 { + if symbol, ok := args[0].(string); ok { + if resource := e.resources[symbol]; resource != nil { + return resource.reference(full) + } + if parsed, err := arm.ParseResourceID(symbol); err == nil && parsed.SubscriptionID != "" { + for _, resource := range e.resources { + if !strings.EqualFold(resource.kind, parsed.ResourceType.String()) { + continue + } + if id, missing := resource.id(); missing == "" && strings.EqualFold(id, symbol) { + return resource.reference(full) + } + } + } + return planUnknown{"runtime reference " + symbol + " is not a local resource/module"} + } + } + } + if len(args) == 2 { + switch name { + case "equals": + if !planScalar(args[0]) || !planScalar(args[1]) { + return planUnknown{"structured or runtime-symbolic equality is not locally resolved"} + } + a, errA := json.Marshal(args[0]) + b, errB := json.Marshal(args[1]) + if errA == nil && errB == nil { + return string(a) == string(b) + } + case "contains": + switch value := args[0].(type) { + case []any: + if !planScalar(args[1]) { + return planUnknown{"contains requires a locally resolved scalar search value"} + } + var unresolved bool + expected, _ := json.Marshal(args[1]) + for _, item := range value { + if !planScalar(item) { + unresolved = true + continue + } + actual, _ := json.Marshal(item) + if string(actual) == string(expected) { + return true + } + } + if unresolved { + return planUnknown{"contains has unresolved or structured array entries"} + } + return false + case string: + if part, ok := args[1].(string); ok { + return strings.Contains(value, part) + } + } + case "split": + if text, ok := args[0].(string); ok { + if separator, ok := args[1].(string); ok { + parts := strings.Split(text, separator) + result := make([]any, len(parts)) + for i, part := range parts { + result[i] = part + } + return result + } + } + } + } + stringsOnly := make([]string, len(args)) + for i, arg := range args { + value, ok := arg.(string) + if !ok { + return planUnknown{"ARM function " + expression.name + " has unsupported plan argument types"} + } + stringsOnly[i] = value + } + switch name { + case "concat": + return strings.Join(stringsOnly, "") + case "format": + if len(stringsOnly) > 0 { + result := stringsOnly[0] + for i, value := range stringsOnly[1:] { + result = strings.ReplaceAll(result, "{"+strconv.Itoa(i)+"}", value) + } + if strings.ContainsAny(result, "{}") { + return planUnknown{"format uses unsupported placeholders"} + } + return result + } + case "replace": + if len(stringsOnly) == 3 { + return strings.ReplaceAll(stringsOnly[0], stringsOnly[1], stringsOnly[2]) + } + case "uniquestring", "guid": + if len(stringsOnly) > 0 { + return e.function(name, stringsOnly) + } + } + return planUnknown{"ARM function " + expression.name + " is not supported by the local plan adapter"} +} + +func (e *planEnvironment) resourceID(function string, args []any) any { + typeIndex := -1 + for i, arg := range args { + value, missing := planString(arg) + if missing == "" && !strings.HasPrefix(value, "/") && strings.Contains(value, ".") && + strings.Contains(value, "/") { + typeIndex = i + break + } + } + if typeIndex < 0 || typeIndex == len(args)-1 { + return planUnknown{"resourceId has no resolved type/name"} + } + scope := e.scope.groupARMID + prefix := make([]string, typeIndex) + for i := range typeIndex { + value, missing := planString(args[i]) + if missing != "" { + known := "" + if function == "resourceid" && i == 1 { + known = "/subscriptions/" + prefix[0] + } + kind, _ := planString(args[typeIndex]) + return planPartialID{kind: kind, scope: known, missing: "resourceId scope: " + missing} + } + prefix[i] = value + } + switch function { + case "extensionresourceid": + if typeIndex != 1 { + return planUnknown{"extensionResourceId scope cannot be resolved"} + } + scope = prefix[0] + case "subscriptionresourceid": + scope = e.scope.subscriptionARMID + if typeIndex == 1 { + scope = "/subscriptions/" + prefix[0] + } else if typeIndex != 0 { + return planUnknown{"subscriptionResourceId scope cannot be resolved"} + } + case "resourceid": + if typeIndex == 1 { + scope = e.scope.subscriptionARMID + "/resourceGroups/" + prefix[0] + } else if typeIndex == 2 { + scope = "/subscriptions/" + prefix[0] + "/resourceGroups/" + prefix[1] + } else if typeIndex != 0 { + return planUnknown{"resourceId scope cannot be resolved"} + } + } + if scope == "" || e.scope.missing != "" && typeIndex == 0 { + return planUnknown{"resourceId deployment scope cannot be resolved"} + } + kind, _ := planString(args[typeIndex]) + var parts []string + var components []any + for part := range strings.SplitSeq(scope, "/") { + components = append(components, part) + } + types := strings.Split(kind, "/") + components = append(components, "providers", types[0]) + var nameMissing string + for _, argument := range args[typeIndex+1:] { + name, missing := planString(argument) + if missing != "" { + nameMissing = "resourceId resource name: " + missing + parts = append(parts, "") + } else { + parts = append(parts, name) + } + } + if nameMissing != "" { + if len(types)-1 == len(parts) { + for i, argument := range args[typeIndex+1:] { + components = append(components, types[i+1], argument) + } + } else { + components = nil + } + return planPartialID{kind: kind, scope: scope, missing: nameMissing, components: components} + } + names := strings.Split(strings.Join(parts, "/"), "/") + if len(types)-1 != len(names) { + return planUnknown{"resourceId type/name hierarchy cannot be resolved"} + } + var result strings.Builder + result.WriteString(scope + "/providers/" + types[0]) + for i, name := range names { + result.WriteString("/" + types[i+1] + "/" + name) + } + return result.String() +} diff --git a/cli/azd/extensions/azure.ai.projects/internal/provisioning/byo_template_plan.go b/cli/azd/extensions/azure.ai.projects/internal/provisioning/byo_template_plan.go new file mode 100644 index 00000000000..229b0d9b0b7 --- /dev/null +++ b/cli/azd/extensions/azure.ai.projects/internal/provisioning/byo_template_plan.go @@ -0,0 +1,543 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +package provisioning + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "maps" + "net/http" + "slices" + "strings" + + "azure.ai.projects/internal/onboarding" + + "github.com/Azure/azure-sdk-for-go/sdk/azcore" + "github.com/Azure/azure-sdk-for-go/sdk/azcore/arm" + "github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/resources/armresources" + "github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/resources/armsubscriptions" +) + +type templatePlanResolver struct { + ctx context.Context + provider *FoundryProvisioningProvider + scopes map[string]planScope + functions map[string]any + references map[*planResource]bool + owningSubscriptions map[string]bool + nodes []*planResource + account *planResource + unresolved []string + err error +} + +type planResource struct { + raw map[string]any + env *planEnvironment + resolver *templatePlanResolver + symbol string + path string + kind string + inactive bool + missing string + child *planEnvironment + childRead bool +} + +func (r *templatePlanResolver) scope(subscription, group string) planScope { + key := strings.ToLower(subscription + "/" + group) + if scope, present := r.scopes[key]; present { + return scope + } + scope := planScope{subscriptionID: subscription, group: group} + if subscription == "" || strings.ContainsAny(subscription+group, "/\\?#%@:\x00 \t\r\n") { + scope.missing = "template deployment subscription/resource-group scope is not resolved" + r.err = errors.Join(r.err, fmt.Errorf("I03: actual template subscription/resource-group scope is invalid")) + return scope + } + subKey := strings.ToLower(subscription + "/") + subscriptionScope, found := r.scopes[subKey] + if !found { + client, err := armsubscriptions.NewClient(r.provider.credential, byoARMOptions(r.provider.armClientOptions)) + if err == nil { + var response armsubscriptions.ClientGetResponse + response, err = client.Get(r.ctx, subscription, nil) + if err == nil { + if response.ID == nil || !strings.EqualFold(*response.ID, "/subscriptions/"+subscription) { + err = fmt.Errorf("I02: subscription lookup did not return the actual selected subscription ARM ID") + } else { + subscriptionScope = planScope{subscriptionID: subscription, subscriptionARMID: *response.ID} + } + } + } + if err != nil { + r.err = errors.Join(r.err, fmt.Errorf("I02: read actual template subscription scope: %w", err)) + subscriptionScope.missing = "actual subscription scope could not be read" + } + r.scopes[subKey] = subscriptionScope + } + scope.subscriptionARMID = subscriptionScope.subscriptionARMID + scope.missing = subscriptionScope.missing + if group == "" || scope.missing != "" { + return scope + } + scope.groupARMID = scope.subscriptionARMID + "/resourceGroups/" + group + client, err := armresources.NewResourceGroupsClient(subscription, r.provider.credential, + byoARMOptions(r.provider.armClientOptions)) + if err == nil { + var response armresources.ResourceGroupsClientGetResponse + response, err = client.Get(r.ctx, group, nil) + if err == nil { + if response.ID == nil || !strings.EqualFold(*response.ID, scope.groupARMID) { + err = fmt.Errorf("I02: resource-group lookup did not return the actual selected resource-group ARM ID") + } else { + scope.groupARMID = *response.ID + scope.groupExists = true + if response.Location != nil { + scope.location = *response.Location + } + } + } + } + if response, ok := errors.AsType[*azcore.ResponseError](err); ok && response.StatusCode == http.StatusNotFound { + err = nil + } + if err != nil { + r.err = errors.Join(r.err, fmt.Errorf("I02: read actual template resource-group scope: %w", err)) + scope.missing = "actual resource-group scope could not be read" + } + r.scopes[key] = scope + return scope +} + +func (r *templatePlanResolver) function(name string, args []string) any { + raw, err := json.Marshal(args) + if err != nil { + r.err = errors.Join(r.err, err) + return planUnknown{"local Bicep arguments could not be encoded"} + } + key := name + string(raw) + if value, present := r.functions[key]; present { + return value + } + value, err := evaluateBYOPlanFunction(r.ctx, r.provider.bicepCli(), name, args) + if err != nil { + if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) { + r.err = errors.Join(r.err, err) + } + unknown := planUnknown{"azd-managed Bicep could not evaluate " + name + ": " + err.Error()} + r.functions[key] = unknown + return unknown + } + r.functions[key] = value + return value +} + +func (r *templatePlanResolver) environment( + template map[string]any, parameters map[string]func() any, scope planScope, path string, deployment any, +) *planEnvironment { + return &planEnvironment{ + template: template, parameters: parameters, scope: scope, path: path, + deployment: deployment, function: r.function, + variables: map[string]any{}, active: map[string]bool{}, resources: map[string]*planResource{}, + } +} + +func (r *templatePlanResolver) walk(env *planEnvironment, inheritedMissing string, depth int) { + if env.walked { + return + } + env.walked = true + if depth > 32 || len(r.nodes) > 2048 { + r.unresolved = append(r.unresolved, env.path+": local module/resource enumeration limit reached") + return + } + var entries []struct { + symbol string + raw map[string]any + } + switch resources := env.template["resources"].(type) { + case map[string]any: + for _, symbol := range slices.Sorted(maps.Keys(resources)) { + if resource, ok := resources[symbol].(map[string]any); ok { + entries = append(entries, struct { + symbol string + raw map[string]any + }{symbol, resource}) + } + } + case []any: + for index, value := range resources { + if resource, ok := value.(map[string]any); ok { + entries = append(entries, struct { + symbol string + raw map[string]any + }{fmt.Sprintf("resources[%d]", index), resource}) + } + } + } + for _, entry := range entries { + count := 1 + copyMissing := "" + if copy, ok := entry.raw["copy"].(map[string]any); ok { + value := env.value(copy["count"]) + number, ok := value.(float64) + if !ok || number < 0 || number > 128 || number != float64(int(number)) { + copyMissing = "resource copy count cannot be resolved within the local plan limit (128)" + } else { + count = int(number) + } + } + for index := range count { + local := *env + local.copyIndex = index + local.variables, local.active = map[string]any{}, map[string]bool{} + path := env.path + "/" + entry.symbol + if count > 1 { + path += fmt.Sprintf("[%d]", index) + } + resource := &planResource{ + raw: entry.raw, env: &local, resolver: r, symbol: entry.symbol, path: path, + missing: strings.Join(slices.DeleteFunc([]string{inheritedMissing, copyMissing}, + func(s string) bool { return s == "" }), "; "), + } + var typeMissing string + resource.kind, typeMissing = planString(local.value(entry.raw["type"])) + if condition, present := entry.raw["condition"]; present { + value, missing := planBool(local.value(condition)) + if missing != "" { + resource.missing = strings.Trim(resource.missing+"; creation condition: "+missing, "; ") + } else { + resource.inactive = !value + } + } + if existing, _ := entry.raw["existing"].(bool); existing { + resource.inactive = true + } + if !resource.inactive && typeMissing != "" { + r.unresolved = append(r.unresolved, path+": resource type: "+typeMissing) + } + env.resources[entry.symbol] = resource + r.nodes = append(r.nodes, resource) + } + } + // Install all symbolic resources before resolving module-output references. + for _, resource := range slices.Clone(r.nodes) { + if resource.env.path != env.path || resource.inactive || + !strings.EqualFold(resource.kind, "Microsoft.Resources/deployments") { + continue + } + if child := resource.module(); child != nil { + r.walk(child, resource.missing, depth+1) + } + } +} + +func (n *planResource) module() *planEnvironment { + if n.childRead { + return n.child + } + n.childRead = true + properties, ok := n.raw["properties"].(map[string]any) + if !ok { + n.missing = "deployment module properties are not an inline object" + return nil + } + template, ok := properties["template"].(map[string]any) + if !ok { + n.missing = "linked or runtime-generated deployment template is not locally available" + n.resolver.unresolved = append(n.resolver.unresolved, n.path+": "+n.missing) + return nil + } + evaluation, _ := properties["expressionEvaluationOptions"].(map[string]any) + if evaluation["scope"] != "inner" { + n.missing = "nested template does not use supported inner expression evaluation" + n.resolver.unresolved = append(n.resolver.unresolved, n.path+": "+n.missing) + return nil + } + scope := n.env.scope + if schema, _ := template["$schema"].(string); strings.Contains(schema, "subscriptionDeploymentTemplate") { + scope.group, scope.groupARMID = "", "" + scope.groupExists = false + } + if raw, present := n.raw["subscriptionId"]; present { + subscription, missing := planString(n.env.value(raw)) + if missing != "" { + scope.missing = "module subscription: " + missing + scope.subscriptionID, scope.subscriptionARMID, scope.groupARMID = "", "", "" + scope.groupExists = false + } else { + scope.subscriptionID = subscription + } + } + if raw, present := n.raw["resourceGroup"]; present { + group, missing := planString(n.env.value(raw)) + if missing != "" { + scope.missing = "module resource group: " + missing + scope.group, scope.groupARMID = "", "" + scope.groupExists = false + } else { + scope.group = group + } + } + if scope.missing == "" { + scope = n.resolver.scope(scope.subscriptionID, scope.group) + if scope.location == "" && !scope.groupExists && scope.groupARMID != "" { + for _, resource := range n.env.resources { + if resource.inactive || resource.missing != "" || + !strings.EqualFold(resource.kind, "Microsoft.Resources/resourceGroups") { + continue + } + if id, missing := resource.id(); missing == "" && strings.EqualFold(id, scope.groupARMID) { + scope.location, _ = planString(resource.env.value(resource.raw["location"])) + } + } + } + } + parameters := map[string]func() any{} + if supplied, ok := properties["parameters"].(map[string]any); ok { + for key, envelope := range supplied { + parameters[strings.ToLower(key)] = func() any { + value := n.env.value(envelope) + return planField(value, "value") + } + } + } + n.child = n.resolver.environment(template, parameters, scope, n.path, n.env.value(n.raw["name"])) + return n.child +} + +func (n *planResource) id() (string, string) { + name, missing := planString(n.env.value(n.raw["name"])) + if missing != "" { + return "", "resource name: " + missing + } + if n.env.scope.missing != "" { + return "", n.env.scope.missing + } + if strings.EqualFold(n.kind, "Microsoft.Resources/resourceGroups") { + return n.env.scope.subscriptionARMID + "/resourceGroups/" + name, "" + } + scope := n.env.scope.groupARMID + if raw, present := n.raw["scope"]; present { + value, why := planString(n.env.value(raw)) + if why != "" { + return "", "resource scope: " + why + } + if strings.HasPrefix(value, "/") { + scope = value + } else { + scope += "/providers/" + value + } + } + if scope == "" || n.kind == "" { + return "", "resource type or deployment resource-group scope is not resolved" + } + types, names := strings.Split(n.kind, "/"), strings.Split(name, "/") + if len(types)-1 != len(names) { + return "", "resource type/name hierarchy is not resolved" + } + var result strings.Builder + result.WriteString(scope + "/providers/" + types[0]) + for i, part := range names { + result.WriteString("/" + types[i+1] + "/" + part) + } + return result.String(), "" +} + +func (n *planResource) reference(full bool) any { + if n.resolver.references[n] { + return planUnknown{"cyclic local resource/module reference at " + n.path} + } + n.resolver.references[n] = true + defer delete(n.resolver.references, n) + if strings.EqualFold(n.kind, "Microsoft.Resources/deployments") { + child := n.module() + if child == nil { + return planUnknown{n.missing} + } + if len(child.resources) == 0 { + // Output evaluation can precede the main walk (e.g. one module uses another's output). + n.resolver.walk(child, n.missing, 1) + } + outputs, _ := child.template["outputs"].(map[string]any) + result := map[string]any{} + for key, output := range outputs { + result[key] = child.value(output) + } + properties := map[string]any{"outputs": result} + if full { + return map[string]any{"properties": properties} + } + return properties + } + id, missing := n.id() + if missing != "" { + return planUnknown{missing} + } + properties := map[string]any{"principalId": planPrincipal{resourceID: id}} + if !full { + return properties + } + return map[string]any{ + "id": id, "identity": map[string]any{"principalId": planPrincipal{resourceID: id}}, + "properties": properties, + "location": planUnknown{"runtime resource location is not a local template value"}, + } +} + +func (r *templatePlanResolver) resource(value any, kind, origin string) onboarding.Resource { + id, missing := planString(value) + result := onboarding.Resource{ID: id, Origin: kind + " at " + origin, Missing: missing} + if missing != "" { + if partial, ok := value.(planPartialID); ok { + if !strings.EqualFold(partial.kind, kind) { + r.err = errors.Join(r.err, fmt.Errorf("I02: actual template resource type differs from expected %s", kind)) + return result + } + if strings.ContainsAny(partial.scope, "?#%\\@:\x00 \t\r\n") { + r.err = errors.Join(r.err, fmt.Errorf("I02: actual template resource scope is malformed")) + return result + } + if scope, err := arm.ParseResourceID(partial.scope); err == nil && scope.SubscriptionID != "" { + result.Parent = partial.scope + r.recordOwningSubscription(kind, scope.SubscriptionID) + if strings.EqualFold(kind, "Microsoft.Network/virtualNetworks/subnets") && + len(partial.components) > 2 { + var prefix []string + for _, component := range partial.components[:len(partial.components)-2] { + part, missing := planString(component) + if missing != "" { + prefix = nil + break + } + prefix = append(prefix, part) + } + parent := strings.Join(prefix, "/") + if _, err := onboarding.ParseID(parent, "Microsoft.Network/virtualNetworks"); err == nil { + result.Parent = parent + } + } + } else if partial.scope != "" { + r.err = errors.Join(r.err, fmt.Errorf("I02: actual template resource scope is malformed")) + } + } else if _, unknown := value.(planUnknown); !unknown { + r.err = errors.Join(r.err, fmt.Errorf("I02: actual template %s resource ID must be a string", kind)) + } + return result + } + parsed, err := onboarding.ParseID(id, kind) + if err != nil { + r.err = errors.Join(r.err, fmt.Errorf("I02: actual template %s: %w", kind, err)) + return result + } + result.Parent = parsed.Parent.String() + r.recordOwningSubscription(kind, parsed.SubscriptionID) + uncertain := strings.Join(r.unresolved, "; ") + for _, node := range r.nodes { + if node.inactive || !strings.EqualFold(node.kind, kind) { + continue + } + actual, why := node.id() + if why != "" { + uncertain = node.path + ": " + why + continue + } + if !strings.EqualFold(actual, id) { + continue + } + if node.missing != "" { + uncertain = node.path + ": " + node.missing + continue + } + if node.env.scope.groupExists { + result.Parent = node.env.scope.groupARMID + } else { + result.Parent = node.env.scope.subscriptionARMID + } + return result + } + if uncertain != "" { + result.LifecycleMissing = "cannot determine creation/reference selection: " + uncertain + } else { + result.Supplied = true + } + return result +} + +func (r *templatePlanResolver) recordOwningSubscription(kind, subscription string) { + switch strings.ToLower(kind) { + case "microsoft.containerservice/managedclusters", "microsoft.storage/storageaccounts", + "microsoft.managedidentity/userassignedidentities", "microsoft.containerregistry/registries", + "microsoft.network/privatednszones": + r.owningSubscriptions[strings.ToLower(subscription)] = true + } +} + +func (p *FoundryProvisioningProvider) identifyBYOTemplatePlan( + ctx context.Context, source *templateSource, +) (onboarding.Plan, error) { + resolver := &templatePlanResolver{ + ctx: ctx, provider: p, scopes: map[string]planScope{}, functions: map[string]any{}, + references: map[*planResource]bool{}, owningSubscriptions: map[string]bool{}, + } + encoded, err := json.Marshal(source.parameters) + if err != nil { + return onboarding.Plan{}, fmt.Errorf("encode actual merged BYO plan parameters: %w", err) + } + var merged map[string]any + if err := json.Unmarshal(encoded, &merged); err != nil { + return onboarding.Plan{}, fmt.Errorf("decode actual merged BYO plan parameters: %w", err) + } + parameters := map[string]func() any{} + for key, envelope := range merged { + parameters[strings.ToLower(key)] = func() any { + if value, ok := envelope.(map[string]any); ok { + if literal, present := value["value"]; present { + return literal + } + } + return planUnknown{"parameter " + key + " requires a runtime parameter reference"} + } + } + if definitions, ok := source.armTemplate["parameters"].(map[string]any); ok { + for name := range definitions { + if strings.EqualFold(name, "kubeletObjectId") && parameters["kubeletobjectid"] == nil { + parameters["kubeletobjectid"] = func() any { return planPrincipal{binding: "kubeletObjectId"} } + } + } + } + scope := resolver.scope(p.subID, "") + env := resolver.environment(source.armTemplate, parameters, scope, "compiled template", p.deploymentName()) + resolver.walk(env, "", 0) + plan := onboarding.Plan{SubscriptionID: p.subID, Grants: []onboarding.PlannedGrant{}} + resolver.owningSubscriptions[strings.ToLower(p.subID)] = true + resolver.bindResources(&plan, env) + group, missing := planString(env.parameter("resourceGroupName")) + if resolver.account != nil { + group, missing = resolver.account.env.scope.group, resolver.account.env.scope.missing + if subscription := resolver.account.env.scope.subscriptionID; subscription != "" { + plan.SubscriptionID = subscription + resolver.owningSubscriptions[strings.ToLower(subscription)] = true + } else { + missing = strings.Trim(missing+"; actual Foundry account subscription is not resolved", "; ") + plan.SkippedChecks = append(plan.SkippedChecks, onboarding.SkippedCheck("I02", + resolver.account.path, "Foundry account subscription owning tenant", missing)) + } + } + plan.ResourceGroup = group + if missing == "" { + plan.ResourceGroupSupplied = resolver.scope(plan.SubscriptionID, group).groupExists + } else { + plan.TargetScopeMissing = "actual Foundry target resource-group selection: " + missing + plan.SkippedChecks = append(plan.SkippedChecks, onboarding.SkippedCheck("I03", env.path, + "actual Foundry target resource-group scope", plan.TargetScopeMissing)) + } + resolver.bindGrants(&plan) + resolver.bindNetwork(&plan) + plan.OwningSubscriptions = append(plan.OwningSubscriptions, slices.Sorted(maps.Keys(resolver.owningSubscriptions))...) + return plan, resolver.err +} diff --git a/cli/azd/extensions/azure.ai.projects/internal/provisioning/byo_template_plan_test.go b/cli/azd/extensions/azure.ai.projects/internal/provisioning/byo_template_plan_test.go new file mode 100644 index 00000000000..7756a3c49a6 --- /dev/null +++ b/cli/azd/extensions/azure.ai.projects/internal/provisioning/byo_template_plan_test.go @@ -0,0 +1,87 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +package provisioning + +import ( + "testing" + + "azure.ai.projects/internal/onboarding" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +func TestBYOPlanNestedDeploymentName(t *testing.T) { + scope := planScope{subscriptionID: "sub", subscriptionARMID: "/subscriptions/sub", + group: "rg", groupARMID: "/subscriptions/sub/resourceGroups/rg", groupExists: true} + for _, name := range []string{"inner", "[parameters('runtimeName')]"} { + t.Run(name, func(t *testing.T) { + resolver := &templatePlanResolver{scopes: map[string]planScope{"sub/rg": scope}} + template := map[string]any{"resources": map[string]any{"nested": map[string]any{ + "type": "Microsoft.Resources/deployments", "name": name, + "properties": map[string]any{ + "expressionEvaluationOptions": map[string]any{"scope": "inner"}, + "template": map[string]any{"resources": map[string]any{"identity": map[string]any{ + "type": "Microsoft.ManagedIdentity/userAssignedIdentities", "name": "[deployment().name]", + }}}, + }, + }}} + env := resolver.environment(template, nil, scope, "root", "outer") + resolver.walk(env, "", 0) + child := env.resources["nested"].module() + require.NotNil(t, child) + id, missing := child.resources["identity"].id() + if name == "inner" { + assert.Equal(t, scope.groupARMID+"/providers/Microsoft.ManagedIdentity/userAssignedIdentities/inner", id) + assert.Empty(t, missing) + } else { + assert.Empty(t, id) + assert.Contains(t, missing, "runtimeName") + } + assert.Equal(t, "outer", planField(env.value("[deployment()]"), "name")) + }) + } +} + +func TestBYOPlanReferencePreservesAncestorScopeGrants(t *testing.T) { + scope := planScope{subscriptionID: "sub", subscriptionARMID: "/subscriptions/sub", + group: "rg", groupARMID: "/subscriptions/sub/resourceGroups/rg", groupExists: true} + for _, selector := range []string{ + "reference(resourceId('Microsoft.ManagedIdentity/userAssignedIdentities', 'hosting'), '2023-01-31').principalId", + "reference(resourceId('Microsoft.ManagedIdentity/userAssignedIdentities', 'hosting'), '2023-01-31', 'Full')." + + "properties.principalId", + } { + t.Run(selector, func(t *testing.T) { + resolver := &templatePlanResolver{scopes: map[string]planScope{"sub/rg": scope}, + references: map[*planResource]bool{}} + template := map[string]any{"resources": map[string]any{ + "hosting": map[string]any{ + "type": "Microsoft.ManagedIdentity/userAssignedIdentities", "name": "hosting", + }, + "grant": map[string]any{ + "type": "Microsoft.Authorization/roleAssignments", "name": "grant", + "properties": map[string]any{ + "principalId": "[" + selector + "]", + "roleDefinitionId": scope.subscriptionARMID + "/providers/Microsoft.Authorization/" + + "roleDefinitions/acdd72a7-3385-48ef-bd42-f606fba81ae7", + }, + }, + }} + env := resolver.environment(template, nil, scope, "root", "deployment") + resolver.walk(env, "", 0) + hostingID, missing := env.resources["hosting"].id() + require.Empty(t, missing) + plan := onboarding.Plan{Hosting: onboarding.Resource{ID: hostingID}, + Cluster: onboarding.Resource{ID: offlineClusterID}} + resolver.bindGrants(&plan) + require.NoError(t, resolver.err) + require.Len(t, plan.Grants, 1) + assert.Equal(t, "G01", plan.Grants[0].CheckID) + assert.Equal(t, scope.groupARMID, plan.Grants[0].AuthorityScope) + assert.Empty(t, plan.Grants[0].Missing) + assert.Equal(t, hostingID, planField(env.value("[reference('hosting', '2023-01-31', 'Full')]"), "id")) + assert.IsType(t, planUnknown{}, env.value("[reference('hosting', '2023-01-31').id]")) + }) + } +} diff --git a/cli/azd/extensions/azure.ai.projects/internal/provisioning/foundry_provisioning_provider.go b/cli/azd/extensions/azure.ai.projects/internal/provisioning/foundry_provisioning_provider.go index 2bdac95b998..f47c6a701e3 100644 --- a/cli/azd/extensions/azure.ai.projects/internal/provisioning/foundry_provisioning_provider.go +++ b/cli/azd/extensions/azure.ai.projects/internal/provisioning/foundry_provisioning_provider.go @@ -1330,7 +1330,7 @@ func (p *FoundryProvisioningProvider) Deploy( if err := p.validateACRRemoteBuild(ctx, src); err != nil { return nil, err } - if err := p.prepareBYO(ctx, src); err != nil { + if err := p.prepareBYO(ctx, src, progress); err != nil { return nil, err } if err := p.recoverDeployedACRBinding(ctx); err != nil { @@ -1677,7 +1677,7 @@ func (p *FoundryProvisioningProvider) Preview( if err != nil { return nil, err } - if err := p.prepareBYO(ctx, src); err != nil { + if err := p.prepareBYO(ctx, src, progress); err != nil { return nil, err } if p.greenfieldAcrMode == "reuse-connect" { diff --git a/cli/azd/extensions/azure.ai.projects/internal/provisioning/onboarding_naming.go b/cli/azd/extensions/azure.ai.projects/internal/provisioning/onboarding_naming.go new file mode 100644 index 00000000000..3895dbf4b38 --- /dev/null +++ b/cli/azd/extensions/azure.ai.projects/internal/provisioning/onboarding_naming.go @@ -0,0 +1,73 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +package provisioning + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "os" + "path/filepath" + "regexp" + "strings" +) + +var officialResourceToken = regexp.MustCompile(`^[a-z2-7]{13}$`) + +func evaluateBYOPlanFunction( + ctx context.Context, compiler bicepCompiler, name string, args []string, +) (value string, err error) { + if name != "uniquestring" && name != "guid" { + return "", fmt.Errorf("unsupported local Bicep plan function") + } + if name == "uniquestring" { + name = "uniqueString" + } + dir, err := os.MkdirTemp("", "azd-foundry-plan-") + if err != nil { + return "", fmt.Errorf("create Bicep plan evaluation directory: %w", err) + } + defer func() { err = errors.Join(err, os.RemoveAll(dir)) }() + raw, err := json.Marshal(args) + if err != nil { + return "", fmt.Errorf("encode Bicep plan arguments: %w", err) + } + if err := os.WriteFile(filepath.Join(dir, "arguments.json"), raw, 0o600); err != nil { + return "", fmt.Errorf("write Bicep plan arguments: %w", err) + } + arguments := make([]string, len(args)) + for i := range args { + arguments[i] = fmt.Sprintf("arguments[%d]", i) + } + probe := "using none\nvar arguments = loadJsonContent('./arguments.json')\nparam result = " + + name + "(" + strings.Join(arguments, ", ") + ")\n" + path := filepath.Join(dir, "plan.bicepparam") + if err := os.WriteFile(path, []byte(probe), 0o600); err != nil { + return "", fmt.Errorf("write local Bicep plan expression: %w", err) + } + result, err := compiler.BuildBicepParam(ctx, path, []string{"BICEP_PARAMETERS_OVERRIDES={}"}) + if err != nil { + return "", fmt.Errorf("evaluate template naming with azd-managed Bicep: %w", err) + } + var envelope struct { + ParametersJSON string `json:"parametersJson"` + } + if err := json.Unmarshal([]byte(result.Compiled), &envelope); err != nil { + return "", fmt.Errorf("decode Bicep plan evaluation: %w", err) + } + var parameters struct { + Parameters map[string]struct { + Value string `json:"value"` + } `json:"parameters"` + } + if err := json.Unmarshal([]byte(envelope.ParametersJSON), ¶meters); err != nil { + return "", fmt.Errorf("decode Bicep plan parameters: %w", err) + } + value = parameters.Parameters["result"].Value + if value == "" || name == "uniqueString" && !officialResourceToken.MatchString(value) { + return "", fmt.Errorf("Bicep returned an invalid local plan value") + } + return value, nil +} diff --git a/cli/azd/extensions/azure.ai.projects/internal/provisioning/onboarding_naming_test.go b/cli/azd/extensions/azure.ai.projects/internal/provisioning/onboarding_naming_test.go new file mode 100644 index 00000000000..788eea2b6ad --- /dev/null +++ b/cli/azd/extensions/azure.ai.projects/internal/provisioning/onboarding_naming_test.go @@ -0,0 +1,81 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +package provisioning + +import ( + "context" + "encoding/json" + "errors" + "os" + "os/exec" + "path/filepath" + "testing" + + "azure.ai.projects/internal/testutil" + + "github.com/azure/azure-dev/cli/azd/pkg/tools/bicep" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +func TestBYOResourceTokenUsesActualBicepAndPreservesScopeAndSalt(t *testing.T) { + tool := testutil.LookupBicep(t) + var tokens []string + for _, salt := range []string{"", "custom"} { + scope := []string{ + "/subscriptions/Target-Sub", + "/subscriptions/Target-Sub/resourceGroups/Target-RG", "westus", + } + if salt != "" { + scope = append(scope, salt) + } + var directory string + compiler := &stubCompiler{buildParam: func( + ctx context.Context, path string, env []string, + ) (bicep.BuildResult, error) { + directory = filepath.Dir(path) + //nolint:gosec // Probe path is created by evaluateBYOPlanFunction in a temporary directory. + raw, err := os.ReadFile(path) + require.NoError(t, err) + assert.Contains(t, string(raw), "uniqueString(arguments[0], arguments[1], arguments[2]") + //nolint:gosec // Scope file is written next to the test-owned probe. + raw, err = os.ReadFile(filepath.Join(directory, "arguments.json")) + require.NoError(t, err) + var actual []string + require.NoError(t, json.Unmarshal(raw, &actual)) + assert.Equal(t, scope, actual) + assert.Equal(t, []string{"BICEP_PARAMETERS_OVERRIDES={}"}, env) + //nolint:gosec // Explicit Bicep tool override used by the existing offline template checks. + command := exec.CommandContext(ctx, tool, "build-params", path, "--stdout") + command.Env = append(os.Environ(), env...) + output, err := command.CombinedOutput() + return bicep.BuildResult{Compiled: string(output)}, err + }} + token, err := evaluateBYOPlanFunction(t.Context(), compiler, "uniquestring", scope) + require.NoError(t, err) + require.Regexp(t, officialResourceToken, token) + assert.Len(t, token, 13) + tokens = append(tokens, token) + _, err = os.Stat(directory) + assert.ErrorIs(t, err, os.ErrNotExist) + } + assert.NotEqual(t, tokens[0], tokens[1], "the existing optional salt remains part of default resource naming") +} + +func TestBYOResourceTokenSurfacesCompilerAndEvidenceFailures(t *testing.T) { + cause := errors.New("offline compiler failure") + _, err := evaluateBYOPlanFunction(t.Context(), &stubCompiler{buildParamErr: cause}, "uniquestring", []string{"scope"}) + require.ErrorIs(t, err, cause) + for _, output := range []string{ + "not-json", `{}`, `{"parametersJson":"{}"}`, + `{"parametersJson":"{\"parameters\":{\"result\":{\"value\":\"invalid\"}}}"}`, + } { + compiler := &stubCompiler{buildParamResult: bicep.BuildResult{Compiled: output}} + _, err := evaluateBYOPlanFunction(t.Context(), compiler, "uniquestring", []string{"scope"}) + require.Error(t, err) + require.Len(t, compiler.buildParamCalls, 1) + _, err = os.Stat(filepath.Dir(compiler.buildParamCalls[0])) + assert.ErrorIs(t, err, os.ErrNotExist) + } +} diff --git a/cli/azd/extensions/azure.ai.projects/internal/provisioning/supplied_resource_bindings.go b/cli/azd/extensions/azure.ai.projects/internal/provisioning/supplied_resource_bindings.go deleted file mode 100644 index 0e8985632cc..00000000000 --- a/cli/azd/extensions/azure.ai.projects/internal/provisioning/supplied_resource_bindings.go +++ /dev/null @@ -1,39 +0,0 @@ -// Copyright (c) Microsoft Corporation. All rights reserved. -// Licensed under the MIT License. - -// cspell:ignore deploymenttemplate - -package provisioning - -import ( - "context" - "fmt" - "strings" - - "github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/msi/armmsi" - "github.com/google/uuid" -) - -func (p *FoundryProvisioningProvider) readBYOIdentityPrincipal(ctx context.Context, resourceID string) (string, error) { - id, err := parseBYOResourceID(resourceID, "Microsoft.ManagedIdentity/userAssignedIdentities") - if err != nil { - return "", err - } - client, err := armmsi.NewUserAssignedIdentitiesClient(id.SubscriptionID, p.credential, byoARMOptions(p.armClientOptions)) - if err != nil { - return "", fmt.Errorf("create supplied identity client: %w", err) - } - identity, err := client.Get(ctx, id.ResourceGroupName, id.Name, nil) - if err != nil { - return "", fmt.Errorf("read supplied identity deployment binding: %w", err) - } - if identity.ID == nil || !strings.EqualFold(*identity.ID, resourceID) || - identity.Properties == nil || identity.Properties.PrincipalID == nil { - return "", fmt.Errorf("supplied identity deployment binding requires a matching resource ID and principal ID") - } - principal := *identity.Properties.PrincipalID - if _, err := uuid.Parse(principal); err != nil { - return "", fmt.Errorf("supplied identity principal ID must be a GUID: %w", err) - } - return principal, nil -} diff --git a/cli/azd/extensions/azure.ai.projects/internal/provisioning/supplied_resources_test.go b/cli/azd/extensions/azure.ai.projects/internal/provisioning/supplied_resources_test.go index eca9ae44d68..44181024e97 100644 --- a/cli/azd/extensions/azure.ai.projects/internal/provisioning/supplied_resources_test.go +++ b/cli/azd/extensions/azure.ai.projects/internal/provisioning/supplied_resources_test.go @@ -5,11 +5,10 @@ package provisioning import ( "context" + "encoding/base64" "encoding/json" - "errors" "fmt" "net/http" - "slices" "strings" "sync" "testing" @@ -17,64 +16,133 @@ import ( "azure.ai.projects/internal/synthesis" + "github.com/Azure/azure-sdk-for-go/sdk/azcore" "github.com/Azure/azure-sdk-for-go/sdk/azcore/arm" "github.com/Azure/azure-sdk-for-go/sdk/azcore/policy" + "github.com/azure/azure-dev/cli/azd/pkg/tools/bicep" "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" ) -const suppliedStorageID = "/subscriptions/data-sub/resourceGroups/data-rg/providers/Microsoft.Storage/storageAccounts/data" -const suppliedHostingID = "/subscriptions/identity-sub/resourceGroups/identity-rg/providers/" + - "Microsoft.ManagedIdentity/userAssignedIdentities/hosting" -const suppliedWorkloadID = "/subscriptions/workload-sub/resourceGroups/workload-rg/providers/" + - "Microsoft.ManagedIdentity/userAssignedIdentities/workload" +const suppliedGroupID = "/subscriptions/target-sub/resourceGroups/target-rg" +const suppliedStorageID = suppliedGroupID + "/providers/Microsoft.Storage/storageAccounts/customer" +const suppliedHostingID = suppliedGroupID + "/providers/Microsoft.ManagedIdentity/userAssignedIdentities/hosting" +const suppliedWorkloadID = suppliedGroupID + "/providers/Microsoft.ManagedIdentity/userAssignedIdentities/workload" +const defaultStorageID = suppliedGroupID + "/providers/Microsoft.Storage/storageAccounts/stabcdefghijklm" +const defaultWorkloadID = suppliedGroupID + + "/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cog-abcdefghijklm-workload-mi" const suppliedHostingPrincipal = "22222222-2222-4222-8222-222222222222" const suppliedWorkloadPrincipal = "33333333-3333-4333-8333-333333333333" +const suppliedOperator = "44444444-4444-4444-8444-444444444444" + +type suppliedCredential struct{} + +func (suppliedCredential) GetToken(context.Context, policy.TokenRequestOptions) (azcore.AccessToken, error) { + raw := base64.RawURLEncoding.EncodeToString([]byte(`{"oid":"` + suppliedOperator + `"}`)) + return azcore.AccessToken{Token: "header." + raw + ".signature", ExpiresOn: time.Now().Add(time.Hour)}, nil +} + +type suppliedReply struct { + status int + body any +} type suppliedARMFake struct { - mu sync.Mutex - base *byoARMFake - beforeRead func(*http.Request) error - reads []string - identityReply map[string]any - replies map[string]any + mu sync.Mutex + base *byoARMFake + overrides map[string]suppliedReply + existingDefaults map[string]bool + missingGrant string + requests []string + selection *synthesis.AgentHostingInput + reads []string } func (f *suppliedARMFake) Do(request *http.Request) (*http.Response, error) { - if request.Method == http.MethodGet && f.beforeRead != nil { - if err := f.beforeRead(request); err != nil { - return nil, err - } - } f.mu.Lock() defer f.mu.Unlock() - if request.Method == http.MethodGet { - f.reads = append(f.reads, request.URL.Path) - if reply, ok := f.replies[request.URL.Path]; ok { - return f.base.response(request, 200, reply), nil + f.requests = append(f.requests, request.Method+" "+request.URL.Path) + if request.Method != http.MethodGet { + return f.base.Do(request) + } + f.reads = append(f.reads, request.URL.Path) + path := request.URL.Path + lower := strings.ToLower(path) + respond := func(status int, body any) (*http.Response, error) { return f.base.response(request, status, body), nil } + if reply, ok := f.overrides[lower]; ok { + return respond(reply.status, reply.body) + } + if strings.HasPrefix(lower, "/subscriptions/") && len(strings.Split(path, "/")) == 3 { + return respond(200, map[string]any{"id": path, "tenantId": "tenant"}) + } + if strings.EqualFold(path, suppliedGroupID) { + return respond(200, map[string]any{"id": suppliedGroupID, "location": "westus"}) + } + selection := f.base.provider.synthResult.Hosting + if selection == nil { + selection = f.selection + } + if path != "" && strings.EqualFold(path, selection.StorageAccountResourceID) || + strings.EqualFold(path, defaultStorageID) && f.existingDefaults[strings.ToLower(path)] { + sharedKey := strings.EqualFold(path, defaultStorageID) + return respond(200, map[string]any{ + "id": path, "kind": "StorageV2", "sku": map[string]any{"name": "Standard_ZRS"}, + "properties": map[string]any{ + "provisioningState": "Succeeded", "accessTier": "Hot", "isHnsEnabled": false, + "supportsHttpsTrafficOnly": true, "minimumTlsVersion": "TLS1_2", + "allowSharedKeyAccess": sharedKey, "allowBlobPublicAccess": false, + "publicNetworkAccess": "Enabled", "networkAcls": map[string]any{"defaultAction": "Allow"}, + "primaryEndpoints": map[string]any{"blob": "https://customer.blob.core.windows.net/"}, + }, + }) + } + for _, identity := range []struct{ id, principal string }{ + {selection.HostingManagementIdentityResourceID, suppliedHostingPrincipal}, + {selection.WorkloadIdentityResourceID, suppliedWorkloadPrincipal}, + } { + if identity.id != "" && strings.EqualFold(path, identity.id) { + return respond(200, map[string]any{"id": path, "properties": map[string]any{ + "principalId": identity.principal, "clientId": identity.principal, "tenantId": "tenant", + }}) } - for _, identity := range []struct{ id, principal string }{ - {suppliedHostingID, suppliedHostingPrincipal}, - {suppliedWorkloadID, suppliedWorkloadPrincipal}, - } { - if strings.EqualFold(request.URL.Path, identity.id) { - body := f.identityReply - if body == nil { - body = map[string]any{"id": identity.id, "properties": map[string]any{ - "principalId": identity.principal, - }} - } - return f.base.response(request, 200, body), nil - } + } + if strings.EqualFold(path, defaultWorkloadID) && f.existingDefaults[strings.ToLower(path)] { + return respond(200, map[string]any{"id": path}) + } + if strings.HasSuffix(lower, "/roledefinitions") { + return respond(200, map[string]any{"value": []any{map[string]any{ + "id": path + "/aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa", + "properties": map[string]any{ + "roleName": "Managed Identity Federated Identity Credential Contributor", "type": "BuiltInRole", + }, + }}}) + } + if strings.Contains(lower, "/roledefinitions/") { + return respond(200, map[string]any{"id": path, "properties": map[string]any{ + "permissions": []any{map[string]any{"actions": []string{"*"}, "dataActions": []string{"*"}}}, + }}) + } + const suffix = "/providers/Microsoft.Authorization/roleAssignments" + if strings.HasSuffix(lower, strings.ToLower(suffix)) { + scope := path[:len(path)-len(suffix)] + principals := []string{suppliedOperator} + if !strings.EqualFold(scope, f.missingGrant) { + principals = append(principals, suppliedHostingPrincipal, suppliedWorkloadPrincipal) } - if request.URL.Path == "/subscriptions/workload-sub/providers/Microsoft.Authorization/roleDefinitions" { - return f.base.response(request, 200, map[string]any{"value": []any{map[string]any{ - "id": request.URL.Path + "/aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa", + var values []any + for _, principal := range principals { + values = append(values, map[string]any{ + "id": scope + suffix + "/assignment-" + principal, "properties": map[string]any{ - "roleName": "Managed Identity Federated Identity Credential Contributor", "type": "BuiltInRole", + "principalId": principal, "scope": scope, + "roleDefinitionId": "/subscriptions/target-sub/providers/Microsoft.Authorization/roleDefinitions/owner", }, - }}}), nil + }) } + return respond(200, map[string]any{"value": values}) + } + if strings.Contains(lower, "/microsoft.storage/") || strings.Contains(lower, "/microsoft.managedidentity/") { + return respond(404, map[string]any{"error": map[string]string{"code": "ResourceNotFound"}}) } return f.base.Do(request) } @@ -95,131 +163,28 @@ func suppliedYAML(mask int) string { return raw.String() } -func suppliedProvider(t *testing.T, mask int) (*FoundryProvisioningProvider, *suppliedARMFake) { +func suppliedProvider(t *testing.T, mask int) (*FoundryProvisioningProvider, *suppliedARMFake, *stubCompiler) { t.Helper() p, base := offlineBYOProvider(t) result, err := synthesis.Synthesize(synthesis.Input{ServiceName: "project", RawAzureYAML: []byte(suppliedYAML(mask))}) require.NoError(t, err) - p.synthResult = result - fake := &suppliedARMFake{base: base} + p.synthResult, p.credential = result, suppliedCredential{} + fake := &suppliedARMFake{ + base: base, overrides: map[string]suppliedReply{}, existingDefaults: map[string]bool{}, selection: result.Hosting, + } p.armClientOptions = &arm.ClientOptions{ClientOptions: policy.ClientOptions{ Transport: fake, Retry: policy.RetryOptions{MaxRetries: -1}, }} - return p, fake -} - -func TestBYOPrepareConcurrentReads(t *testing.T) { - paths := []string{ - offlineClusterID, - "/subscriptions/workload-sub/providers/Microsoft.Authorization/roleDefinitions", - suppliedHostingID, - suppliedWorkloadID, - } - for _, test := range []struct { - name string - failures []string - cancel bool - }{ - {name: "success"}, - {name: "cluster failure", failures: paths[:1]}, - {name: "role failure", failures: paths[1:2]}, - {name: "hosting identity failure", failures: paths[2:3]}, - {name: "workload identity failure", failures: paths[3:]}, - {name: "all reads fail", failures: paths}, - {name: "role and identity failures", failures: paths[1:]}, - {name: "identity failures", failures: paths[2:]}, - {name: "cancellation", cancel: true}, - } { - t.Run(test.name, func(t *testing.T) { - p, fake := suppliedProvider(t, 7) - source, err := p.resolveTemplate(t.Context(), func(string) {}) - require.NoError(t, err) - source.parameters["includeAcr"] = map[string]any{"value": true} - original, err := json.Marshal(source.parameters) - require.NoError(t, err) - - ctx, cancel := context.WithTimeout(t.Context(), 10*time.Second) - started := make(chan string, len(paths)) - release := make(chan struct{}) - releaseReads := sync.OnceFunc(func() { close(release) }) - readFailure := errors.New("offline binding read failure") - fake.beforeRead = func(request *http.Request) error { - started <- request.URL.Path - select { - case <-request.Context().Done(): - return request.Context().Err() - case <-release: - if slices.Contains(test.failures, request.URL.Path) { - return fmt.Errorf("%s: %w", request.URL.Path, readFailure) - } - if len(test.failures) > 0 { - <-request.Context().Done() - return request.Context().Err() - } - return nil - } - } - done := make(chan struct{}) - var prepareErr error - go func() { - defer close(done) - prepareErr = p.prepareBYO(ctx, source) - }() - t.Cleanup(func() { - cancel() - releaseReads() - <-done - }) - - var observed []string - for range paths { - select { - case path := <-started: - observed = append(observed, path) - case <-ctx.Done(): - t.Fatalf("binding reads did not start concurrently: %v", observed) - } - } - assert.ElementsMatch(t, paths, observed) - if test.cancel { - cancel() - } else { - releaseReads() - } - select { - case <-done: - case <-time.After(5 * time.Second): - t.Fatal("binding reads did not finish") - } - - switch { - case test.cancel: - require.ErrorIs(t, prepareErr, context.Canceled) - case len(test.failures) > 0: - require.ErrorIs(t, prepareErr, readFailure) - assert.True(t, slices.ContainsFunc(test.failures, func(path string) bool { - return strings.Contains(prepareErr.Error(), path) - }), "expected the original error from a failed binding read, got %v", prepareErr) - default: - require.NoError(t, prepareErr) - for name, value := range map[string]string{ - "kubeletObjectId": offlineKubeletObjectID, - "hostingPrincipalId": suppliedHostingPrincipal, - "workloadPrincipalId": suppliedWorkloadPrincipal, - "federatedIdentityRoleDefinitionId": paths[1] + "/aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa", - } { - assert.Equal(t, map[string]any{"value": value}, source.parameters[name]) - } - } - if prepareErr != nil { - current, err := json.Marshal(source.parameters) - require.NoError(t, err) - assert.JSONEq(t, string(original), string(current), "failed reads must not publish partial bindings") - } - assert.Zero(t, fake.base.writes) - assert.Zero(t, fake.base.previews) - }) - } + inner, err := json.Marshal(map[string]any{"parameters": map[string]any{ + "resourceToken": map[string]any{"value": "abcdefghijklm"}, + "result": map[string]any{"value": "abcdefghijklm"}, + }}) + require.NoError(t, err) + outer, err := json.Marshal(map[string]any{"parametersJson": string(inner)}) + require.NoError(t, err) + compiler := &stubCompiler{buildParamResult: bicep.BuildResult{Compiled: string(outer)}} + p.bicepCliInstance = compiler + return p, fake, compiler } func TestBYOSuppliedResourcesDeployIndependently(t *testing.T) { @@ -227,29 +192,22 @@ func TestBYOSuppliedResourcesDeployIndependently(t *testing.T) { for _, registry := range []bool{false, true} { for _, preview := range []bool{false, true} { t.Run(fmt.Sprintf("%d/registry=%t/preview=%t", mask, registry, preview), func(t *testing.T) { - p, fake := suppliedProvider(t, mask) + p, fake, compiler := suppliedProvider(t, mask) p.armTemplate["contentVersion"] = "2.0.0.0" - resources, ok := p.armTemplate["resources"].(map[string]any) - require.True(t, ok) - resourceGroup, ok := resources["resourceGroup"].(map[string]any) - require.True(t, ok) - resourceGroup["name"] = "custom-resource-group" - compiler := &stubCompiler{} - p.bicepCliInstance = compiler p.onDiskSource = &templateSource{ mode: templateModeBicep, armTemplate: p.armTemplate, parameters: map[string]any{"includeAcr": map[string]any{"value": registry}}, } + var progress strings.Builder if preview { - result, err := p.Preview(t.Context(), func(string) {}) + result, err := p.Preview(t.Context(), func(message string) { progress.WriteString(message) }) require.NoError(t, err) require.NotNil(t, result.Preview) assert.Zero(t, fake.base.writes) assert.Equal(t, 1, fake.base.previews) } else { - result, err := p.Deploy(t.Context(), func(string) {}) + result, err := p.Deploy(t.Context(), func(message string) { progress.WriteString(message) }) require.NoError(t, err) - require.NotNil(t, result.Deployment) expected := "" if registry { expected = offlineRegistryID @@ -257,12 +215,14 @@ func TestBYOSuppliedResourcesDeployIndependently(t *testing.T) { assert.Equal(t, expected, result.Deployment.Outputs["AZURE_CONTAINER_REGISTRY_RESOURCE_ID"].Value) assert.Equal(t, 1, fake.base.writes) } - envelope, ok := fake.base.parameters["agentHosting"].(map[string]any) - require.True(t, ok) - raw, err := json.Marshal(envelope["value"]) - require.NoError(t, err) - var hosting map[string]any - require.NoError(t, json.Unmarshal(raw, &hosting)) + assert.Equal(t, "2.0.0.0", fake.base.template["contentVersion"]) + if registry { + assert.Equal(t, map[string]any{"value": offlineKubeletObjectID}, + fake.base.parameters["kubeletObjectId"]) + } else { + assert.NotContains(t, fake.base.parameters, "kubeletObjectId") + } + hosting := fake.base.parameters["agentHosting"].(map[string]any)["value"].(map[string]any) for i, selection := range []struct{ name, id string }{ {"storageAccountResourceId", suppliedStorageID}, {"hostingManagementIdentityResourceId", suppliedHostingID}, @@ -275,207 +235,162 @@ func TestBYOSuppliedResourcesDeployIndependently(t *testing.T) { assert.Equal(t, expected, hosting[selection.name]) } for _, identity := range []struct { - id, binding, principal string - selected bool + name, principal string + selected bool }{ - {suppliedHostingID, "hostingPrincipalId", suppliedHostingPrincipal, mask&2 != 0}, - {suppliedWorkloadID, "workloadPrincipalId", suppliedWorkloadPrincipal, mask&4 != 0}, + {"hostingPrincipalId", suppliedHostingPrincipal, mask&2 != 0}, + {"workloadPrincipalId", suppliedWorkloadPrincipal, mask&4 != 0}, } { if identity.selected { - assert.Equal(t, map[string]any{"value": identity.principal}, - fake.base.parameters[identity.binding]) - assert.Contains(t, fake.reads, identity.id) + assert.Equal(t, identity.principal, + fake.base.parameters[identity.name].(map[string]any)["value"]) } else { - assert.NotContains(t, fake.base.parameters, identity.binding) - assert.NotContains(t, fake.reads, identity.id) + assert.NotContains(t, fake.base.parameters, identity.name) } } - roleSubscription := "target-sub" - if mask&4 != 0 { - roleSubscription = "workload-sub" - } - assert.Equal(t, map[string]any{"value": "/subscriptions/" + roleSubscription + - "/providers/Microsoft.Authorization/roleDefinitions/aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa"}, - fake.base.parameters["federatedIdentityRoleDefinitionId"]) - assert.Contains(t, fake.reads, offlineClusterID) - if registry { - assert.Contains(t, fake.base.parameters, "kubeletObjectId") + assert.NotEmpty(t, compiler.buildParamCalls, "the actual Foundry account name uses the resource token") + assert.Contains(t, progress.String(), "S03") + assert.Contains(t, progress.String(), "P02 PASS") + if mask&2 == 0 { + assert.Contains(t, progress.String(), "P09 PASS ["+offlineClusterGroupID+"]") } else { - assert.NotContains(t, fake.base.parameters, "kubeletObjectId") + assert.NotContains(t, progress.String(), "P09 PASS ["+offlineClusterGroupID+"]") } - assert.Empty(t, compiler.buildCalls) - assert.Empty(t, compiler.buildParamCalls) assert.Empty(t, p.virtualEnv) - assert.Equal(t, p.armTemplate, fake.base.template) - }) - } - } - } -} - -func TestBYOSuppliedBindingsPreserveExplicitValues(t *testing.T) { - for _, binding := range []string{"hostingPrincipalId", "workloadPrincipalId", "federatedIdentityRoleDefinitionId"} { - mask := 0 - switch binding { - case "hostingPrincipalId": - mask = 2 - case "workloadPrincipalId": - mask = 4 - } - for _, declared := range []bool{false, true} { - for _, preview := range []bool{false, true} { - t.Run(fmt.Sprintf("%s/declared=%t/preview=%t", binding, declared, preview), func(t *testing.T) { - p, fake := suppliedProvider(t, mask) - source, err := p.resolveTemplate(t.Context(), func(string) {}) - require.NoError(t, err) - explicit := map[string]any{"value": "explicit-" + binding} - parameterName := strings.ToUpper(binding) - source.parameters[parameterName] = explicit - if !declared { - definitions, ok := source.armTemplate["parameters"].(map[string]any) - require.True(t, ok) - delete(definitions, binding) - } - p.onDiskSource = source + method := "PUT " if preview { - _, err = p.Preview(t.Context(), func(string) {}) - } else { - _, err = p.Deploy(t.Context(), func(string) {}) + method = "POST " } - if declared { - require.NoError(t, err) - } else { - require.ErrorContains(t, err, "InvalidTemplate") - require.ErrorContains(t, err, "offline ARM undeclared parameter: "+parameterName) - assert.Zero(t, fake.base.writes) - assert.Zero(t, fake.base.previews) - } - assert.Equal(t, explicit, fake.base.parameters[parameterName]) - assert.NotContains(t, fake.base.parameters, binding) - assert.Contains(t, fake.reads, offlineClusterID) - switch binding { - case "hostingPrincipalId": - assert.NotContains(t, fake.reads, suppliedHostingID) - case "workloadPrincipalId": - assert.NotContains(t, fake.reads, suppliedWorkloadID) - case "federatedIdentityRoleDefinitionId": - assert.NotContains(t, fake.reads, - "/subscriptions/target-sub/providers/Microsoft.Authorization/roleDefinitions") + submissions := 0 + for _, request := range fake.requests { + if strings.HasPrefix(request, method) { + submissions++ + } else { + assert.True(t, strings.HasPrefix(request, "GET "), request) + } } + assert.Equal(t, 1, submissions) }) } } } } -func TestBYOSuppliedUsesEffectiveDeploymentInputs(t *testing.T) { - for _, name := range []string{ - "parameters", "template default", "without YAML", "disabled", "undeclared", "explicit bindings", - } { - t.Run(name, func(t *testing.T) { - p, fake := suppliedProvider(t, 7) - clusterID := offlineClusterID + "-effective" - hostingID := suppliedHostingID + "-effective" - workloadID := strings.Replace(suppliedWorkloadID, "workload-sub", "effective-sub", 1) + "-effective" - hosting := map[string]any{ - "enabled": true, "hostingType": "custom", "name": "custom", - "clusterResourceId": clusterID, "storageAccountResourceId": suppliedStorageID + "-effective", - "hostingManagementIdentityResourceId": hostingID, "workloadIdentityResourceId": workloadID, - "customSetting": true, - } - parameters := map[string]any{ - "AGENTHOSTING": map[string]any{"value": hosting}, - "includeAcr": map[string]any{"value": true}, - "location": map[string]any{"value": "[parameters('customRegion')]"}, - "resourceTokenSalt": map[string]any{"value": nil}, - "resourceGroupName": map[string]any{"value": false}, - } - definitions, ok := p.armTemplate["parameters"].(map[string]any) - require.True(t, ok) - switch name { - case "template default", "without YAML": - result, err := synthesis.Synthesize(synthesis.Input{ServiceName: "project", - RawAzureYAML: []byte("services:\n project:\n host: azure.ai.project\n")}) - require.NoError(t, err) - p.synthResult = result - if name == "template default" { - delete(parameters, "AGENTHOSTING") - definition, ok := definitions["agentHosting"].(map[string]any) - require.True(t, ok) - definition["defaultValue"] = hosting +func TestBYOSuppliedPreflightBlocksDeployAndWhatIfBeforeMutations(t *testing.T) { + for _, preview := range []bool{false, true} { + for _, failure := range []string{"missing Storage", "forbidden Storage", "missing identity", "tenant", "grant"} { + t.Run(fmt.Sprintf("%s/preview=%t", failure, preview), func(t *testing.T) { + p, fake, _ := suppliedProvider(t, 7) + switch failure { + case "missing Storage": + fake.overrides[strings.ToLower(suppliedStorageID)] = suppliedReply{404, + map[string]any{"error": map[string]string{"code": "ResourceNotFound"}}} + case "forbidden Storage": + fake.overrides[strings.ToLower(suppliedStorageID)] = suppliedReply{403, + map[string]any{"error": map[string]string{"code": "AuthorizationFailed"}}} + case "missing identity": + fake.overrides[strings.ToLower(suppliedHostingID)] = suppliedReply{404, + map[string]any{"error": map[string]string{"code": "ResourceNotFound"}}} + case "tenant": + fake.overrides["/subscriptions/cluster-sub"] = suppliedReply{200, + map[string]any{"tenantId": "other-tenant"}} + case "grant": + fake.missingGrant = suppliedWorkloadID } - case "disabled": - hosting["enabled"] = false - hosting["clusterResourceId"] = 42 - case "undeclared": - delete(definitions, "agentHosting") - parameters["AGENTHOSTING"] = map[string]any{"value": "leave undeclared parameters to ARM"} - case "explicit bindings": - hosting["hostingManagementIdentityResourceId"] = map[string]any{"custom": true} - hosting["workloadIdentityResourceId"] = 42 - for _, binding := range []string{ - "hostingPrincipalId", "workloadPrincipalId", "federatedIdentityRoleDefinitionId", "kubeletObjectId", - } { - parameters[binding] = map[string]any{"value": "explicit-" + binding} + var err error + if preview { + _, err = p.Preview(t.Context(), func(string) {}) + } else { + _, err = p.Deploy(t.Context(), func(string) {}) } - } - p.onDiskSource = &templateSource{ - mode: templateModeBicep, armTemplate: p.armTemplate, parameters: parameters, - } - rolePath := "/subscriptions/effective-sub/providers/Microsoft.Authorization/roleDefinitions" - fake.replies = map[string]any{ - clusterID: map[string]any{"properties": fake.base.clusterProperties}, - hostingID: map[string]any{ - "id": hostingID, "properties": map[string]any{"principalId": suppliedHostingPrincipal}, - }, - workloadID: map[string]any{ - "id": workloadID, "properties": map[string]any{"principalId": suppliedWorkloadPrincipal}, - }, - rolePath: map[string]any{"value": []any{map[string]any{ - "id": rolePath + "/aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa", - "properties": map[string]any{ - "roleName": "Managed Identity Federated Identity Credential Contributor", "type": "BuiltInRole", - }, - }}}, - } - source, err := p.resolveTemplate(t.Context(), func(string) {}) - require.NoError(t, err) - require.NoError(t, p.prepareBYO(t.Context(), source)) - for parameter, value := range parameters { - assert.Equal(t, value, source.parameters[parameter]) - } - assert.NotContains(t, source.parameters, "agentHosting", "the original object or template default is preserved") - switch name { - case "disabled", "undeclared": - assert.Empty(t, fake.reads) - assert.NotContains(t, source.parameters, "kubeletObjectId") - assert.NotContains(t, source.parameters, "federatedIdentityRoleDefinitionId") - case "explicit bindings": - assert.Equal(t, []string{clusterID}, fake.reads) - default: - assert.ElementsMatch(t, []string{clusterID, rolePath, hostingID, workloadID}, fake.reads) - assert.Equal(t, map[string]any{"value": suppliedHostingPrincipal}, source.parameters["hostingPrincipalId"]) - assert.Equal(t, map[string]any{"value": suppliedWorkloadPrincipal}, source.parameters["workloadPrincipalId"]) - assert.Equal(t, map[string]any{"value": offlineKubeletObjectID}, source.parameters["kubeletObjectId"]) - assert.Equal(t, map[string]any{"value": rolePath + "/aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa"}, - source.parameters["federatedIdentityRoleDefinitionId"]) - } - assert.Empty(t, p.virtualEnv) - }) + require.ErrorContains(t, err, "preflight blocked") + assert.Zero(t, fake.base.writes) + assert.Zero(t, fake.base.previews) + assert.Empty(t, p.virtualEnv) + for _, request := range fake.requests { + assert.True(t, strings.HasPrefix(request, "GET "), request) + } + }) + } } } -func TestBYOSuppliedIdentityRequiresRealPrincipalBinding(t *testing.T) { - for _, body := range []map[string]any{ - {"id": suppliedHostingID}, - {"id": suppliedHostingID, "properties": map[string]any{"principalId": "invalid"}}, - {"id": suppliedWorkloadID, "properties": map[string]any{"principalId": suppliedHostingPrincipal}}, - } { - p, fake := suppliedProvider(t, 2) - fake.identityReply = body - _, err := p.Deploy(t.Context(), func(string) {}) - require.Error(t, err) - assert.Zero(t, fake.base.writes) - assert.Zero(t, fake.base.previews) +func TestBYOSuppliedPreviewUsesNormalWhatIfAfterReadOnlyChecks(t *testing.T) { + p, fake, _ := suppliedProvider(t, 7) + result, err := p.Preview(t.Context(), func(string) {}) + require.NoError(t, err) + require.Len(t, result.Preview.Changes, 1) + assert.Equal(t, 1, fake.base.previews) + assert.Zero(t, fake.base.writes) + assert.True(t, strings.HasSuffix(fake.requests[len(fake.requests)-1], "/whatIf")) +} + +func TestBYOExistingDefaultStorageRetainsOrdinaryARMUpdate(t *testing.T) { + p, fake, _ := suppliedProvider(t, 2) + fake.existingDefaults[strings.ToLower(defaultStorageID)] = true + var progress strings.Builder + _, err := p.Deploy(t.Context(), func(message string) { progress.WriteString(message) }) + require.NoError(t, err) + hosting := fake.base.parameters["agentHosting"].(map[string]any)["value"].(map[string]any) + assert.Equal(t, "", hosting["storageAccountResourceId"]) + assert.Contains(t, progress.String(), "S01 NOT_APPLICABLE") + assert.Equal(t, 1, fake.base.writes) +} + +func TestBYOExistingDefaultDependencyMissingGrantBlocksRetry(t *testing.T) { + p, fake, _ := suppliedProvider(t, 2) + fake.base.failWrite = true + _, err := p.Deploy(t.Context(), func(string) {}) + require.ErrorContains(t, err, "offline deployment failure") + assert.Equal(t, 1, fake.base.writes) + fake.base.failWrite = false + fake.existingDefaults[strings.ToLower(defaultWorkloadID)] = true + fake.missingGrant = defaultWorkloadID + _, err = p.Deploy(t.Context(), func(string) {}) + require.ErrorContains(t, err, "G02 FAIL") + assert.Equal(t, 1, fake.base.writes, "no historical failed-attempt exemption may open the write gate") +} + +func TestBYOSuppliedWorkloadUsesItsOwnSubscriptionForFederationRole(t *testing.T) { + p, fake, _ := suppliedProvider(t, 7) + remote := strings.Replace(suppliedWorkloadID, "/target-sub/", "/identity-sub/", 1) + result, err := synthesis.Synthesize(synthesis.Input{ServiceName: "project", + RawAzureYAML: []byte(strings.Replace(suppliedYAML(7), suppliedWorkloadID, remote, 1))}) + require.NoError(t, err) + p.synthResult = result + _, err = p.Deploy(t.Context(), func(string) {}) + require.NoError(t, err) + assert.Equal(t, "/subscriptions/identity-sub/providers/Microsoft.Authorization/"+ + "roleDefinitions/aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa", + fake.base.parameters["federatedIdentityRoleDefinitionId"].(map[string]any)["value"]) +} + +func TestBYOSuppliedExplicitPrincipalDoesNotOverwriteUserInput(t *testing.T) { + p, fake, _ := suppliedProvider(t, 7) + p.onDiskSource = &templateSource{ + mode: templateModeBicep, armTemplate: p.armTemplate, + parameters: map[string]any{"hostingPrincipalId": map[string]any{"value": suppliedOperator}}, } + _, err := p.Deploy(t.Context(), func(string) {}) + require.NoError(t, err) + assert.Equal(t, 1, fake.base.writes) + assert.Equal(t, map[string]any{"value": suppliedOperator}, fake.base.parameters["hostingPrincipalId"]) + assert.Equal(t, suppliedOperator, p.onDiskSource.parameters["hostingPrincipalId"].(map[string]any)["value"]) + assert.Empty(t, p.virtualEnv) +} + +func TestBYODisabledSupportingSelectionsRetainARMPrecedence(t *testing.T) { + p, fake, _ := suppliedProvider(t, 7) + p.synthResult.Hosting = nil + p.credential = nil + source := &templateSource{armTemplate: p.armTemplate, parameters: map[string]any{ + "agentHosting": map[string]any{"value": map[string]any{ + "enabled": false, "storageAccountResourceId": suppliedStorageID, + }}, + }} + require.NoError(t, p.prepareBYO(t.Context(), source, func(string) {})) + assert.Equal(t, map[string]any{"value": map[string]any{ + "enabled": false, "storageAccountResourceId": suppliedStorageID, + }}, source.parameters["agentHosting"]) + assert.Empty(t, fake.requests) } diff --git a/cli/azd/extensions/azure.ai.projects/internal/synthesis/hosting_test.go b/cli/azd/extensions/azure.ai.projects/internal/synthesis/hosting_test.go index 19754cdfeea..7db2cd836a5 100644 --- a/cli/azd/extensions/azure.ai.projects/internal/synthesis/hosting_test.go +++ b/cli/azd/extensions/azure.ai.projects/internal/synthesis/hosting_test.go @@ -11,6 +11,8 @@ import ( "strings" "testing" + "azure.ai.projects/internal/testutil" + "github.com/azure/azure-dev/cli/azd/pkg/tools/bicep" "github.com/santhosh-tekuri/jsonschema/v6" "github.com/stretchr/testify/assert" @@ -633,10 +635,10 @@ func TestHostingGraphIndependentlyReferencesSuppliedResources(t *testing.T) { } func TestBicepToolVersionAndExplicitOverride(t *testing.T) { - require.NoError(t, validateBicepVersion("Bicep CLI version "+bicep.Version.String()+" (hash)")) - require.ErrorContains(t, validateBicepVersion("Bicep CLI version 0.38.33 (hash)"), "below azd SDK minimum") - require.ErrorContains(t, validateBicepVersion("other CLI version 99.0.0"), "unrecognized Bicep") - path, err := findBicep(t.Context(), filepath.Join(t.TempDir(), "missing-bicep")) + require.NoError(t, testutil.ValidateBicepVersion("Bicep CLI version "+bicep.Version.String()+" (hash)")) + require.ErrorContains(t, testutil.ValidateBicepVersion("Bicep CLI version 0.38.33 (hash)"), "below azd SDK minimum") + require.ErrorContains(t, testutil.ValidateBicepVersion("other CLI version 99.0.0"), "unrecognized Bicep") + path, err := testutil.FindBicep(t.Context(), filepath.Join(t.TempDir(), "missing-bicep")) require.Error(t, err) assert.Empty(t, path) } diff --git a/cli/azd/extensions/azure.ai.projects/internal/synthesis/schema_test.go b/cli/azd/extensions/azure.ai.projects/internal/synthesis/schema_test.go index a3dad4ade3c..3e537c0c39c 100644 --- a/cli/azd/extensions/azure.ai.projects/internal/synthesis/schema_test.go +++ b/cli/azd/extensions/azure.ai.projects/internal/synthesis/schema_test.go @@ -5,21 +5,15 @@ package synthesis import ( "bytes" - "context" "encoding/json" "errors" - "fmt" "os" "os/exec" "path/filepath" - "regexp" "slices" - "strings" "testing" - "time" - "github.com/azure/azure-dev/cli/azd/pkg/tools" - "github.com/azure/azure-dev/cli/azd/pkg/tools/bicep" + "azure.ai.projects/internal/testutil" "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" ) @@ -100,7 +94,7 @@ func assertNetworkRejectsAgentSubnetWithIsolationMode(t *testing.T, allOf []json // this catches a forgotten `bicep build`. Skipped when the bicep CLI is not on // PATH (e.g. minimal CI images) so it never produces a phantom failure. func TestARMTemplate_MatchesBicepBuild(t *testing.T) { - bicep := lookupBicep(t) + bicep := testutil.LookupBicep(t) templatesDir := "templates" committed, err := os.ReadFile(filepath.Join(templatesDir, "main.arm.json")) @@ -130,7 +124,7 @@ func TestARMTemplate_MatchesBicepBuild(t *testing.T) { // of TestARMTemplate_MatchesBicepBuild: it catches a forgotten `bicep build` after // editing the existing-project template. Skipped when bicep is absent. func TestExistingProjectARMTemplate_MatchesBicepBuild(t *testing.T) { - bicep := lookupBicep(t) + bicep := testutil.LookupBicep(t) templatesDir := "templates" committed, err := os.ReadFile(filepath.Join(templatesDir, "existing-project.arm.json")) @@ -189,69 +183,3 @@ func stripBicepGeneratorMetadata(value any) { } } - -func lookupBicep(t *testing.T) string { - t.Helper() - override := os.Getenv("AZD_BICEP_TOOL_PATH") - path, err := findBicep(t.Context(), override) - if err != nil { - if override != "" { - t.Fatal(err) - } - t.Skip(err) - } - return path -} - -func findBicep(ctx context.Context, override string) (string, error) { - var candidates []string - if override != "" { - candidates = append(candidates, override) - } else { - if path, err := exec.LookPath("bicep"); err == nil { - candidates = append(candidates, path) - } - if home, err := os.UserHomeDir(); err == nil { - for _, name := range []string{"bicep", "bicep.exe"} { - path := filepath.Join(home, ".azure", "bin", name) - if _, err := os.Stat(path); err == nil { - candidates = append(candidates, path) - } - } - } - } - var rejected []string - for _, path := range candidates { - ctx, cancel := context.WithTimeout(ctx, 10*time.Second) - //nolint:gosec // Explicit override or locally discovered Bicep; never installs or upgrades tools. - output, err := exec.CommandContext(ctx, path, "--version").CombinedOutput() - cancel() - if err != nil { - err = fmt.Errorf("read Bicep version: %w: %s", err, strings.TrimSpace(string(output))) - } else { - err = validateBicepVersion(string(output)) - } - if err == nil { - return path, nil - } - rejected = append(rejected, fmt.Sprintf("%s: %v", path, err)) - } - return "", fmt.Errorf("no supported Bicep CLI found (azd SDK minimum %s); "+ - "set AZD_BICEP_TOOL_PATH to a supported binary or add it to PATH; rejected: %s", - bicep.Version, strings.Join(rejected, "; ")) -} - -func validateBicepVersion(output string) error { - match := regexp.MustCompile(`^Bicep CLI version (\d+\.\d+\.\d+)(?:\s|$)`).FindStringSubmatch(strings.TrimSpace(output)) - if len(match) != 2 { - return fmt.Errorf("unrecognized Bicep version output %q", strings.TrimSpace(output)) - } - version, err := tools.ExtractVersion(match[1]) - if err != nil { - return fmt.Errorf("parse Bicep version: %w", err) - } - if version.LT(bicep.Version) { - return fmt.Errorf("Bicep version %s is below azd SDK minimum %s", version, bicep.Version) - } - return nil -} diff --git a/cli/azd/extensions/azure.ai.projects/internal/synthesis/templates_embed.go b/cli/azd/extensions/azure.ai.projects/internal/synthesis/templates_embed.go index 66feb2ea6d3..f13b8c9bdb7 100644 --- a/cli/azd/extensions/azure.ai.projects/internal/synthesis/templates_embed.go +++ b/cli/azd/extensions/azure.ai.projects/internal/synthesis/templates_embed.go @@ -7,8 +7,9 @@ import "embed" // Re-generate main.arm.json from main.bicep. Run via `go generate ./...` // from the extension root after editing any *.bicep file under templates/. -// The compiled ARM JSON is embedded so the provisioning provider never -// needs a bicep CLI at user runtime. +// Deployment uses embedded ARM JSON. Preflight may use azd-managed Bicep for +// constant name expressions in the actual compiled plan, including plans +// that reference supplied resources. // //go:generate bicep build templates/main.bicep --outfile templates/main.arm.json //go:generate bicep build templates/existing-project.bicep --outfile templates/existing-project.arm.json diff --git a/cli/azd/extensions/azure.ai.projects/internal/testutil/bicep.go b/cli/azd/extensions/azure.ai.projects/internal/testutil/bicep.go new file mode 100644 index 00000000000..e1bd71eb3d2 --- /dev/null +++ b/cli/azd/extensions/azure.ai.projects/internal/testutil/bicep.go @@ -0,0 +1,88 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +package testutil + +import ( + "context" + "fmt" + "os" + "os/exec" + "path/filepath" + "regexp" + "strings" + "testing" + "time" + + "github.com/azure/azure-dev/cli/azd/pkg/tools" + "github.com/azure/azure-dev/cli/azd/pkg/tools/bicep" +) + +// LookupBicep finds a supported local tool, failing invalid overrides and skipping unavailable automatic discovery. +func LookupBicep(t *testing.T) string { + t.Helper() + override := os.Getenv("AZD_BICEP_TOOL_PATH") + path, err := FindBicep(t.Context(), override) + if err != nil { + if override != "" { + t.Fatal(err) + } + t.Skip(err) + } + return path +} + +// FindBicep checks the explicit override, or PATH and the Azure CLI bundled location, without installing tools. +func FindBicep(ctx context.Context, override string) (string, error) { + var candidates []string + if override != "" { + candidates = append(candidates, override) + } else { + if path, err := exec.LookPath("bicep"); err == nil { + candidates = append(candidates, path) + } + if home, err := os.UserHomeDir(); err == nil { + for _, name := range []string{"bicep", "bicep.exe"} { + path := filepath.Join(home, ".azure", "bin", name) + if _, err := os.Stat(path); err == nil { + candidates = append(candidates, path) + } + } + } + } + var rejected []string + for _, path := range candidates { + ctx, cancel := context.WithTimeout(ctx, 10*time.Second) + //nolint:gosec // Explicit override or locally discovered Bicep; never installs or upgrades tools. + output, err := exec.CommandContext(ctx, path, "--version").CombinedOutput() + cancel() + if err != nil { + err = fmt.Errorf("read Bicep version: %w: %s", err, strings.TrimSpace(string(output))) + } else { + err = ValidateBicepVersion(string(output)) + } + if err == nil { + return path, nil + } + rejected = append(rejected, fmt.Sprintf("%s: %v", path, err)) + } + return "", fmt.Errorf("no supported Bicep CLI found (azd SDK minimum %s); "+ + "set AZD_BICEP_TOOL_PATH to a supported binary or add it to PATH; rejected: %s", + bicep.Version, strings.Join(rejected, "; ")) +} + +// ValidateBicepVersion verifies the CLI output against the referenced azd SDK minimum. +func ValidateBicepVersion(output string) error { + match := regexp.MustCompile(`^Bicep CLI version (\d+\.\d+\.\d+)(?:\s|$)`).FindStringSubmatch(strings.TrimSpace(output)) + if len(match) != 2 { + return fmt.Errorf("unrecognized Bicep version output %q", strings.TrimSpace(output)) + } + version, err := tools.ExtractVersion(match[1]) + if err != nil { + return fmt.Errorf("parse Bicep version: %w", err) + } + if version.LT(bicep.Version) { + return fmt.Errorf("Bicep version %s is below azd SDK minimum %s", version, bicep.Version) + } + return nil +} diff --git a/docs/architecture/extension-framework.md b/docs/architecture/extension-framework.md index 9be0e41e746..6737446c2fe 100644 --- a/docs/architecture/extension-framework.md +++ b/docs/architecture/extension-framework.md @@ -86,6 +86,15 @@ Extensions declare their capabilities in `extension.yaml`: | `provisioning-provider` | Add custom infrastructure provisioning support | | `metadata` | Provide metadata about commands and capabilities | +The first-party `microsoft.foundry` provisioning provider supports existing AKS hosting +with independently supplied Storage and hosting/workload identities. Read-only AKS, +selected Agent-subnet, supporting-resource and authorization preflight gates deployment and +ARM what-if for default and supplied-resource plans. Editable templates use their actual +compiled resource plan; unresolved dependencies warn and skip only the affected checks. +Omitted resources retain their normal ARM creation/update path and registry selection follows the standard Foundry rules. See +[Existing AKS hosting](../../cli/azd/extensions/azure.ai.projects/README.md#existing-aks-hosting) +for the preview opt-in, checks, current-state retry semantics, and resource-group teardown scope. + ## Available gRPC Services Extensions can access these azd services via gRPC: diff --git a/docs/guides/creating-an-extension.md b/docs/guides/creating-an-extension.md index 0a6d72b2194..4cc83f65e7c 100644 --- a/docs/guides/creating-an-extension.md +++ b/docs/guides/creating-an-extension.md @@ -51,6 +51,14 @@ capabilities: Implement the required interfaces for your declared capabilities. See the extension framework services documentation for interface details. +For a provisioning-provider example that combines supplied references with independently +created defaults, see [Foundry existing AKS hosting](../../cli/azd/extensions/azure.ai.projects/README.md#existing-aks-hosting). +Its AKS, selected Agent-subnet, supporting-resource and authorization preflight is read-only +and runs before deployment or ARM what-if, including when supporting resources use defaults. +The editable-template path identifies actual resources and scopes, warning and skipping +only checks whose required plan information cannot be resolved. Supplied-resource references +do not imply protection from resource-group teardown. + If your Go extension creates role assignments, use the stable [`Account().GetCurrentPrincipal`](../../cli/azd/docs/extensions/extension-framework.md#getcurrentprincipal) method with the target subscription ID and request types from `azdext` or `contracts/v1`. The host resolves the resource-tenant object ID and principal type without returning an access token. Pin the published azd SDK to v1.36.0 or later and set `requiredAzdVersion: ">=1.36.0"` before replacing an existing lookup. The equivalent beta API is deprecated. Go extensions that resolve local Foundry configuration `$ref` files should use [`foundry.ResolveFileRefs`](../../cli/azd/pkg/foundry/includes.go). Each referenced file must contain exactly one YAML or JSON object; additional documents, trailing content, arrays, and scalars are rejected. Resolution retains YAML value types, aliases, and sibling overlays. Pass only the selected configuration when unrelated references should remain unopened. Extensions must consume an SDK release containing this validation before their binaries enforce it. diff --git a/docs/reference/feature-status.md b/docs/reference/feature-status.md index adb3723a455..5b4009d80ca 100644 --- a/docs/reference/feature-status.md +++ b/docs/reference/feature-status.md @@ -53,6 +53,7 @@ Current maturity status of Azure Developer CLI features. See [Feature Stages](.. | Azure Functions | Stable | | Azure Kubernetes Service (AKS) | Beta | | Azure AI | Beta | +| [Microsoft Foundry on existing AKS](../../cli/azd/extensions/azure.ai.projects/README.md#existing-aks-hosting) (default or supplied supporting resources, read-only preflight) | In development (infrastructure enabled by default; Agents init/adoption disabled by default) | ## Clients