Add Cluster Policy support to bundle resources #6225
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| Add support for the `cluster_policies` resource type in Declarative Automation Bundles. Cluster policies are only supported in direct deployment mode. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| bundle: | ||
| name: test-bundle-$UNIQUE_NAME | ||
|
|
||
| resources: | ||
| cluster_policies: | ||
| foo: | ||
| name: test-cluster-policy-$UNIQUE_NAME | ||
| definition: '{"spark_version":{"type":"fixed","value":"13.3.x-scala2.12"}}' |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| bundle: | ||
| name: test_cluster_policy | ||
|
|
||
| resources: | ||
| cluster_policies: | ||
| test_cluster_policy: | ||
| name: my_cluster_policy | ||
| definition: '{"spark_version":{"type":"fixed","value":"13.3.x-scala2.12"}}' |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,126 @@ | ||
|
|
||
| >>> [CLI] bundle validate | ||
| Name: test_cluster_policy | ||
| Target: default | ||
| Workspace: | ||
| User: [USERNAME] | ||
| Path: /Workspace/Users/[USERNAME]/.bundle/test_cluster_policy/default | ||
|
|
||
| Validation OK! | ||
|
|
||
| >>> [CLI] bundle validate -o json | ||
| { | ||
| "test_cluster_policy": { | ||
| "definition": "{\"spark_version\":{\"type\":\"fixed\",\"value\":\"13.3.x-scala2.12\"}}", | ||
| "name": "my_cluster_policy" | ||
| } | ||
| } | ||
|
|
||
| >>> [CLI] bundle summary | ||
| Name: test_cluster_policy | ||
| Target: default | ||
| Workspace: | ||
| User: [USERNAME] | ||
| Path: /Workspace/Users/[USERNAME]/.bundle/test_cluster_policy/default | ||
| Resources: | ||
| Cluster Policies: | ||
| test_cluster_policy: | ||
| Name: my_cluster_policy | ||
| URL: (not deployed) | ||
|
|
||
| >>> [CLI] bundle deploy | ||
| Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test_cluster_policy/default/files... | ||
| Deploying resources... | ||
| Updating deployment state... | ||
| Deployment complete! | ||
|
|
||
| === Verify the create request | ||
| >>> jq select(.method == "POST" and (.path | contains("/policies/clusters/create"))) out.requests.txt | ||
| { | ||
| "method": "POST", | ||
| "path": "/api/2.0/policies/clusters/create", | ||
| "body": { | ||
| "definition": "{\"spark_version\":{\"type\":\"fixed\",\"value\":\"13.3.x-scala2.12\"}}", | ||
| "name": "my_cluster_policy" | ||
| } | ||
| } | ||
|
|
||
| >>> [CLI] bundle summary | ||
| Name: test_cluster_policy | ||
| Target: default | ||
| Workspace: | ||
| User: [USERNAME] | ||
| Path: /Workspace/Users/[USERNAME]/.bundle/test_cluster_policy/default | ||
| Resources: | ||
| Cluster Policies: | ||
| test_cluster_policy: | ||
| Name: my_cluster_policy | ||
| URL: [DATABRICKS_URL]/compute/policies/[UUID]?w=[NUMID] | ||
|
|
||
| === Update the cluster policy name | ||
| >>> update_file.py databricks.yml my_cluster_policy my_cluster_policy_2 | ||
|
|
||
| >>> [CLI] bundle deploy | ||
| Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test_cluster_policy/default/files... | ||
| Deploying resources... | ||
| Updating deployment state... | ||
| Deployment complete! | ||
|
|
||
| === Verify the update request | ||
| >>> jq select(.method == "POST" and (.path | contains("/policies/clusters/edit"))) out.requests.txt | ||
| { | ||
| "method": "POST", | ||
| "path": "/api/2.0/policies/clusters/edit", | ||
| "body": { | ||
| "definition": "{\"spark_version\":{\"type\":\"fixed\",\"value\":\"13.3.x-scala2.12\"}}", | ||
| "name": "my_cluster_policy_2", | ||
| "policy_id": "[UUID]" | ||
| } | ||
| } | ||
|
|
||
| >>> [CLI] bundle summary | ||
| Name: test_cluster_policy | ||
| Target: default | ||
| Workspace: | ||
| User: [USERNAME] | ||
| Path: /Workspace/Users/[USERNAME]/.bundle/test_cluster_policy/default | ||
| Resources: | ||
| Cluster Policies: | ||
| test_cluster_policy: | ||
| Name: my_cluster_policy_2 | ||
| URL: [DATABRICKS_URL]/compute/policies/[UUID]?w=[NUMID] | ||
|
|
||
| === Destroy the cluster policy | ||
| >>> [CLI] bundle destroy --auto-approve | ||
| The following resources will be deleted: | ||
| delete resources.cluster_policies.test_cluster_policy | ||
|
|
||
| All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/test_cluster_policy/default | ||
|
|
||
| Deleting files... | ||
| Destroy complete! | ||
|
|
||
| === Verify the destroy request | ||
| >>> jq select(.method == "POST" and (.path | contains("/policies/clusters/delete"))) out.requests.txt | ||
| { | ||
| "method": "POST", | ||
| "path": "/api/2.0/policies/clusters/delete", | ||
| "body": { | ||
| "policy_id": "[UUID]" | ||
| } | ||
| } | ||
|
|
||
| >>> [CLI] bundle summary | ||
| Name: test_cluster_policy | ||
| Target: default | ||
| Workspace: | ||
| User: [USERNAME] | ||
| Path: /Workspace/Users/[USERNAME]/.bundle/test_cluster_policy/default | ||
| Resources: | ||
| Cluster Policies: | ||
| test_cluster_policy: | ||
| Name: my_cluster_policy_2 | ||
| URL: (not deployed) | ||
|
|
||
| >>> [CLI] bundle destroy --auto-approve | ||
| No active deployment found to destroy! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,33 @@ | ||
| trace $CLI bundle validate | ||
| trace $CLI bundle validate -o json | jq ".resources.cluster_policies" | ||
|
|
||
| trace $CLI bundle summary | ||
|
|
||
| cleanup() { | ||
| trace $CLI bundle destroy --auto-approve | ||
| rm out.requests.txt | ||
| } | ||
| trap cleanup EXIT | ||
| trace $CLI bundle deploy | ||
|
|
||
| title "Verify the create request" | ||
| trace jq 'select(.method == "POST" and (.path | contains("/policies/clusters/create")))' out.requests.txt | ||
|
|
||
| trace $CLI bundle summary | ||
|
|
||
| title "Update the cluster policy name" | ||
| trace update_file.py databricks.yml my_cluster_policy my_cluster_policy_2 | ||
| trace $CLI bundle deploy | ||
|
|
||
| title "Verify the update request" | ||
| trace jq 'select(.method == "POST" and (.path | contains("/policies/clusters/edit")))' out.requests.txt | ||
|
|
||
| trace $CLI bundle summary | ||
|
|
||
| title "Destroy the cluster policy" | ||
| trace $CLI bundle destroy --auto-approve | ||
|
|
||
| title "Verify the destroy request" | ||
| trace jq 'select(.method == "POST" and (.path | contains("/policies/clusters/delete")))' out.requests.txt | ||
|
|
||
| trace $CLI bundle summary | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| bundle: | ||
| name: cluster_policy_producer | ||
|
|
||
| resources: | ||
| cluster_policies: | ||
| pol: | ||
| name: shared_policy | ||
| definition: '{"spark_version":{"type":"fixed","value":"13.3.x-scala2.12"}}' |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,14 @@ | ||
| bundle: | ||
| name: cluster_policy_consumer | ||
|
|
||
| resources: | ||
| jobs: | ||
| j: | ||
| name: consumer_job | ||
| tasks: | ||
| - task_key: main | ||
| new_cluster: | ||
| # Replaced at test time with the policy id created by bundle_a. | ||
| policy_id: PLACEHOLDER_POLICY_ID | ||
| spark_version: 13.3.x-scala2.12 | ||
| num_workers: 1 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,73 @@ | ||
|
|
||
| === Bundle A creates the cluster policy | ||
| >>> withdir bundle_a [CLI] bundle deploy | ||
| Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/cluster_policy_producer/default/files... | ||
| Deploying resources... | ||
| Updating deployment state... | ||
| Deployment complete! | ||
|
|
||
| === Inject A's policy id into bundle B, then deploy B | ||
| >>> update_file.py bundle_b/databricks.yml PLACEHOLDER_POLICY_ID [POL_ID] | ||
|
|
||
| >>> withdir bundle_b [CLI] bundle deploy | ||
| Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/cluster_policy_consumer/default/files... | ||
| Deploying resources... | ||
| Updating deployment state... | ||
| Deployment complete! | ||
|
|
||
| === Policy created by A, then B's job carries that same policy id | ||
| >>> print_requests.py //policies/clusters //jobs | ||
| { | ||
| "method": "POST", | ||
| "path": "/api/2.0/policies/clusters/create", | ||
| "body": { | ||
| "definition": "{\"spark_version\":{\"type\":\"fixed\",\"value\":\"13.3.x-scala2.12\"}}", | ||
| "name": "shared_policy" | ||
| } | ||
| } | ||
| { | ||
| "method": "POST", | ||
| "path": "/api/2.2/jobs/create", | ||
| "body": { | ||
| "deployment": { | ||
| "kind": "BUNDLE", | ||
| "metadata_file_path": "/Workspace/Users/[USERNAME]/.bundle/cluster_policy_consumer/default/state/metadata.json" | ||
| }, | ||
| "edit_mode": "UI_LOCKED", | ||
| "format": "MULTI_TASK", | ||
| "max_concurrent_runs": 1, | ||
| "name": "consumer_job", | ||
| "queue": { | ||
| "enabled": true | ||
| }, | ||
| "tasks": [ | ||
| { | ||
| "new_cluster": { | ||
| "num_workers": 1, | ||
| "policy_id": "[POL_ID]", | ||
| "spark_version": "13.3.x-scala2.12" | ||
| }, | ||
| "task_key": "main" | ||
| } | ||
| ] | ||
| } | ||
| } | ||
|
|
||
| === Cleanup | ||
| >>> withdir bundle_b [CLI] bundle destroy --auto-approve | ||
| The following resources will be deleted: | ||
| delete resources.jobs.j | ||
|
|
||
| All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/cluster_policy_consumer/default | ||
|
|
||
| Deleting files... | ||
| Destroy complete! | ||
|
|
||
| >>> withdir bundle_a [CLI] bundle destroy --auto-approve | ||
| The following resources will be deleted: | ||
| delete resources.cluster_policies.pol | ||
|
|
||
| All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/cluster_policy_producer/default | ||
|
|
||
| Deleting files... | ||
| Destroy complete! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,17 @@ | ||
| title "Bundle A creates the cluster policy" | ||
|
Contributor
There was a problem hiding this comment. not sure about the value of the cross_bundle_ tests - this is not really resource specific and standard DABs behaviuor |
||
| trace withdir bundle_a $CLI bundle deploy | ||
|
|
||
| # Capture A's server-generated policy id and register [POL_ID]. | ||
| pol_id=$(withdir bundle_a read_id.py pol) | ||
|
|
||
| title "Inject A's policy id into bundle B, then deploy B" | ||
| trace update_file.py bundle_b/databricks.yml PLACEHOLDER_POLICY_ID "$pol_id" | ||
| trace withdir bundle_b $CLI bundle deploy | ||
|
|
||
| title "Policy created by A, then B's job carries that same policy id" | ||
| trace print_requests.py //policies/clusters //jobs | ||
|
|
||
| title "Cleanup" | ||
| trace withdir bundle_b $CLI bundle destroy --auto-approve | ||
| trace withdir bundle_a $CLI bundle destroy --auto-approve | ||
| rm -f out.requests.txt | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| Ignore = [".databricks"] |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can you add a new test case (not in basic/) for an out-of-band edit to the policy?