internal: Refactor immutable folder upload as an internal resource in direct engine by andrewnester · Pull Request #6084 · databricks/cli

andrewnester · 2026-07-28T11:29:51Z

Changes

Refactor immutable folder upload as an internal resource in direct engine

Why

This enables us:

Keep track of upload state and upload paths, so next deployment can check the status of the uploaded folder. Needed for breakglass
Easier path resolution: now all resources can just reference to internal resource like ${internal.snapshot.full_path} and it's automatically resolved during deploy
Plan is correct now and can easily skip uploads if nothing changed and no need to upload

Tests

Existing tests pass

… direct engine

github-actions · 2026-07-28T11:31:44Z

Approval status: pending

`/acceptance/bundle/` - needs approval

15 files changed
Suggested: @denik
Also eligible: @pietern, @janniklasrose, @shreyas-goenka, @lennartkats-db, @anton-107

`/bundle/` - needs approval

32 files changed
Suggested: @denik
Also eligible: @pietern, @janniklasrose, @shreyas-goenka, @lennartkats-db, @anton-107

`/cmd/bundle/` - needs approval

Files: cmd/bundle/plan.go
Suggested: @denik
Also eligible: @pietern, @janniklasrose, @shreyas-goenka, @lennartkats-db, @anton-107

`/libs/sync/` - needs approval

Files: libs/sync/path.go
Suggested: @simonfaltum
Also eligible: @Divyansh-db, @renaudhartert-db, @tanmay-db, @hectorcast-db, @parthban-db, @tejaskochar-db, @mihaimitrea-db, @chrisst, @rauchy

General files (require maintainer)

4 files changed
Based on git history:

@denik -- recent work in bundle/direct/dresources/, bundle/phases/, bundle/direct/

_{Any maintainer (@anton-107, @denik, @pietern, @shreyas-goenka, @simonfaltum, @renaudhartert-db, @janniklasrose, @lennartkats-db, @rugpanov, @rclarey) can approve all areas.

See OWNERS for ownership rules.}

eng-dev-ecosystem-bot · 2026-07-28T11:51:46Z

Integration test report

Commit: 03d79bd

Run: 31687789675

	Env	🔄flaky	💚RECOVERED	🙈SKIP	✅pass	🙈skip	Time
💚	aws linux		4	4	286	1137	4:55
💚	aws windows		4	4	288	1135	5:25
🔄	azure linux	2	4	4	283	1137	5:01
💚	azure windows		4	4	287	1135	5:27
💚	gcp linux		1	5	286	1137	4:44
💚	gcp windows		1	5	288	1135	5:36

10 interesting tests: 4 RECOVERED, 4 SKIP, 2 flaky

	Test Name	aws linux	aws windows	azure linux	azure windows	gcp linux	gcp windows
💚	TestAccept	💚R	💚R	💚R	💚R	💚R	💚R
🙈	TestAccept/bundle/invariant/no_drift	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🙈	TestAccept/bundle/resources/vector_search_endpoints/drift/recreated_same_name	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🙈	TestAccept/bundle/resources/vector_search_indexes/recreate/embedding_dimension	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🙈	TestAccept/ssh/connection	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🔄	TestFilerWorkspaceNotebook	✅p	✅p	🔄f	✅p	✅p	✅p
🔄	TestFilerWorkspaceNotebook/scalaJupyterNb.ipynb	✅p	✅p	🔄f	✅p	✅p	✅p
💚	TestFetchRepositoryInfoAPI_FromRepo	💚R	💚R	💚R	💚R	🙈S	🙈S
💚	TestFetchRepositoryInfoAPI_FromRepo/root	💚R	💚R	💚R	💚R
💚	TestFetchRepositoryInfoAPI_FromRepo/subdir	💚R	💚R	💚R	💚R

Top 6 slowest tests (at least 2 minutes):

duration	env	testname
5:26	gcp windows	TestAccept
5:16	azure windows	TestAccept
5:16	aws windows	TestAccept
3:01	aws linux	TestAccept
2:57	azure linux	TestAccept
2:54	gcp linux	TestAccept

denik

I think we should not have edge case for "internal" resources. We can just create resources.internal_files, populate it ourselves (forbid users from touching it) and then just let the framework handle it as a regular resource.

denik · 2026-07-29T08:56:20Z

        "deployments": [
          {
-            "command_path": "${workspace.snapshot_path}/files/src/main.py",
+            "command_path": "${internal.snapshot.full_path}/files/src/main.py",


Instead of custom path, can we keep using ${workspace.file_path}, just make it an alias internally to internal resource's snapshot path.

That way:

users know what this means

users can use the variable themselves and it'll work both in regular and immutable case.

${workspace.file_path} is ${internal.snapshot.full_path}/files and because we resolve the variables ${workspace.file_path} resolved to this pass. I'd prefer not to change the way variable reslution works and introduce some custom aliases

denik · 2026-08-03T13:35:31Z

@@ -35,12 +36,15 @@ type ACLEntry struct {
 // the same workspace location.
 // This interface exists so the implementation can later be replaced with a Go SDK call.


This does not explain why we need the interface - do we plan on supporting multiple implementation at the same time?

Strictly speaking we don't need it for SDK replacement later but we need this now in unit tests to replace the uploader with mock one

internal: Refactor immutable folder upload as an internal resource in…

84608d6

… direct engine

andrewnester requested a review from denik July 28, 2026 11:29

andrewnester and others added 4 commits July 28, 2026 14:00

fix test and tidy up

a5ab287

updated the output

4a0d4bd

Merge branch 'main' into feat/upload-resource

c869f1d

fix lint

aa73e34

denik reviewed Jul 29, 2026

View reviewed changes

andrewnester added 6 commits July 31, 2026 17:00

make snapshot a normal resource

b4f272f

clean up

f31a466

fix test

5943979

fixes

195974d

refschema

b6bb79f

lint fixes

2b7b48f

andrewnester requested a review from denik August 3, 2026 13:00

denik reviewed Aug 3, 2026

View reviewed changes

andrewnester added 2 commits August 3, 2026 16:59

use HasInternalResources

54a7dc0

fix test and etc

fbd5229

andrewnester requested a review from denik August 4, 2026 09:11

andrewnester added 5 commits August 4, 2026 19:20

stable bundle id

7825722

fix tests

b1e0be1

fix typo

b8cdcec

update refschema

f32d7fc

remove deadcode + update test output

94eefc7

denik reviewed Aug 5, 2026

View reviewed changes

adr feedback

769c0c3

andrewnester requested a review from denik August 5, 2026 11:16

andrewnester added 2 commits August 5, 2026 13:57

refschema

c5cbf31

fix invariant coverage test

af7f3ab

andrewnester and others added 3 commits August 6, 2026 11:12

fix fmt

5a05041

Merge branch 'main' into feat/upload-resource

6c74b0b

fmt

e816436

pietern self-requested a review August 11, 2026 11:29

andrewnester and others added 2 commits August 13, 2026 11:04

Merge branch 'main' into feat/upload-resource

0c8c6b6

update out.test.toml

03d79bd

		@@ -35,12 +36,15 @@ type ACLEntry struct {
		// the same workspace location.
		// This interface exists so the implementation can later be replaced with a Go SDK call.

Uh oh!

Conversation

andrewnester commented Jul 28, 2026

Changes

Why

Tests

Uh oh!

github-actions Bot commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Approval status: pending

/acceptance/bundle/ - needs approval

/bundle/ - needs approval

/cmd/bundle/ - needs approval

/libs/sync/ - needs approval

General files (require maintainer)

Uh oh!

eng-dev-ecosystem-bot commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Integration test report

Uh oh!

denik left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

denik Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

andrewnester Jul 31, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

denik Aug 3, 2026

Choose a reason for hiding this comment

Uh oh!

andrewnester Aug 4, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

github-actions Bot commented Jul 28, 2026 •

edited

Loading

`/acceptance/bundle/` - needs approval

`/bundle/` - needs approval

`/cmd/bundle/` - needs approval

`/libs/sync/` - needs approval

eng-dev-ecosystem-bot commented Jul 28, 2026 •

edited

Loading

denik left a comment •

edited

Loading