Portable standalone tarball build with Docker by cjnolet · Pull Request #2443 · NVIDIA/cuvs

cjnolet · 2026-08-13T15:42:49Z

This PR is based on @msarahan's original POC, with the logic of the ci sript moved into build.sh and invoked through the CI script. The readme is also moved into the build and install guide in the docs.

copy-pr-bot · 2026-08-13T15:42:54Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

cjnolet · 2026-08-13T17:28:48Z

/ok to test 0743a95

cjnolet · 2026-08-13T17:55:15Z

/ok to test 05fbbfe

cjnolet · 2026-08-13T18:10:59Z

/ok to test 472089c

cjnolet · 2026-08-13T18:21:50Z

/ok to test 266bd65

cjnolet · 2026-08-13T18:42:07Z

/ok to test 04c2e2b

cjnolet · 2026-08-13T19:27:37Z

/ok to test a0e7f83

cjnolet · 2026-08-14T16:59:58Z

/ok to test 4430d5e

cjnolet · 2026-08-14T19:18:22Z

/ok to test 4430d5e

jameslamb

Spent some time with this today. In general, I think it'd benefit from a pass for strictness... I feel that the current state has a lot of unnecessary flexibility and indirection. "a lot" = "so much that it makes it hard to understand the flow of configuration"

After that, it's worth trying out the commands in the docs again (I tried and a few failed).

I tested like this:

$ CUDA_VERSION=12.9.2 PYTHON_VERSION=3.11 ./build.sh tarball

That succeeded and produced a tarball. Its layout looks right to me, assuming that it's expected that the SOVERSION is 1.4 (libcuvs_c.so.1.4).

tarball contents (click me)

$ tar -tvf ./libcuvs_c.tar.gz
drwxr-xr-x root/root         0 2026-08-14 15:52 ./
drwxr-xr-x root/root         0 2026-08-14 15:52 ./lib64/
lrwxrwxrwx root/root         0 2026-08-14 15:52 ./lib64/libcuvs_c.so.1 -> libcuvs_c.so.1.4
lrwxrwxrwx root/root         0 2026-08-14 15:52 ./lib64/libcuvs_c.so -> libcuvs_c.so.1
drwxr-xr-x root/root         0 2026-08-14 15:52 ./lib64/cmake/
drwxr-xr-x root/root         0 2026-08-14 15:52 ./lib64/cmake/cuvs/
-rw-r--r-- root/root       835 2026-08-14 15:52 ./lib64/cmake/cuvs/cuvs-c_api-c-targets-release.cmake
-rw-r--r-- root/root      3681 2026-08-14 15:52 ./lib64/cmake/cuvs/cuvs-config-version.cmake
-rw-r--r-- root/root      7309 2026-08-14 15:52 ./lib64/cmake/cuvs/cuvs-config.cmake
-rw-r--r-- root/root       766 2026-08-14 15:52 ./lib64/cmake/cuvs/cuvs-dependencies.cmake
-rw-r--r-- root/root      4036 2026-08-14 15:52 ./lib64/cmake/cuvs/cuvs-targets.cmake
-rw-r--r-- root/root      4264 2026-08-14 15:52 ./lib64/cmake/cuvs/cuvs-c_api-c-targets.cmake
-rwxr-xr-x root/root 392490568 2026-08-14 15:52 ./lib64/libcuvs_c.so.1.4
drwxr-xr-x root/root         0 2026-08-14 15:52 ./include/
drwxr-xr-x root/root         0 2026-08-14 15:52 ./include/cuvs/
drwxr-xr-x root/root         0 2026-08-14 15:52 ./include/cuvs/preprocessing/
-rw-r--r-- root/root      8918 2026-07-10 10:06 ./include/cuvs/preprocessing/pca.h
drwxr-xr-x root/root         0 2026-08-14 15:52 ./include/cuvs/preprocessing/quantize/
-rw-r--r-- root/root      7525 2026-05-15 15:03 ./include/cuvs/preprocessing/quantize/pq.h
-rw-r--r-- root/root      3752 2026-05-15 15:03 ./include/cuvs/preprocessing/quantize/scalar.h
-rw-r--r-- root/root      4428 2026-05-15 15:03 ./include/cuvs/preprocessing/quantize/binary.h
drwxr-xr-x root/root         0 2026-08-14 15:52 ./include/cuvs/distance/
-rw-r--r-- root/root      1585 2026-05-15 15:03 ./include/cuvs/distance/pairwise_distance.h
-rw-r--r-- root/root      1593 2026-05-15 15:03 ./include/cuvs/distance/distance.h
drwxr-xr-x root/root         0 2026-08-14 15:52 ./include/cuvs/cluster/
-rw-r--r-- root/root      7687 2026-08-03 15:21 ./include/cuvs/cluster/kmeans.h
-rw-r--r-- root/root      2265 2026-08-03 15:21 ./include/cuvs/cluster/mg_kmeans.h
drwxr-xr-x root/root         0 2026-08-14 15:52 ./include/cuvs/neighbors/
-rw-r--r-- root/root      7403 2026-05-15 15:03 ./include/cuvs/neighbors/mg_ivf_pq.h
-rw-r--r-- root/root      1754 2026-05-15 15:03 ./include/cuvs/neighbors/refine.h
-rw-r--r-- root/root     19917 2026-05-15 15:03 ./include/cuvs/neighbors/hnsw.h
-rw-r--r-- root/root      1648 2026-08-03 15:21 ./include/cuvs/neighbors/common.h
-rw-r--r-- root/root      8057 2026-05-15 15:03 ./include/cuvs/neighbors/tiered_index.h
-rw-r--r-- root/root      7265 2026-05-15 15:03 ./include/cuvs/neighbors/nn_descent.h
-rw-r--r-- root/root     40535 2026-08-03 15:21 ./include/cuvs/neighbors/cagra.h
-rw-r--r-- root/root     11124 2026-05-29 16:18 ./include/cuvs/neighbors/ivf_sq.h
-rw-r--r-- root/root      5146 2026-08-03 15:21 ./include/cuvs/neighbors/all_neighbors.h
-rw-r--r-- root/root     12520 2026-05-15 15:03 ./include/cuvs/neighbors/ivf_flat.h
-rw-r--r-- root/root      1274 2026-05-15 15:03 ./include/cuvs/neighbors/mg_common.h
-rw-r--r-- root/root     23817 2026-05-15 15:03 ./include/cuvs/neighbors/ivf_pq.h
-rw-r--r-- root/root      7104 2026-05-15 15:03 ./include/cuvs/neighbors/brute_force.h
-rw-r--r-- root/root      6389 2026-05-15 15:03 ./include/cuvs/neighbors/vamana.h
-rw-r--r-- root/root      8066 2026-08-03 15:21 ./include/cuvs/neighbors/mg_cagra.h
-rw-r--r-- root/root      7611 2026-05-15 15:03 ./include/cuvs/neighbors/mg_ivf_flat.h
drwxr-xr-x root/root         0 2026-08-14 15:52 ./include/cuvs/selection/
-rw-r--r-- root/root      1267 2026-08-03 15:21 ./include/cuvs/selection/select_k.h
drwxr-xr-x root/root         0 2026-08-14 15:52 ./include/cuvs/core/
-rw-r--r-- root/root      3813 2026-08-03 15:21 ./include/cuvs/core/dataset.h
-rw-r--r-- root/root       635 2026-05-15 15:03 ./include/cuvs/core/export.h
-rw-r--r-- root/root      1424 2026-08-03 15:21 ./include/cuvs/core/all.h
-rw-r--r-- root/root       167 2026-08-14 15:52 ./include/cuvs/core/c_config.h
-rw-r--r-- root/root     11809 2026-08-03 15:21 ./include/cuvs/core/c_api.h
-rw-r--r-- root/root   3164590 2026-08-14 15:52 ./licenses.json
-rw-r--r-- root/root     61212 2026-08-14 15:52 ./LICENSE

I'd be happy to review again and to look more closely at the CI side of this once my suggestions have been addressed.

jameslamb · 2026-08-14T20:10:42Z

    },
    "cacheFrom": [
-      "ghcr.io/rapidsai/cuvs/devcontainer:26.08-cuda12.9-conda"
+      "ghcr.io/rapidsai/cuvs/devcontainer:26.10-cuda12.9-conda"


These references being there on main means ci/release/update-version.sh is missing some update rules. I'll put up a separate PR for that, and then these files can drop out out of the diff.

Thank you. yeah I happened to notice them as i was looking through the branch so figured I'd make the changes here too.

I'll put up a separate PR for that

@bdice beat me to that: #2479

Merge main into this and these files will drop out of the diff.

jameslamb · 2026-08-14T21:08:50Z

Oh one other thing... after running locally, these files are untracked:

$ git status
infrequent_licenses/
libcuvs_c.tar.gz

I think it'd be worthwhile to add .gitignore rules for them. There aren't any .tar.gz files in source control right now, so I'd support a generic rule like *.tar.gz.

Co-authored-by: James Lamb <jaylamb20@gmail.com>

cjnolet · 2026-08-14T21:39:48Z

/ok to test 1ba4f83

cjnolet · 2026-08-14T23:32:53Z

/ok to test 1ba4f83

cjnolet · 2026-08-14T23:34:41Z

/ok to testr f812e92

cjnolet · 2026-08-15T18:06:14Z

/ok to test f812e92

cjnolet · 2026-08-15T21:28:26Z

/ok to test d073064

copy-pr-bot · 2026-08-15T21:28:30Z

/ok to test d073064

@cjnolet, there was an error processing your request: E2

See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/2/

cjnolet · 2026-08-15T21:51:03Z

/ok to test de7e247

cjnolet · 2026-08-18T16:47:34Z

-pyenv rehash
+# rapids-configure-sccache enables anonymous S3 access by default. CI forwards
+# temporary AWS credentials, and sccache rejects both modes at once.
+if [[ -n "${AWS_ACCESS_KEY_ID:-}" ]]; then


@jameslamb I admit I don't know enough about this SCCACHE_S3_NO_CREDENTIALS parameter, but I did get a failure in CI and asked codex to fix it (this is what it did).

cjnolet · 2026-08-18T16:49:01Z

+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+REPO_ROOT="$(cd "${SCRIPT_DIR}/.." && pwd)"
+
+EXTRA_TARBALL_DOCKER_ARGS="--env AWS_ACCESS_KEY_ID \


This seems to be the minimal set of additional args that need to be passed into docker. Please let me know if you disagree @jameslamb. I used this EXTRA_TARBALL_DOCKER_ARGS as the decoupling point.

cjnolet requested review from a team as code owners August 13, 2026 15:42

cjnolet requested a review from KyleFromNVIDIA August 13, 2026 15:42

cjnolet self-assigned this Aug 13, 2026

cjnolet added improvement Improves an existing functionality non-breaking Introduces a non-breaking change labels Aug 13, 2026

cjnolet added this to Unstructured Data Processing Aug 13, 2026

cjnolet moved this to In Progress in Unstructured Data Processing Aug 13, 2026

msarahan and others added 5 commits August 13, 2026 11:53

PoC for Dockerfile for building standalone

0342601

add forgotten standalone files

a936a82

Move standalone tarball build to root

6e1d8b6

Move standalone build instructions to Fern docs

e86938b

Reconcile standalone build with upstream main

3b6132c

cjnolet force-pushed the codex/pr-1814-standalone-build branch from b61beb0 to 3b6132c Compare August 13, 2026 15:55

cjnolet requested a review from a team as a code owner August 13, 2026 15:55

cjnolet added 3 commits August 13, 2026 12:43

Move standalone Docker build into tarball target

f06426b

Rename standalone tarball test flag

eac4049

Merge branch 'main' into codex/pr-1814-standalone-build

0743a95

Normalize standalone Dockerfile copyright

05fbbfe

Fix checkout version annotations in workflows

472089c

Restore executable bit on standalone build script

266bd65

Update devcontainer image references to 26.10

04c2e2b

Fix authenticated sccache configuration

a0e7f83

Find standalone C library in lib64

4430d5e

jameslamb requested changes Aug 14, 2026

View reviewed changes

cjnolet and others added 11 commits August 14, 2026 17:31

Fix standalone Docker build examples

8a7b7fc

Apply suggestion from @jameslamb

b45f475

Co-authored-by: James Lamb <jaylamb20@gmail.com>

Apply suggestion from @jameslamb

fb6d382

Co-authored-by: James Lamb <jaylamb20@gmail.com>

Apply suggestion from @jameslamb

c8e26d4

Co-authored-by: James Lamb <jaylamb20@gmail.com>

Apply suggestion from @jameslamb

2443799

Co-authored-by: James Lamb <jaylamb20@gmail.com>

Apply suggestion from @jameslamb

bbbe993

Co-authored-by: James Lamb <jaylamb20@gmail.com>

Apply suggestion from @jameslamb

5419933

Co-authored-by: James Lamb <jaylamb20@gmail.com>

Apply suggestion from @jameslamb

3ca9566

Co-authored-by: James Lamb <jaylamb20@gmail.com>

Apply suggestion from @jameslamb

279e6cd

Co-authored-by: James Lamb <jaylamb20@gmail.com>

Apply suggestion from @jameslamb

143a10e

Co-authored-by: James Lamb <jaylamb20@gmail.com>

Apply suggestion from @jameslamb

1ba4f83

Co-authored-by: James Lamb <jaylamb20@gmail.com>

Fix upload-artifact version annotation

f812e92

cjnolet added 3 commits August 15, 2026 16:12

Refine standalone tarball build setup

6f1cf27

Refine standalone tarball configuration

d073064

Merge branch 'main' into codex/pr-1814-standalone-build

de7e247

Merge branch 'main' into codex/pr-1814-standalone-build

86b45f7

cjnolet commented Aug 18, 2026

View reviewed changes

Uh oh!

Conversation

cjnolet commented Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

copy-pr-bot Bot commented Aug 13, 2026

Uh oh!

cjnolet commented Aug 13, 2026

Uh oh!

cjnolet commented Aug 13, 2026

Uh oh!

cjnolet commented Aug 13, 2026

Uh oh!

cjnolet commented Aug 13, 2026

Uh oh!

cjnolet commented Aug 13, 2026

Uh oh!

cjnolet commented Aug 13, 2026

Uh oh!

cjnolet commented Aug 14, 2026

Uh oh!

cjnolet commented Aug 14, 2026

Uh oh!

jameslamb left a comment

Choose a reason for hiding this comment

Uh oh!

jameslamb Aug 14, 2026

Choose a reason for hiding this comment

Uh oh!

cjnolet Aug 14, 2026

Choose a reason for hiding this comment

Uh oh!

jameslamb Aug 18, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jameslamb commented Aug 14, 2026

Uh oh!

cjnolet commented Aug 14, 2026

Uh oh!

cjnolet commented Aug 14, 2026

Uh oh!

cjnolet commented Aug 14, 2026

Uh oh!

cjnolet commented Aug 15, 2026

Uh oh!

cjnolet commented Aug 15, 2026

Uh oh!

copy-pr-bot Bot commented Aug 15, 2026

Uh oh!

cjnolet commented Aug 15, 2026

Uh oh!

cjnolet Aug 18, 2026

Choose a reason for hiding this comment

Uh oh!

cjnolet Aug 18, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

cjnolet commented Aug 13, 2026 •

edited

Loading