Add installer scripts for Linux, macOS and Windows - #187
Conversation
RandBLAS has had no installer. RandLAPACK's declines to cover it ("RandBLAS is
intentionally not covered: it stays a git submodule"), so anyone wanting
RandBLAS alone had to hand-build BLAS++ and Random123 and copy a cmake line
out of INSTALL.md. Release-plan item B1.
The script builds RandBLAS and its dependencies into a RandNLA-project
directory laid out exactly as RandLAPACK's installer lays one out, and honours
RANDNLA_PROJECT_DIR, so running both installers on one machine shares a single
dependency tree instead of building BLAS++ twice.
Design points worth calling out:
* Pins, not branches. BLAS++, Random123, GoogleTest and LAPACK++ are fetched
at an immutable ref by shallow fetch of one commit. Each install and each
source tree carries a provenance stamp, and is reused only when the stamp
matches what we would build now. Reuse keyed on mere presence makes changing
a pin a silent no-op for anyone who already ran the script.
* The integer width is read back, not assumed. BLAS++ probes int32 before
int64 and blas_int only filters library *names*, so a successful
blas_int=int64 configure proves ILP64 only for MKL, where mkl_intel_ilp64 is
a distinct library. For OpenBLAS there is just -lopenblas, and an LP64 build
passes the int32 probe and is accepted. The resolved width therefore comes
from BLAS++'s generated blas/defines.h after the build. Trusting the request
would stamp an LP64 install as ILP64 and every later run would reuse it
believing otherwise.
* ILP64 is preferred wherever it is real, and the fallback is loud. Accelerate
is refused outright for ILP64: Apple has shipped one since macOS 13.3 but
BLAS++ implements only the legacy LP64 interface (icl-utk-edu/lapackpp#43).
* Verification runs, it does not just link. A conftest sketches a matrix,
multiplies through BLAS++ and checks the result is symmetric positive
semidefinite. It goes through BLAS++ because that is how RandBLAS reaches
the BLAS.
* GoogleTest is provisioned rather than assumed, because BUILD_TESTS defaults
to ON while find_package(GTest) is not REQUIRED.
* Examples are a post-install offer, not a default. They need two dependencies
RandBLAS does not (LAPACK++, fast_matrix_market) and require OpenMP, which
stock Apple Clang cannot supply.
* The clone is never moved. RandLAPACK's installer relocates its own
repository into lib/, which breaks git worktrees; a symlink gives the same
layout without touching anything.
.gitignore matched "**install/", which was meant for build-output trees but
also swallowed a source directory named install/. Narrowed to "**/*-install/",
the pattern RandLAPACK already uses. The "!**install/.gitkeep" exception it
carried was vestigial: no .gitkeep is tracked.
The new install-script workflow runs the installer per OS and asserts what the
core workflows cannot: idempotent re-runs, dependency discovery, the LP64
fallback warning firing for stock OpenBLAS, ILP64 for MKL, Accelerate refusing
ILP64, and piped output containing no terminal escape sequences. A separate
packager lane deliberately does not run the installer at all -- it configures
with plain CMake against hand-installed dependencies inside a network
namespace with no interfaces, which is the contract conda-forge and Spack
actually depend on.
Verified on Linux: fresh MKL build, idempotent re-run, dependency discovery,
the examples path (all seven binaries, fast_matrix_market at the pinned
v1.7.6), a clean actionable failure when no OpenBLAS exists, and redirected
output free of escape sequences.
macOS: the installer set only the OpenMP_CXX_* variables for Homebrew libomp. BLAS++'s installed config calls find_dependency(OpenMP) without restricting components, so configuring RandBLAS against it also resolves OpenMP_C and failed with "Could NOT find OpenMP_C" -- after BLAS++ itself had built cleanly, which made it look like a RandBLAS problem. Set the C variables and export CFLAGS alongside CXXFLAGS. CI, MKL lane: oneAPI's apt packages do not put MKL on the linker's search path, so BLAS++ found the headers, failed to link, and reported "BLAS library not found". Source setvars.sh and propagate the variables, which is what the installer's own error message tells a user to do. Also dropped the "try the package, else add the repository" fallback: the first attempt always failed with "Unable to locate package", so it was noise pretending to be resilience. CI, everywhere: steps piping the installer into tee reported tee's exit status rather than the installer's, so the MKL lane's genuine failure was recorded as a passing step and only surfaced two steps later as a confusing assertion failure. Added set -euo pipefail to every step that pipes. Also fetch tags on checkout. rb_version.cmake runs `git describe --tags`, and without them the version degrades to 0.0.0-0-gunknown -- which the macOS log showed being baked into the install and reported by the configuration summary.
linux-openblas and macos-accelerate now pass, which confirms the OpenMP_C fix and the LP64-fallback assertions. The two remaining failures were both in the workflow rather than the installer. MKL lane: "set -u" and oneAPI's setvars.sh are incompatible. Its vars.sh reads OCL_ICD_FILENAMES and other variables without a default, so nounset made sourcing it fail with "unbound variable" before it exported anything. Enable nounset after the source instead. Packager lane: GitHub's runners restrict unprivileged user namespaces, so "unshare -rn" fails with "write failed /proc/self/uid_map: Operation not permitted" -- it works locally, which is how it got written that way. Use "sudo unshare -n" and hand the resulting build tree back to the runner user, since otherwise the root-owned tree breaks the non-root build step after it.
55cf3f8 to
8598d1e
Compare
Completes the installer pair. install.ps1 produces the same RandNLA-project layout as install.sh, honours RANDNLA_PROJECT_DIR with the same precedence, and delegates dependency provisioning to the setup script CI already uses so there is one implementation rather than two that drift. The x64 toolchain guard is the reason this exists in the form it does. "Developer PowerShell for VS" and "Developer Command Prompt for VS" both default to an *x86* toolchain, and an x86 linker cannot use the x64 import libraries every BLAS backend ships. Left unchecked the failure surfaces three layers down as BLAS++ reporting "BLAS library not found", blaming the libraries when the compiler is at fault -- which is exactly how this was diagnosed in RandLAPACK. toolchain-arch.ps1 reads VSCMD_ARG_TGT_ARCH, then the bin\Host<host>\<target>\ convention, then cl.exe's banner, and refuses x86 and arm64 with different messages because they need different answers. Provisioner changes: * Off personal forks. BLAS++ and LAPACK++ came from RaphaelArkadyMeyerNYU/*; both MSVC fixes merged upstream on 2026-08-06 (icl-utk-edu/blaspp#132, icl-utk-edu/lapackpp#87), so both now come from icl-utk-edu pinned to the merge commits. * Pinned and provenance-stamped. Clone-Head took a branch name and returned early whenever the destination merely existed, so a branch tip could move between runs and changing a ref was a silent no-op for anyone who already had the directory. Random123 in particular was fetched at the default branch, unpinned. Clone-Pinned fetches one ref and records it. Also exports randblas_stage_runtime_dlls() from the installed package. Windows searches an executable's own directory first and PATH last, so a downstream project linking installed RandBLAS could not find the BLAS DLLs at run time -- the function existed only in the build tree. Found because the installer's own verification step is such a consumer and could not configure without it. Verified on Windows 11 with Windows PowerShell 5.1 and VS 2022 Build Tools: missing-prerequisite path, x86 toolchain refused at preflight under a real vcvars32 environment, and a full x64 install from scratch -- oneMKL through vcpkg, BLAS++, GoogleTest, RandBLAS -- ending with the verification program compiling, linking, staging its DLLs and running, reporting ILP64.
…ackaging Three additions and one correction. A quick-start section at the top, because the installers are now the shortest path to a working RandBLAS and the guide opened by describing the manual one. It states the toolchain contract plainly -- you bring a C++20 compiler, CMake 3.21 and Git, the script supplies everything above that -- and covers sharing a dependency tree with RandLAPACK through RANDNLA_PROJECT_DIR, and why examples are opt-in. Appendix B, a tested-configuration table. Every row is a CI lane, so it says what is actually exercised rather than what we believe should work. It also explains the integer-width policy and, in particular, why OpenBLAS is the awkward case: BLAS++ probes int32 before int64 and blas_int only filters library names, so for MKL the choice is real and verifiable while for OpenBLAS an LP64 build passes the int32 probe and is accepted. Anyone with an ILP64 OpenBLAS has to point at it explicitly, and now the documentation says so. Appendix B also carries the packaging contract for conda-forge and Spack: which dependencies are actually required, that nothing is downloaded during configure, that LP64 is the default and therefore agrees with conda-forge's libblas metapackage, that RandBLAS never selects a BLAS itself, and that the installed package relocates correctly. Correction: the guide said RandBLAS selects /openmp:experimental under MSVC. It selects /openmp:llvm, and has since #184.
Three rendering tiers, chosen by probing the terminal: 2 a terminal with colour and UTF-8: a bar redrawn in place, block glyphs 1 a terminal without one of those: the same bar in ASCII 0 not a terminal: one line per step, no escapes, no carriage returns Tier 0 is a requirement rather than a fallback. Redirected output becomes install.log, CI transcripts and bug reports, and control characters make all three unreadable, so run_build_step falls straight through to run_step there and the bytes are identical to before this change. CI already asserts that redirected output contains no escape sequence and no carriage return. The bar is determinate. Both build tools already report progress on the stream being captured anyway -- Ninja writes "[12/34]" and Make writes "[ 42%]" -- so it tracks real work rather than elapsed time. A spinner would have conveyed nothing. Steps with no parseable progress keep their existing one-line form rather than growing a fake bar. Implementation note: the parsing loop necessarily runs in a subshell, so the build command's exit status is passed back through a file rather than a variable, and checked explicitly. Verified: redirected output byte-identical and escape-free; under a pty the bar advances through 5%, 20%, 49% during the RandBLAS build and each step resolves to a "done" line.
|
Combined — #190, #191 and #192 are now folded in here and closed, and the description above is rewritten to cover the whole thing as one change. Two notes on what I did with the edges of your request: #192 came along too, though you named only three. It was pushed last so you likely hadn't seen it; it adds the progress rendering to the #186 stayed separate and has since merged. It was CMake hygiene with no dependency on the installer — the version floor, the Nothing was lost in the consolidation: the branch had been rebased onto One thing in the description worth your attention specifically, since it is the load-bearing decision: BLAS++ probes |
Problem
RandBLAS has no installer. RandLAPACK's explicitly declines to cover it (
install/install.sh: "RandBLAS is intentionally not covered: it stays a git submodule"), so anyone who wants RandBLAS on its own has to hand-build BLAS++ and Random123 and copy acmakeinvocation out ofINSTALL.md. Release-plan item B1.Two audiences matter and only one of them runs a script. conda-forge and Spack never execute installers — they run
cmake, often with no network. So the installer is a convenience for people, and the packaging-critical requirement is that plain CMake keeps working when nobody runs it. Both are covered.What this PR adds
install/install.sh,install/install.ps1, aninstall-scriptworkflow that exercises both, the documentation for them, and the console progress rendering.Both scripts build into a
RandNLA-projecttree laid out exactly as RandLAPACK's installer lays one out, and both honourRANDNLA_PROJECT_DIR. Running both projects' installers on one machine therefore shares one dependency tree rather than building BLAS++ twice.The integer width is read back, not assumed
The part most worth reviewing. BLAS++ probes
int32first andint64second, andblas_intonly filters which library names to consider (cmake/BLASFinder.cmake). That makes ILP64 genuinely detectable for MKL, wheremkl_intel_ilp64is a different library frommkl_intel_lp64— but not for OpenBLAS, where BLAS++ only ever tries-lopenblas. An LP64 OpenBLAS passes theint32probe and is accepted, so a successfulblas_int=int64configure proves nothing there.The resolved width therefore comes from BLAS++'s generated
blas/defines.hafter the build. My first version recorded which configure attempt succeeded, which would have stamped an LP64 install as ILP64 and let every later run reuse it believing otherwise.Resulting policy: ILP64 wherever it is real, LP64 with a named warning elsewhere. Accelerate is refused outright for ILP64 — Apple has shipped one since macOS 13.3, but BLAS++ implements only the legacy interface (icl-utk-edu/lapackpp#43, tracked our side as #189). An ILP64 OpenBLAS is reachable through
--blas=custom.Everything pinned, reuse gated on provenance
BLAS++ (
3057185), Random123 (v1.14.0), GoogleTest (v1.18.0) and LAPACK++ (40b9d0d) at immutable refs — the same refs RandLAPACK validated. Each install and each source tree carries a provenance stamp and is reused only when it matches what we would build now, including backend and integer width. Without that, changing a pin is a silent no-op for anyone who already ran the script. The source stamp exists because a shallow checkout of a tag does not keep the tag ref locally.On the Windows side this replaces
Clone-Head, which took a branch name and returned early whenever the destination merely existed — so a branch tip could move between runs, and Random123 was fetched at its default branch entirely unpinned. It also moves BLAS++/LAPACK++ off theRaphaelArkadyMeyerNYU/*forks; both MSVC fixes merged upstream on 2026-08-06.Verification runs, it does not just link
The final step compiles, links and runs a program against the finished install: it sketches a matrix, multiplies through BLAS++, and checks the result is symmetric positive semidefinite. Through BLAS++ rather than raw
dgemm_, because that is how RandBLAS actually reaches the BLAS.The x64 toolchain guard (Windows)
"Developer PowerShell for VS 2022" and "Developer Command Prompt for VS 2022" both default to an x86 toolchain, and an x86 linker cannot use the x64 import libraries every BLAS backend ships. Unchecked, the failure surfaces three layers down as BLAS++ reporting
BLAS library not found. Shell bitness is not a usable signal either — the Developer Command Prompt is a 64-bit process that still selects x86 tools.toolchain-arch.ps1readsVSCMD_ARG_TGT_ARCH, then thebin\Host<host>\<target>\convention, thencl.exe's banner, and refuses x86 and arm64 with different messages because they need different answers.Smaller decisions
BUILD_TESTSdefaults to ON whilefind_package(GTest)is notREQUIRED— so a machine without it silently produces a build with zero tests.fast_matrix_market) andexamples/CMakeLists.txtrequires OpenMP, which stock Apple Clang cannot supply. Declining prints the exact follow-up command; it re-invokes the same script, so there is one code path.randblas_stage_runtime_dlls()is now exported from the installed package. Windows searches an executable's own directory first andPATHlast, so a downstream project linking installed RandBLAS had no way to find the BLAS DLLs — the function lived only in the build tree. Found because the installer's own verification step is such a consumer.lib/, which breaks git worktrees. A symlink gives the same layout..gitignorematched**install/, intended for build-output trees but also swallowing a source directory namedinstall/. Narrowed to**/*-install/, the pattern RandLAPACK already uses; the!**install/.gitkeepexception it carried was vestigial.install.logand CI transcripts. The bar is determinate, parsed from Ninja's[12/34]and Make's[ 42%]on the stream already being captured.CI
The
install-scriptworkflow asserts what the core workflows structurally cannot: idempotent re-runs, dependency discovery, the LP64 fallback warning firing for stock OpenBLAS, ILP64 for MKL, Accelerate refusing ILP64, and piped output containing no terminal escape sequences — which is what stops the progress bar from quietly filling every CI log.A separate packager lane deliberately does not run the installer. It configures with plain CMake against hand-installed dependencies, inside a network namespace with no interfaces, then builds, installs and consumes the result via
find_package. If that lane stays green a conda-forge recipe and a Spackpackage.pyare possible.Verification
Linux (gcc 15.2, oneAPI MKL):
blas/defines.h*_INSTALL_DIRpre-seeded)--examplesfast_matrix_marketatv1.7.6--blas=openblaswith no OpenBLAS present--blas=accelerate --blas-int=ilp64Windows 11, Windows PowerShell 5.1 (not pwsh 7), VS 2022 Build Tools:
cl.exe/cmakevcvars64one-linervcvars32.bat,VSCMD_ARG_TGT_ARCH=x86)All
install-scriptlanes green, including the offline packager lane.Notes for reviewers
install.shthis PR introduces — though only three were named. The tree is byte-identical to the state that was tested and green as four separate PRs.$LASTEXITCODEis unset until some native command runs andSet-StrictModemakes reading an unset variable an error, so the installer failed on exactly the runs where every dependency was cached. And the missingrandblas_stage_runtime_dllsexport only appears when something consumes the installed package.core-windowslanes cover the provisioner changes.