Introduce Wally development containers #1502

doganulus · 2025-08-08T20:04:16Z

This PR introduces two development containers, tagged as builder and devel, each including all dependencies required to develop and test the Wally processor.

builder images intended for CI workflows
devel images intended for end-users

These containers will serve as a quick start for setting up a development environment and testing the Wally processor. The container build workflow produces images for both amd64 and arm64 platforms.

Currently marked as a draft, as the ARM build depends on an upcoming upstream release from the SAIL project. Container images without sail installed are available temporarily here for testing/demo purposes:

https://github.com/doganulus/cvw/pkgs/container/wally

Note that the current base image is Debian 12. We can discuss this choice along with what should be included in the containers.

jordancarlin

This is great! I was working on a Docker image a few months ago, but unfortunately never found time to finish it. I left a first round of detailed comments inline.

One more general question is whether we really need separate builder and devel images. While I see the value of that for some projects, I'm not sure what would really be different between them in the case of the Wally images. It is also simpler for users if there is just a single image for them to pull.

.github/workflows/container-release.yml

jordancarlin · 2025-08-09T00:31:36Z

.github/workflows/container-release.yml

+      - .github/workflows/container-release.yml # Self-trigger
+  workflow_dispatch:
+    inputs:
+      publish:


It would be nice to have an option to set a different version in the workflow dispatch inputs so that we can keep tagged versions of the images around (like the versions of the tools corresponding to the upcoming 1.0 release of the repo).

This or adding tagged versions to the build matrix. Both are fine.

jordancarlin · 2025-08-09T04:38:59Z

.github/workflows/container-release.yml

+        type: boolean
+
+env:
+  REGISTRY: ghcr.io/doganulus


Will need to be changed for the final version (just noting this so we don't forget before merging).

jordancarlin · 2025-08-09T04:39:34Z

.github/workflows/container-release.yml

+  PODMAN_ARCH: amd64
+  PLATFORM: linux-amd64


Should these be set as part of the matrix so they are set appropriately for arm builds?

These variables are overwritten in matrix builds. Set to an initial value to keep the linter happy.

jordancarlin · 2025-08-09T04:46:17Z

.github/workflows/container-release.yml

+        uses: actions/checkout@v4
+
+      - name: Log in to the GitHub Container registry
+        uses: redhat-actions/podman-login@v1


What is the rationale for using the redhat-actions instead of the more common docker-build-push action?

Such multi-arch builds must build in separate runners and merge their manifests.

If I recall correctly, my reason to switch redhat-actions was this issue: docker/build-push-action#1287

containers/wally-devel/Dockerfile

jordancarlin · 2025-08-09T04:55:24Z

containers/wally-devel/Dockerfile

+RUN --mount=type=bind,source=./bin,target=/wally/bin \
+    bash /wally/bin/wally-package-install.sh && \
+    apt-get autoremove -y && rm -rf /var/lib/apt/lists/*
+
+RUN --mount=type=bind,source=./bin,target=/wally/bin \
+    export WALLY=/wally && \
+    export RISCV=/opt/riscv && \
+    bash "$WALLY/bin/installation/riscv-gnu-toolchain-install.sh" --clean
+
+RUN --mount=type=bind,source=./bin,target=/wally/bin \
+    export WALLY=/wally && \
+    export RISCV=/opt/riscv && \
+    bash "$WALLY/bin/installation/elf2hex-install.sh" --clean && \
+    bash "$WALLY/bin/installation/qemu-install.sh" --clean && \
+    bash "$WALLY/bin/installation/spike-install.sh" --clean && \
+    # bash "$WALLY/bin/installation/sail-install.sh" --clean && \
+    bash "$WALLY/bin/installation/verilator-install.sh" --clean


Have these been tested yet? It seems like right now the site-setup.sh script is not installed into $RISCV, which is likely to cause issues. It also doesn't run the python-setup.sh script, which is needed to run RISCOF.

I wonder if we would be better off running the full $WALLY/bin/wally-tool-chain-install.sh script. That ensures all components are installed and would probably make maintenance easier if new tools are added. If you're worried about layers, we could pre-run the riscv-gnu-toolchain-install.sh script so it is in a separate layer. The main installation script will skip it if it sees it is already the correct version.

Not tested yet. Some of those setup scripts can be reevaluated for container use cases. But I haven't looked up at them yet.

doganulus · 2025-08-09T05:41:53Z

This builder/devel separation follows a pattern I’ve used in other projects. The builder image is populated with all tools and dependencies needed to build and test the project in CI environments. The devel image, in turn, includes the recommended tools for developers, ranging from editors like nano to pre-commit hooks and linters. I find this separation useful in practice, though it’s entirely possible to merge everything into a single devel image if preferred.

davidharrishmc · 2025-08-09T10:49:40Z

I'm very excited to see this capability. @doganulus thank you for stepping up with this. Docker is outside my main skill set and I have questions about maintainability. Everything in Wally has been subject to fast bit rot and I'd like to make sure Docker is maintainable if we direct users this way. @doganulus will you / your team be willing to own this capability for a good while (say initially 3 years). This would include preparing documentation sufficient for low-skill users, fixing problems, and providing basic support (preferably little because the documentation is good and there are no bugs, but that won't be the case immediately). It would also be great to get this into a regular regression that would notify everyone, especially you, when some dependencies change and the container stops working.

I don't have experience to judge between builder & devel vs. just devel, but my bias is toward minimalism if devel could be sufficient. Fewer things to maintain and for other people to have to understand.

I don't fully understand the security issues on Docker. Two years ago, we had some concerns about whether running the cvw environment in Docker would give too much privilege to users. Is this setup now something that a paranoid corporate IT department would comfortably allow on their system?

Regarding Debian, I'd like to understand tradeoffs. I'm ok with it if there are compelling technical reasons to go there. So far, everything for Wally including the textbook documentation has been on Rocky and Ubuntu. Rocky is the free Red Hat distro that is compatible with EDA vendors including Synopsys, Cadence, Siemens, and Breker. Ubuntu is popular with desktop users and has been sufficient to run almost all of these tools except recent Design Compiler. We've moved our work from Ubuntu to Rocky to be able to use all the EDA. If a user wants to synthesize the RTL in Design Compiler, does it matter whether the Docker container is running Debian or Red Hat?

doganulus · 2025-08-09T11:22:45Z

I don't fully understand the security issues on Docker. Two years ago, we had some concerns about whether running the cvw environment in Docker would give too much privilege to users. Is this setup now something that a paranoid corporate IT department would comfortably allow on their system?

This is caused by the fact that the Docker daemon requires root access by default, which may be rightfully unwanted in shared user systems. However, containers are now a general technology, and it is possible (and preferred) to run them rootless in these environments. Then, actually, a container image may be preferred by users and IT admins alike, as it is just pull and run.

Regarding Debian, I'd like to understand tradeoffs. I'm ok with it if there are compelling technical reasons to go there.

The differences would be minimal. Currently, there is a bug in the installation scripts (a hardcoded path preventing arm64 builds) regarding Ubuntu, so I switched to Debian for this demonstration. Yes, I agree that Red Hat based distros may be better due to their ecosystem.

@doganulus will you / your team be willing to own this capability for a good while (say initially 3 years).

We can do that, yes.

davidharrishmc · 2025-08-09T17:47:01Z

This sounds great!

…

On Sat, Aug 9, 2025 at 4:23 AM Doğan Ulus ***@***.***> wrote: *doganulus* left a comment (openhwgroup/cvw#1502) <#1502 (comment)> I don't fully understand the security issues on Docker. Two years ago, we had some concerns about whether running the cvw environment in Docker would give too much privilege to users. Is this setup now something that a paranoid corporate IT department would comfortably allow on their system? This is caused by the fact that the Docker daemon requires root access by default, which may be *rightfully* unwanted in shared user systems. However, containers are now a general technology, and it is possible (and preferred) to run them *rootless* in these environments. Then, actually, a container image may be preferred by users and IT admins alike, as it is just pull and run. Regarding Debian, I'd like to understand tradeoffs. I'm ok with it if there are compelling technical reasons to go there. The differences would be minimal. Currently, there is a bug in the installation scripts (a hardcoded path for amd64 builds) regarding Ubuntu, so I switched to Debian for this demonstration. Yes, I agree that Red Hat based distros may be better due to their ecosystem. @doganulus <https://github.com/doganulus> will you / your team be willing to own this capability for a good while (say initially 3 years). We can do that, yes. — Reply to this email directly, view it on GitHub <#1502 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AR4AA35VXX33UADJSSRQGLL3MXKZZAVCNFSM6AAAAACDOYQZESVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTCNZQGYZTKNJVGY> . You are receiving this because you commented.Message ID: ***@***.***>

jordancarlin · 2025-08-09T19:34:38Z

containers/wally-devel/Dockerfile

+ARG TARGETARCH TARGETOS TARGETPLATFORM TARGETVARIANT
+
+ARG RISCV_GNU_TOOLCHAIN_VERSION
+ENV RISCV_GNU_TOOLCHAIN_VERSION=${RISCV_GNU_TOOLCHAIN_VERSION}


This is already set in the installation script. Any reason to set it here as well?

jordancarlin · 2025-08-09T19:40:40Z

containers/wally-devel/Dockerfile

+ENV PATH="${RISCV_INSTALL_PREFIX}/bin${PATH:+:${PATH}}"
+ENV CPATH="${RISCV_INSTALL_PREFIX}/include${CPATH:+:${CPATH}}"
+ENV LD_LIBRARY_PATH="\
+        ${RISCV_INSTALL_PREFIX}/lib\
+        :${RISCV_INSTALL_PREFIX}/lib64\
+        ${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}"
+ENV PKG_CONFIG_PATH="\
+        ${RISCV_INSTALL_PREFIX}/lib/pkgconfig\
+        :${RISCV_INSTALL_PREFIX}/lib64/pkgconfig\
+        :${RISCV_INSTALL_PREFIX}/share/pkgconfig\
+        ${PKG_CONFIG_PATH:+:${PKG_CONFIG_PATH}}"


All of these paths (plus several other variables) are set in setup.sh and site-setup.sh. What about having the CMD or ENTRYPOINT step source the setup.sh script so everything is set when the user enters the container?

Or we could modify the .bashrc in the container. That would allow the file to be sourced only if it’s found.

I am experimenting with those variables currently. Entrypoint might be a good idea. Need to check how they are overwritten when run as a devcontainer for vscode.

jordancarlin · 2025-08-09T19:42:29Z

containers/wally-devel/Dockerfile

+
+RUN ${WALLY_PYTHON_EXECUTABLE} -m pip install --no-cache-dir \
+        reuse \
+        pre-commit 


pre-commit is already installed from the requirements.txt file.

doganulus · 2025-08-12T16:59:09Z

@jordancarlin Now riscof tests work in the container. Also removed Skywater libs to save space, but tell me if they are not optional. Surely, what goes inside the container is always an open discussion.

But what is the problem regarding verilator and regression-wally tests? Is it a flake, or is there a larger issue? I was planning to use verilator this semester with Wally.

jordancarlin · 2025-08-12T20:26:57Z

But what is the problem regarding verilator and regression-wally tests? Is it a flake, or is there a larger issue? I was planning to use verilator this semester with Wally.

It’s not a Verilator issue. There is an issue with the arch64pmp tests themselves that results in the failure. That should hopefully be resolved soon, and if not we’ll disable those tests until they’re fixed. You should be fine to use Verilator with Wally.

I’ll try to test out the container soon and follow up with any other comments.

doganulus force-pushed the devel-container branch from f7ab78b to a60ae16 Compare August 8, 2025 21:12

jordancarlin requested changes Aug 9, 2025

View reviewed changes

doganulus force-pushed the devel-container branch from a60ae16 to 3b49e30 Compare August 9, 2025 18:40

jordancarlin reviewed Aug 9, 2025

View reviewed changes

doganulus added 13 commits September 5, 2025 12:14

Initial devel container

1f50ef0

Fix container build context in the workflow

435eb67

Clean after build

1626e36

Update names

d68dc8a

Fix yaml

9140641

Change publish default value temporarily

896889f

Upload digests

5bbbb6a

Move to Almalinux

22eae08

Add container setup script

fad174d

Do not install Skywater library

7a16066

Maximize build space on runner

3f9bc70

Fix custom TMPDIR location

6287841

Create TMPDIR after maximize

20d74bd

doganulus force-pushed the devel-container branch from 0dc8ff7 to 20d74bd Compare September 5, 2025 09:16

Introduce Wally development containers #1502

Are you sure you want to change the base?

Introduce Wally development containers #1502

Uh oh!

Conversation

doganulus commented Aug 8, 2025

Uh oh!

jordancarlin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

doganulus commented Aug 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

davidharrishmc commented Aug 9, 2025

Uh oh!

doganulus commented Aug 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

davidharrishmc commented Aug 9, 2025 via email

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

doganulus Aug 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

doganulus commented Aug 12, 2025

Uh oh!

jordancarlin commented Aug 12, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

doganulus commented Aug 9, 2025 •

edited

Loading

doganulus commented Aug 9, 2025 •

edited

Loading

doganulus Aug 9, 2025 •

edited

Loading