check-yaml pre-commit hooks failing after ruamel.yaml 0.19.0 update

[metaloid2k21]

Seems like check-yaml pre-commit hook now requires gcc + python3.11-devel packages (debian) to be installed, after ruamel.yaml dependency 0.19.0 update.
Getting the following error:

[INFO] Installing environment for https://github.com/pre-commit/pre-commit-hooks.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
An unexpected error has occurred: CalledProcessError: command: ('/root/.cache/pre-commit/repojv29_o8m/py_env-python3/bin/python', '-mpip', 'install', '.')
return code: 1
stdout:
    Processing /root/.cache/pre-commit/repojv29_o8m
      Installing build dependencies: started
      Installing build dependencies: finished with status 'done'
      Getting requirements to build wheel: started
      Getting requirements to build wheel: finished with status 'done'
      Preparing metadata (pyproject.toml): started
      Preparing metadata (pyproject.toml): finished with status 'done'
    Collecting ruamel.yaml>=0.15 (from pre_commit_hooks==4.6.0)
      Downloading ruamel_yaml-0.19.0-py3-none-any.whl.metadata (15 kB)
    Collecting ruamel.yaml.clibz>=0.3.3 (from ruamel.yaml>=0.15->pre_commit_hooks==4.6.0)
      Downloading ruamel_yaml_clibz-0.3.5.tar.gz (231 kB)
      Installing build dependencies: started
      Installing build dependencies: finished with status 'done'
      Getting requirements to build wheel: started
      Getting requirements to build wheel: finished with status 'done'
      Preparing metadata (pyproject.toml): started
      Preparing metadata (pyproject.toml): finished with status 'done'
    Downloading ruamel_yaml-0.19.0-py3-none-any.whl (117 kB)
    Building wheels for collected packages: pre_commit_hooks, ruamel.yaml.clibz
      Building wheel for pre_commit_hooks (pyproject.toml): started
      Building wheel for pre_commit_hooks (pyproject.toml): finished with status 'done'
      Created wheel for pre_commit_hooks: filename=pre_commit_hooks-4.6.0-py2.py3-none-any.whl size=41231 sha256=4106dca7dee80f65e6c4f72ef1455508369f96a867e2ab9befa71db3b6e427bb
      Stored in directory: /tmp/pip-ephem-wheel-cache-3aw3mi17/wheels/1f/58/02/2714cbda0820c23c585af0dd20ddee6950c50aa4afa2947115
      Building wheel for ruamel.yaml.clibz (pyproject.toml): started
      Building wheel for ruamel.yaml.clibz (pyproject.toml): finished with status 'error'
    Successfully built pre_commit_hooks
    Failed to build ruamel.yaml.clibz
stderr:
      error: subprocess-exited-with-error
      
      × Building wheel for ruamel.yaml.clibz (pyproject.toml) did not run successfully.
      │ exit code: 1
      ╰─> [9 lines of output]
          /tmp/pip-build-env-ga4pt8f_/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py:289: UserWarning: Unknown distribution option: 'build_zig'
            warnings.warn(msg)
          running bdist_wheel
          running build
          running build_ext
          building '_ruamel_yaml_clibz' extension
          creating build/temp.linux-x86_64-cpython-311
          gcc -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -ftree-vectorize -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -ftree-vectorize -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -ftree-vectorize -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/root/.cache/pre-commit/repojv29_o8m/py_env-python3/include -I/usr/include/python3.11 -c _ruamel_yaml_clibz.c -o build/temp.linux-x86_64-cpython-311/_ruamel_yaml_clibz.o
          error: command 'gcc' failed: No such file or directory
          [end of output]
      
      note: This error originates from a subprocess, and is likely not a problem with pip.
      ERROR: Failed building wheel for ruamel.yaml.clibz
    error: failed-wheel-build-for-install
    
    × Failed to build installable wheels for some pyproject.toml based projects
    ╰─> ruamel.yaml.clibz
Check the log at /root/.cache/pre-commit/pre-commit.log

Would you have any alternative workaround? Setting the additional_dependencies property seems not to be working.

Thanks in advance.

[markus-gerstel-sp]

-> #1229

[AvdN]

Have you made sure you have an up-to-date wheel and setuptools in your environment. There is no dependency on gcc, that only seems to happen as a fall back by an older (bundled) setuptool

[metaloid2k21]

Hi @AvdN seems like since ruamel.yaml dependency 0.19.0 update, gcc is mandatory... As seen in #1229

I wonder if there'll be some fix not requiring us to change all repos .pre-commit-config.yaml

[AAraKKe]

Hi all! Just went through the story around this since I have a service for which pre-commit hooks is not building anymore.

@AvdN seems that since the latest update in ruamel many projects that were building before are not building anymore, i.e. the update was a breaking change. Following your comments I see that you suggest wheel and setuptools to be the most up-to-date version. Are these versions required by the build system rerquiements in ruamel? I would naively assume that those versions should be a build requirement in ruamel to allow it to build. I am curious as to what was the expected workflow of anyone that depends on ruamel but has no requirement on those tools version.

Given the nature of ruamel which is likely massively used as a transitive dependency of tools that rely on YAML configs, I can image most people do not really keep track of ruamel as a dependency on their project since it is one of many transitive dependencies being installed. Maybe providing the proper wheels will be the easiest path forward for everyone? WDYT?

[AvdN]

Hi @AvdN seems like since ruamel.yaml dependency 0.19.0 update, gcc is mandatory... As seen in #1229

I wonder if there'll be some fix not requiring us to change all repos .pre-commit-config.yaml

No gcc is not mandatory I have tested this on systems and docker containers that have no gcc installed. It seems to be a fall-back from an outdated (bundled) setuptools. See e.g. #1229 (comment)

[sibizaestruch]

This will help, #1234
At least we stick with the ruamel.yaml version that was working.

[AvdN]

Hi all! Just went through the story around this since I have a service for which pre-commit hooks is not building anymore.

@AvdN seems that since the latest update in ruamel many projects that were building before are not building anymore, i.e. the update was a breaking change. Following your comments I see that you suggest wheel and setuptools to be the most up-to-date version. Are these versions required by the build system rerquiements in ruamel? I would naively assume that those versions should be a build requirement in ruamel to allow it to build. I am curious as to what was the expected workflow of anyone that depends on ruamel but has no requirement on those tools version.

Given the nature of ruamel which is likely massively used as a transitive dependency of tools that rely on YAML configs, I can image most people do not really keep track of ruamel as a dependency on their project since it is one of many transitive dependencies being installed. Maybe providing the proper wheels will be the easiest path forward for everyone? WDYT?

I actually tried adding setuptools as a build dependency to ruamel.yaml some time ago (when I first encountered non-compiling during testing and before adding the lines to the README). By the time the build requirements are gathered, the build system seem to have already loaded the broken setuptools and I would be surprised if there is a mechanism in place to re-import setuptools if a newr (or older) version is specified in (build) dependency.

Just generating the 80 wheels will take me about a day, and that is with only limited testing of the 80 wheel I now would have to generate. That is one of the reasons I wanted to move away from them. ruamel.yaml traditionally has had a dependency on ruamel.yaml.clib and now ruamel.yaml.clibz and I left that dependency in. Taking it out would affect users that use YAML(typ="safe") on larger files containing YAML documents.

However if ruamel.yaml is used for its round-tripping, which is currently implemneted in pure-Python only, installing
with --no-deps would solve the issue. If you use the SafeLoader and the C library is not available then there is also a fall-back to using pure Python, for smaller YAML documents it might not even be noticeably slower. Maybe making the dependency on ruamel.yaml.clibz optional ( so you have to specify ruamel.yaml[clibz] ) is the better way to go.

I know of only a handful of tools that actually use ruamel.yaml, but I have failed to check if these actually pin the versions they are dependent upon (as I have recommded to do for a long time).

[AvdN]

Also, at least on my Macbook M1 running macOS and under Linux on my x86_64 server, the wheels are actually build only once and cached, and any further install for ruamel.yaml.clibz (for the same Python version), takes the wheel from the local cache.
If that is not happening, I would really like to hear that.

[rdbisme]

My guess is that the upstream docker images are not shipping latest wheel and setuptools (already happened to me). Since we have this in our CI, it's a bit cumbersome to inject the pip install --upgrade.

I don't seem to find how you @AvdN specify the build-system.requires with a fast skim on your source code, but technically you should specify there setuptools and wheels with the right version discriminators? Because otherwise on these docker images, if setuptools is already installed on a previous version, I don't think it will be bumped.

I'm wild guessing.

[asottile]

please search the issue tracker next time