CUDA 10.1 on Windows 10 shows error: pygpu.gpuarray.GpuArrayException: b'Could not load "nvrtc64_70.dll": The specified module could not be found.\r\n' theano

[anupverma20]

OS: Windows 10
CUDA: 10.1

 Name                     Version                           Build  Channel
python                     3.5.6                            he025d50_0
libgpuarray                0.7.6                            vc14_0  [vc14]  mila-udem
pygpu                      0.7.6            	            py35h452e1ab_0
theano                     1.0.2            		    py35he980bc4_0
keras                      2.2.2                            0

Content of .theanorc.txt file:
[global]
device = cuda
floatX = float32

In System variables, the following are set:
CUDA_PATH
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1

CUDA_PATH_V10_1
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1

DEVICE
cuda

In the Path Variable of System variables, the following are set:
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\libnvvp
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\bin

Error:

(gpuml) C:\Users\Anup\Desktop\zdata\wiki_crop_out>python -c "from keras import backend; print(backend._BACKEND")
Using Theano backend.
ERROR (theano.gpuarray): Could not initialize pygpu, support disabled
Traceback (most recent call last):
  File "C:\Users\Anup\Anaconda3\envs\gpuml\lib\site-packages\theano\gpuarray\__init__.py", line 227, in <module>
    use(config.device)
  File "C:\Users\Anup\Anaconda3\envs\gpuml\lib\site-packages\theano\gpuarray\__init__.py", line 214, in use
    init_dev(device, preallocate=preallocate)
  File "C:\Users\Anup\Anaconda3\envs\gpuml\lib\site-packages\theano\gpuarray\__init__.py", line 99, in init_dev
    **args)
  File "pygpu\gpuarray.pyx", line 658, in pygpu.gpuarray.init
  File "pygpu\gpuarray.pyx", line 587, in pygpu.gpuarray.pygpu_init
pygpu.gpuarray.GpuArrayException: b'Could not load "nvrtc64_70.dll": The specified module could not be found.\r\n'
theano

Similar issue was raised but closed without any solution. If CUDA 10.1 is not supported currently, when will it be supported?

Thanks!

[nouiz]

It was fixed upstream, but no new release was done. So you must compile libgpuarray yourself.

…

On Tue, May 14, 2019 at 3:00 PM Anup Verma ***@***.***> wrote: OS: Windows 10 CUDA: 10.1 Name Version Build Channel python 3.5.6 he025d50_0 libgpuarray 0.7.6 vc14_0 [vc14] mila-udem pygpu 0.7.6 py35h452e1ab_0 theano 1.0.2 py35he980bc4_0 keras 2.2.2 0 *Content of .theanorc.txt file:* [global] device = cuda floatX = float32 In *System variables*, the following are set: *CUDA_PATH* C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1 *CUDA_PATH_V10_1* C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1 *GPUARRAY_TEST_DEVICE* cuda In the *Path Variable of System variables*, the following are set: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\libnvvp C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\bin *Error:* (gpuml) C:\Users\Anup\Desktop\zdata\wiki_crop_out>python -c "from keras import backend; print(backend._BACKEND") Using Theano backend. ERROR (theano.gpuarray): Could not initialize pygpu, support disabled Traceback (most recent call last): File "C:\Users\Anup\Anaconda3\envs\gpuml\lib\site-packages\theano\gpuarray\__init__.py", line 227, in <module> use(config.device) File "C:\Users\Anup\Anaconda3\envs\gpuml\lib\site-packages\theano\gpuarray\__init__.py", line 214, in use init_dev(device, preallocate=preallocate) File "C:\Users\Anup\Anaconda3\envs\gpuml\lib\site-packages\theano\gpuarray\__init__.py", line 99, in init_dev **args) File "pygpu\gpuarray.pyx", line 658, in pygpu.gpuarray.init File "pygpu\gpuarray.pyx", line 587, in pygpu.gpuarray.pygpu_init pygpu.gpuarray.GpuArrayException: b'Could not load "nvrtc64_70.dll": The specified module could not be found.\r\n' theano Similar issue was raised but closed without any solution. If CUDA 10.1 is not supported currently, when will it be supported? Thanks! — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#587?email_source=notifications&email_token=AABMF65F4OFHAHWYO5QTOEDPVMD5VA5CNFSM4HM4YTW2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4GTYBGHA>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABMF64MUDG3BCDXM6LPBP3PVMD5VANCNFSM4HM4YTWQ> .

[anupverma20]

It was fixed upstream, but no new release was done. So you must compile libgpuarray yourself.
…
On Tue, May 14, 2019 at 3:00 PM Anup Verma @.***> wrote: OS: Windows 10 CUDA: 10.1 Name Version Build Channel python 3.5.6 he025d50_0 libgpuarray 0.7.6 vc14_0 [vc14] mila-udem pygpu 0.7.6 py35h452e1ab_0 theano 1.0.2 py35he980bc4_0 keras 2.2.2 0 Content of .theanorc.txt file: [global] device = cuda floatX = float32 In System variables, the following are set: CUDA_PATH C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1 CUDA_PATH_V10_1 C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1 GPUARRAY_TEST_DEVICE cuda In the Path Variable of System variables, the following are set: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\libnvvp C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\bin Error: (gpuml) C:\Users\Anup\Desktop\zdata\wiki_crop_out>python -c "from keras import backend; print(backend.BACKEND") Using Theano backend. ERROR (theano.gpuarray): Could not initialize pygpu, support disabled Traceback (most recent call last): File "C:\Users\Anup\Anaconda3\envs\gpuml\lib\site-packages\theano\gpuarray_init.py", line 227, in use(config.device) File "C:\Users\Anup\Anaconda3\envs\gpuml\lib\site-packages\theano\gpuarray_init_.py", line 214, in use init_dev(device, preallocate=preallocate) File "C:\Users\Anup\Anaconda3\envs\gpuml\lib\site-packages\theano\gpuarray_init_.py", line 99, in init_dev **args) File "pygpu\gpuarray.pyx", line 658, in pygpu.gpuarray.init File "pygpu\gpuarray.pyx", line 587, in pygpu.gpuarray.pygpu_init pygpu.gpuarray.GpuArrayException: b'Could not load "nvrtc64_70.dll": The specified module could not be found.\r\n' theano Similar issue was raised but closed without any solution. If CUDA 10.1 is not supported currently, when will it be supported? Thanks! — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#587?email_source=notifications&email_token=AABMF65F4OFHAHWYO5QTOEDPVMD5VA5CNFSM4HM4YTW2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4GTYBGHA>, or mute the thread https://github.com/notifications/unsubscribe-auth/AABMF64MUDG3BCDXM6LPBP3PVMD5VANCNFSM4HM4YTWQ .

Thanks for the reply.
I made libgpuarray.sln using cmake in Anaconda.:
(gpuml) C:\Users\Anup\Desktop\lib\libgpuarray\build>cmake .. -DCMAKE_BUILD_TYPE=Release -G "Visual Studio 16 2019" -A X64"

I opened libgpuarray.sln in Visual Studio.
When I select Build > gpuarray, I get the following error:

1>------ Build started: Project: gpuarray, Configuration: Release X64 ------
1>Generating ../../src/gpuarray_types.c, ../../src/gpuarray/types.h
1>'python' is not recognized as an internal or external command,
1>operable program or batch file.
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Microsoft\VC\v160\Microsoft.CppCommon.targets(209,5): error MSB6006: "cmd.exe" exited with code 9009.
1>Done building project "gpuarray.vcxproj" -- FAILED.
========== Build: 0 succeeded, 1 failed, 1 up-to-date, 0 skipped ==========

I could not fix the python-related error
I am new to cmake, Visual Studio and dll. Please help!

[NellTrico]

Hello! I experience something similar.
I have Windows 10, CUDA 10.1, and tried to install from the main branch and got the following error:

File "pygpu\gpuarray.pyx", line 658, in pygpu.gpuarray.init
File "pygpu\gpuarray.pyx", line 587, in pygpu.gpuarray.pygpu_init
pygpu.gpuarray.GpuArrayException: b'Could not load "nvrtc64_102.dll": The specified module could not be found.\r\n'

I checked which version of nvrtc I have installed and it turns out I have nvrtc64_101_0.dll.

Could I please get some help with that?

[anupverma20]

I installed python, added its path in Environment Variables > Path. Now, that error is fixed.
I also installed Mako, restarted the computer. But, I get the following error while building the sln file in Visual Studio:

1>------ Build started: Project: gpuarray, Configuration: Release X64 ------
1>Generating ../../src/gpuarray_types.c, ../../src/gpuarray/types.h
1>Traceback (most recent call last):
1>  File "gen_types.py", line 3, in <module>
1>    from mako import exceptions
1>ModuleNotFoundError: No module named 'mako'
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Microsoft\VC\v160\Microsoft.CppCommon.targets(209,5): error MSB6006: "cmd.exe" exited with code 1.
1>Done building project "gpuarray.vcxproj" -- FAILED.
========== Build: 0 succeeded, 1 failed, 1 up-to-date, 0 skipped ==========

I was able to build gpuarray.dll in my office laptop. But, in the personal laptop, I am getting the above error. Trying to fix it now.

[anupverma20]

Hello! I experience something similar.
I have Windows 10, CUDA 10.1, and tried to install from the main branch and got the following error:

File "pygpu\gpuarray.pyx", line 658, in pygpu.gpuarray.init
File "pygpu\gpuarray.pyx", line 587, in pygpu.gpuarray.pygpu_init
pygpu.gpuarray.GpuArrayException: b'Could not load "nvrtc64_102.dll": The specified module could not be found.\r\n'

I checked which version of nvrtc I have installed and it turns out I have nvrtc64_101_0.dll.

Could I please get some help with that?

As far as I have learnt from answers to similar issues on the internet, there are 2 options:

1. The easier option is to install CUDA 9.
2. The tougher option is to build libgpuarray yourself as pointed out by @nouiz above.
   Refer the answer of cknave at https://www.bountysource.com/issues/68975408-theano-1-0-3-issue-with-pygpu-error-theano-gpuarray-could-not-initialize-pygpu-support-disabled to get a hint of the process.

[NellTrico]

Hello! I experience something similar.
I have Windows 10, CUDA 10.1, and tried to install from the main branch and got the following error:
File "pygpu\gpuarray.pyx", line 658, in pygpu.gpuarray.init
File "pygpu\gpuarray.pyx", line 587, in pygpu.gpuarray.pygpu_init
pygpu.gpuarray.GpuArrayException: b'Could not load "nvrtc64_102.dll": The specified module could not be found.\r\n'
I checked which version of nvrtc I have installed and it turns out I have nvrtc64_101_0.dll.
Could I please get some help with that?

As far as I have learnt from answers to similar issues on the internet, there are 2 options:

1. The easier option is to install CUDA 9.

2. The tougher option is to build libgpuarray yourself as pointed out by @nouiz above.
   Refer the answer of cknave at https://www.bountysource.com/issues/68975408-theano-1-0-3-issue-with-pygpu-error-theano-gpuarray-could-not-initialize-pygpu-support-disabled to get a hint of the process.

Thanks for your reply @anupverma20 !
I tried both options without success. (Note that I don't use conda so I was already building libgpuarray myself)

I am confused as to why the library insists on pointing to nvrtc64_102.dll, even after I uninstalled CUDA 10 and kept 9. I looked at the code in gpuarray_buffer_cuda.c and I am at loss to understand why and how I can even obtain major=10, minor=2... Nevertheless I tried changing major and minor in the source code to 9 and 1 respectively (having installed CUDA 9.1 with the corresponding nvrtc64_91.dll), but I was then faced with

Could not load "nvrtc64_91.dll": %1 is not a valid Win32 application.\r\n'

Similarly, when I still had CUDA 10.1, I tried renaming the dll to the expected "nvrtc64_102.dll", and had a similar error message.

EDIT: It seems I get this error because I am using Python 36-32. I thought a compatible 32/64 bit version of Python would let me do it though? Am I wrong? If yes, do I have any option, like Running GPU acceleration with OpenCL for instance, which I saw in the tutorials, but have not given time yet, or is it limited for 64 bit architecture only?

[kumarnikhil936]

Any solutions so far?
Still facing this issue.

[ThoenigAdrian]

same issue here with cuda 11

[nouiz]

This MR is the last one that update the supported CUDA version: https://github.com/Theano/libgpuarray/pull/580/files
So the most recent CUDA version supported is 10.0

This lib isn't maintained. But maybe you can find now to support newer version with that url.

[ThoenigAdrian]

@nouiz
Apparently there is an up to date fork : https://github.com/pymc-devs/Theano-PyMC . Haven't tried it though.
Afaik it should be possible to install multiple CUDA versions on a machine. So one could install CUDA 9.0 and use Theano with it.

[nouiz]

This is a Theano fork. I do not know if they forked libgpuarray too. Maybe you can ask them?
I would recommend to try CUDA 10.0 first and not 9.0 as from the quick look at the code ,10.0 was supported.