File system registry is not thread safe

[emfdavid]

Is there a way good option to make the registry thread safe?

exc_info: "Traceback (most recent call last):
...
    files = fsspec.open_files(query_path)
  File "/app/megaservice/platform.mega.service.py3_image.binary.runfiles/common_deps_fsspec/site-packages/fsspec/core.py", line 282, in open_files
    fs, fs_token, paths = get_fs_token_paths(
  File "/app/megaservice/platform.mega.service.py3_image.binary.runfiles/common_deps_fsspec/site-packages/fsspec/core.py", line 594, in get_fs_token_paths
    chain = _un_chain(urlpath0, storage_options or {})
  File "/app/megaservice/platform.mega.service.py3_image.binary.runfiles/common_deps_fsspec/site-packages/fsspec/core.py", line 325, in _un_chain
    cls = get_filesystem_class(protocol)
  File "/app/megaservice/platform.mega.service.py3_image.binary.runfiles/common_deps_fsspec/site-packages/fsspec/registry.py", line 216, in get_filesystem_class
    register_implementation(protocol, _import_class(bit["class"]))
  File "/app/megaservice/platform.mega.service.py3_image.binary.runfiles/common_deps_fsspec/site-packages/fsspec/registry.py", line 48, in register_implementation
    raise ValueError(
ValueError: Name (gs) already in the registry and clobber is False"

[martindurant]

#1237 cc @yarikoptic

[martindurant]

#1249 was supposed to fix this

[emfdavid]

Am I just behind on updates?
I am runing fsspec==2023.4.0

[martindurant]

I would appreciate if you have the time to try with an upgrade

[emfdavid]

Sorry I missed the update - I did search the issue for "registry"
Will close now.

[yarikoptic]

I believe that even with those changes fsspec is not entirely multi-thread safe. I believe we are running a heavily "overlocked" #1127 which makes it safe but damn slow :-/ would be nice if someone really resolved this properly

[yarikoptic]

having said that -- that PR/version is based on older release

❯ git describe --tags gh-jwodder/rlock-cache
2022.11.0-7-gc5d685a

and I have not tried anything more recent, but also do not spot among recent log messages upon quick look anything which might be relevant to 'lock' or 'thread'. So would be eager to see/hear if may be the problem was already addressed indeed.

[emfdavid]

I will report back - thank you both!