ArrowFSWrapper with S3 does not like leading slash

[TLCFEM]

Issue

This line defines the root marker as /.

filesystem_spec/fsspec/implementations/arrow.py

Line 49 in 2b5ed0f

root_marker = "/"

Then in the parent class, the _parent method would always return a string with leading slash.

filesystem_spec/fsspec/spec.py

Lines 1266 to 1272 in 2b5ed0f

	def _parent(cls, path):
	path = cls._strip_protocol(path)
	if "/" in path:
	parent = path.rsplit("/", 1)[0].lstrip(cls.root_marker)
	return cls.root_marker + parent
	else:
	return cls.root_marker

However, when used with a S3 such as ArrowFSWrapper(pyarrow.fs.S3FileSystem()), pyarrow does not like leading slashes.

../.venv/lib/python3.12/site-packages/fsspec/spec.py:1106: in put
    self.put_file(lpath, rpath, callback=child, **kwargs)
../.venv/lib/python3.12/site-packages/fsspec/spec.py:1031: in put_file
    self.mkdirs(self._parent(os.fspath(rpath)), exist_ok=True)
../.venv/lib/python3.12/site-packages/fsspec/spec.py:1773: in mkdirs
    return self.makedirs(path, exist_ok=exist_ok)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
../.venv/lib/python3.12/site-packages/fsspec/implementations/arrow.py:23: in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
../.venv/lib/python3.12/site-packages/fsspec/implementations/arrow.py:195: in makedirs
    self.fs.create_dir(path, recursive=True)
pyarrow/_fs.pyx:638: in pyarrow._fs.FileSystem.create_dir
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   ???
E   pyarrow.lib.ArrowInvalid: Path cannot start with a separator ('/test-bucket-dcf315a3aff54e8db4eb2bf8e9e85f5b/path')

pyarrow/error.pxi:92: ArrowInvalid

If the root marker is reverted using something as follows, pyarrow is happy.

class Wrapper(ArrowFSWrapper):
        root_marker = ""

Wrapper(pyarrow.fs.S3FileSystem())

Potential Fix

Since it is a class member, probably it is necessary to have a separate wrapper for S3.

class ArrowS3FSWrapper(ArrowFSWrapper):
        root_marker = ""

[martindurant]

Another solution might be to provide root_marker as @property, so you can take a decision based on the backend.

[TLCFEM]

Another solution might be to provide root_marker as @property, so you can take a decision based on the backend.

It looks like the mental model behind it is that root_marker is a class level property.
Quite some class methods need to be redefined as normal methods to change root_marker to an instance level property.

[martindurant]

Quite some class methods need to be redefined as normal methods to change root_marker to an instance level property.

It could be done for this class alone?

[TLCFEM]

If I make the following change.

diff --git a/fsspec/implementations/arrow.py b/fsspec/implementations/arrow.py
index 227d509..ed79220 100644
--- a/fsspec/implementations/arrow.py
+++ b/fsspec/implementations/arrow.py
@@ -46,7 +46,9 @@ class ArrowFSWrapper(AbstractFileSystem):
 
     """
 
-    root_marker = "/"
+    @property
+    def root_marker(self):
+        return "Hello"
 
     def __init__(self, fs, **kwargs):
         global PYARROW_VERSION

Then the class methods will fail, for example:

from fsspec.implementations.arrow import ArrowFSWrapper

print(ArrowFSWrapper._parent("something"))

# prints:
# <property object at 0x7f57e5c66610>

[martindurant]

The descriptor protocol:

class desc:
    def __get__(self, obj, type=None):
        print(obj)
        return "/"

class ArrowFSWrapper(AbstractFileSystem):

    root_marker = desc()

will have obj=None on the class attribute, or the instance for regular attribute access.

On the other hand, you could set self.root_marker within __init__ if required.