Zip Store #37

brokkoli71 · 2025-11-28T12:59:20Z

fixes Add ZipStore #18
was on top of Memory store #36 and is rebased onto main now

Further Changes

adds store methods getSize and getInputStream
fixes S3 store get range

TODO:

Implement ZipStore
Test ZipStore (merge with MemoryStore tests)
buffered implementation

Requirements derived from https://ngff.openmicroscopy.org/rfc/9/index.html

ZIP64 format
Disable ZIP-level compression
Option to add archive comments in the ZIP file header
no use of nested or multi-part ZIP archives
"The root-level zarr.json file SHOULD be the first ZIP file entry and the first entry in the central directory header; other zarr.json files SHOULD follow immediately afterwards, in breadth-first order."

EDIT:

enforcing zip64 on smaller files resulted in header errors when writing with org.apache.commons.compress. For now zip64 will only be used when needed

normanrz · 2025-11-30T19:39:16Z

I wonder how writing would be implemented on the ZipStore? Maybe it could have an internal MemoryStore that is used for buffering. And then, there could be a close or flush method that flushes the buffered contents into a zip file.

Reading could use a different implementation that works on the zip file directly. Maybe it even makes sense to have 2 separate ZipStore implementations.

In any case, it would be cool if the ZipStore(s) could wrap around any other store so that zip files can be written to disk, s3 etc.

brokkoli71 · 2025-12-12T13:43:35Z

I am thinking about the default behavior of writing to the buffered zip store. I think it might be confusing if calling array.write(...) does not actually write something to the zip file unless zipstore.flush() is called. Maybe per default flush should be called within every write, or there could be an optional argument when creating a buffered zip store like autoFlush or flushOnWrite that defaults to true but can be turned off.

normanrz · 2025-12-12T15:09:43Z

I think the name BufferedZipStore actually conveys pretty nicely that things are not written directly. But an autoFlush option that is false by default would be an interesting addition.

brokkoli71 · 2025-12-12T16:12:23Z

src/main/java/dev/zarr/zarrjava/store/ReadOnlyZipStore.java

+    @Nullable
+    @Override
+    public ByteBuffer get(String[] keys, long start, long end) {
+        ByteBuffer buffer = underlyingStore.read();


The ReadOnlyZipStore will read the entire ZIP file here, but only parses the required regions.
This could be changed so that it only reads the required bytes from the ZIP file.
One possibility would be to add store.getInputStream() and parse this stream instead. I think that would enable reading from large zip files, that don't fit into memory. Do you think this should be taken care of?

Yeah. I think the primary purpose of the ReadOnlyZipStore is to read large ZIP files directly from the storage.

normanrz · 2025-12-12T15:11:48Z

src/main/java/dev/zarr/zarrjava/store/BufferedZipStore.java

+    private final Store.ListableStore bufferStore;
+    private String archiveComment;
+
+    private void writeBuffer() throws IOException{


I think this is fine for now, but it creates copies of the entire zip archive in memory, so it could be expensive. Maybe we could add a streaming-write option the the store interface in the future.

normanrz · 2025-12-12T16:14:01Z

src/main/java/dev/zarr/zarrjava/store/ReadOnlyZipStore.java

+    @Nullable
+    @Override
+    public ByteBuffer get(String[] keys, long start, long end) {
+        ByteBuffer buffer = underlyingStore.read();


Yeah. I think the primary purpose of the ReadOnlyZipStore is to read large ZIP files directly from the storage.

@nonnull

diff --git c/src/main/java/dev/zarr/zarrjava/store/ZipStore.java i/src/main/java/dev/zarr/zarrjava/store/ZipStore.java new file mode 100644 index 0000000..054917f --- /dev/null +++ i/src/main/java/dev/zarr/zarrjava/store/ZipStore.java @@ -0,0 +1,72 @@ +package dev.zarr.zarrjava.store; + +import javax.annotation.Nonnull; +import javax.annotation.Nullable; +import java.nio.ByteBuffer; +import java.nio.file.Path; +import java.nio.file.Paths; +import java.util.stream.Stream; + +public class ZipStore implements Store, Store.ListableStore { + @nonnull + private final Path path; + + public ZipStore(@nonnull Path path) { + this.path = path; + } + + public ZipStore(@nonnull String path) { + this.path = Paths.get(path); + } + + + @OverRide + public Stream<String> list(String[] keys) { + return Stream.empty(); + } + + @OverRide + public boolean exists(String[] keys) { + return false; + } + + @nullable + @OverRide + public ByteBuffer get(String[] keys) { + return null; + } + + @nullable + @OverRide + public ByteBuffer get(String[] keys, long start) { + return null; + } + + @nullable + @OverRide + public ByteBuffer get(String[] keys, long start, long end) { + return null; + } + + @OverRide + public void set(String[] keys, ByteBuffer bytes) { + + } + + @OverRide + public void delete(String[] keys) { + + } + + @nonnull + @OverRide + public StoreHandle resolve(String... keys) { + return new StoreHandle(this, keys); + } + + @OverRide + public String toString() { + return this.path.toUri().toString().replaceAll("\\/$", ""); + } + +} diff --git c/src/test/java/dev/zarr/zarrjava/Utils.java i/src/test/java/dev/zarr/zarrjava/Utils.java new file mode 100644 index 0000000..0026200 --- /dev/null +++ i/src/test/java/dev/zarr/zarrjava/Utils.java @@ -0,0 +1,40 @@ +package dev.zarr.zarrjava; + +import java.io.File; +import java.io.FileInputStream; +import java.io.IOException; +import java.util.zip.ZipEntry; +import java.util.zip.ZipOutputStream; + +public class Utils { + + static void zipFile(File fileToZip, String fileName, ZipOutputStream zipOut) throws IOException { + if (fileToZip.isHidden()) { + return; + } + if (fileToZip.isDirectory()) { + if (fileName.endsWith("/")) { + zipOut.putNextEntry(new ZipEntry(fileName)); + zipOut.closeEntry(); + } else { + zipOut.putNextEntry(new ZipEntry(fileName + "/")); + zipOut.closeEntry(); + } + File[] children = fileToZip.listFiles(); + for (File childFile : children) { + zipFile(childFile, fileName + "/" + childFile.getName(), zipOut); + } + return; + } + FileInputStream fis = new FileInputStream(fileToZip); + ZipEntry zipEntry = new ZipEntry(fileName); + zipOut.putNextEntry(zipEntry); + byte[] bytes = new byte[1024]; + int length; + while ((length = fis.read(bytes)) >= 0) { + zipOut.write(bytes, 0, length); + } + fis.close(); + } + +} diff --git c/src/test/java/dev/zarr/zarrjava/ZarrStoreTest.java i/src/test/java/dev/zarr/zarrjava/ZarrStoreTest.java index 4a369c9..c7d2ab4 100644 --- c/src/test/java/dev/zarr/zarrjava/ZarrStoreTest.java +++ i/src/test/java/dev/zarr/zarrjava/ZarrStoreTest.java @@ -7,16 +7,22 @@ import dev.zarr.zarrjava.v3.*; import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.MethodSource; import org.junit.jupiter.params.provider.CsvSource; import software.amazon.awssdk.auth.credentials.AnonymousCredentialsProvider; import software.amazon.awssdk.regions.Region; import software.amazon.awssdk.services.s3.S3Client; +import java.io.File; +import java.io.FileOutputStream; import java.io.IOException; import java.nio.file.Files; import java.util.Arrays; import java.util.stream.Stream; +import java.nio.file.Path; +import java.util.zip.ZipOutputStream; +import static dev.zarr.zarrjava.Utils.zipFile; import static dev.zarr.zarrjava.v3.Node.makeObjectMapper; public class ZarrStoreTest extends ZarrTest { @@ -132,4 +138,72 @@ public class ZarrStoreTest extends ZarrTest { Assertions.assertEquals("test group", attrs.getString("description")); } + + @test + public void testZipStore() throws ZarrException, IOException { + Path sourceDir = TESTOUTPUT.resolve("testZipStore"); + Path targetDir = TESTOUTPUT.resolve("testZipStore.zip"); + FilesystemStore fsStore = new FilesystemStore(sourceDir); + writeTestGroupV3(fsStore, true); + + FileOutputStream fos = new FileOutputStream(targetDir.toFile()); + ZipOutputStream zipOut = new ZipOutputStream(fos); + + File fileToZip = new File(sourceDir.toUri()); + zipFile(fileToZip, fileToZip.getName(), zipOut); + zipOut.close(); + fos.close(); + + ZipStore zipStore = new ZipStore(targetDir); + assertIsTestGroupV3(Group.open(zipStore.resolve()), true); + } + + static Stream<Store> localStores() { + return Stream.of( +// new ConcurrentMemoryStore(), + new FilesystemStore(TESTOUTPUT.resolve("testLocalStoresFS")), + new ZipStore(TESTOUTPUT.resolve("testLocalStoresZIP.zip")) + ); + } + + @ParameterizedTest + @MethodSource("localStores") + public void testLocalStores(Store store) throws IOException, ZarrException { + boolean useParallel = true; + Group group = writeTestGroupV3(store, useParallel); + assertIsTestGroupV3(group, useParallel); + } + + int[] testData(){ + int[] testData = new int[1024 * 1024]; + Arrays.setAll(testData, p -> p); + return testData; + } + + Group writeTestGroupV3(Store store, boolean useParallel) throws ZarrException, IOException { + StoreHandle storeHandle = store.resolve(); + + Group group = Group.create(storeHandle); + Array array = group.createArray("array", b -> b + .withShape(1024, 1024) + .withDataType(DataType.UINT32) + .withChunkShape(5, 5) + ); + array.write(ucar.ma2.Array.factory(ucar.ma2.DataType.UINT, new int[]{1024, 1024}, testData()), useParallel); + group.createGroup("subgroup"); + group.setAttributes(new Attributes(b -> b.set("some", "value"))); + return group; + } + + void assertIsTestGroupV3(Group group, boolean useParallel) throws ZarrException { + Stream<dev.zarr.zarrjava.core.Node> nodes = group.list(); + Assertions.assertEquals(2, nodes.count()); + Array array = (Array) group.get("array"); + Assertions.assertNotNull(array); + ucar.ma2.Array result = array.read(useParallel); + Assertions.assertArrayEquals(testData(), (int[]) result.get1DJavaArray(ucar.ma2.DataType.UINT)); + Attributes attrs = group.metadata().attributes; + Assertions.assertNotNull(attrs); + Assertions.assertEquals("value", attrs.getString("some")); + } }

There are apparently cases where release: [created] leads to the deploy not being triggered. Attempting an expansion to [created, published]. diff --git c/pom.xml i/pom.xml index f4c1091..9c9d45d 100644 --- c/pom.xml +++ i/pom.xml @@ -6,7 +6,7 @@ <groupId>dev.zarr</groupId> <artifactId>zarr-java</artifactId> - <version>0.0.9</version> + <version>0.0.6</version> <name>zarr-java</name> @@ -123,6 +123,17 @@ </dependency> </dependencies> + <distributionManagement> + <snapshotRepository> + <id>ossrh</id> + <url>https://s01.oss.sonatype.org/content/repositories/snapshots</url> + </snapshotRepository> + <repository> + <id>ossrh</id> + <url>https://s01.oss.sonatype.org/service/local/staging/deploy/maven2/</url> + </repository> + </distributionManagement> + <repositories> <repository> <id>unidata-all</id> @@ -221,16 +232,6 @@ </execution> </executions> </plugin> - <plugin> - <groupId>org.sonatype.central</groupId> - <artifactId>central-publishing-maven-plugin</artifactId> - <version>0.8.0</version> - <extensions>true</extensions> - <configuration> - <publishingServerId>central</publishingServerId> - <tokenAuth>true</tokenAuth> - </configuration> - </plugin> </plugins> </build> </project>

Use new sonatype plugin for upload

…nstead of own implementation

1. leading slashes in paths 2. no sizes in entry headers

…loadBuffer

brokkoli71 changed the title ~~add ZipStore tests~~ Zip Store Nov 30, 2025

brokkoli71 self-assigned this Dec 12, 2025

brokkoli71 commented Dec 12, 2025

View reviewed changes

normanrz reviewed Dec 12, 2025

View reviewed changes

brokkoli71 and others added 23 commits December 18, 2025 15:46

refactor and unify outputs of Store.list

2611738

read zip store

5e2e017

Bump to 0.0.7

99081e5

Bump to 0.0.8

268890e

Use new sonatype plugin for upload

Bump to 0.0.9

b9e6db4

write buffer of zip store

08afc36

use apache commons compress for zip file read and write

0cacc5b

set Zip64Mode.AsNeeded

5b74372

test Zipped OME-Zarr requirements

ee92e27

Sort zarr.json files in breadth-first order within BufferedZipStore

a344655

manually read zip comment

ea16692

refactor read zip comment

7e0164f

test zip store with v2

02445e0

use com.fasterxml.jackson.databind.util.ByteBufferBackedInputStream i…

caafad0

…nstead of own implementation

add ReadOnlyZipStore

dbc559c

fix ReadOnlyZipStore for zips with

db57be7

1. leading slashes in paths 2. no sizes in entry headers

add BufferedZipStore parameter flushOnWrite

768bd62

fix testMemoryStore

38bec27

default flushOnWrite to false

bdcbc46

fix s3 store get range

9014fef

add store.getInputStream

5c74445

brokkoli71 added 2 commits December 18, 2025 15:47

add store.getSize

7e0e90e

improve performance of ReadOnlyZipStore.getArchiveComment

086d3f8

brokkoli71 force-pushed the zip-store branch from 40de1be to 086d3f8 Compare December 18, 2025 14:48

inherit zipstores from common parent and reduce buffers in memory in …

7d4f487

…loadBuffer

brokkoli71 marked this pull request as ready for review December 18, 2025 17:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Zip Store #37

Zip Store #37

Uh oh!

brokkoli71 commented Nov 28, 2025 •

edited

Loading

Uh oh!

normanrz commented Nov 30, 2025

Uh oh!

brokkoli71 commented Dec 12, 2025

Uh oh!

normanrz commented Dec 12, 2025

Uh oh!

brokkoli71 Dec 12, 2025

Uh oh!

normanrz Dec 12, 2025

Uh oh!

normanrz Dec 12, 2025

Uh oh!

normanrz Dec 12, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Zip Store #37

Are you sure you want to change the base?

Zip Store #37

Uh oh!

Conversation

brokkoli71 commented Nov 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Further Changes

TODO:

EDIT:

Uh oh!

normanrz commented Nov 30, 2025

Uh oh!

brokkoli71 commented Dec 12, 2025

Uh oh!

normanrz commented Dec 12, 2025

Uh oh!

brokkoli71 Dec 12, 2025

Choose a reason for hiding this comment

Uh oh!

normanrz Dec 12, 2025

Choose a reason for hiding this comment

Uh oh!

normanrz Dec 12, 2025

Choose a reason for hiding this comment

Uh oh!

normanrz Dec 12, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

brokkoli71 commented Nov 28, 2025 •

edited

Loading