File tree Expand file tree Collapse file tree
src/datajoint/builtin_codecs Expand file tree Collapse file tree Original file line number Diff line number Diff line change 11"""
2- Hash-addressed storage codec with SHA256 deduplication.
2+ Hash-addressed storage codec with MD5-based deduplication.
33"""
44
55from __future__ import annotations
1212
1313class HashCodec (Codec ):
1414 """
15- Hash-addressed storage with SHA256 deduplication.
15+ Hash-addressed storage with content-addressed deduplication.
1616
1717 The ``<hash@>`` codec stores raw bytes using hash-addressed storage.
18- Data is identified by its SHA256 hash and stored in a hierarchical directory:
19- ``_hash/{hash[:2]}/{hash[2:4]}/{hash}``
18+ Data is identified by a 26-character Base32-encoded MD5 digest of the
19+ content, and stored at ``{hash_prefix}/{schema}/{hash}`` (``hash_prefix``
20+ defaults to ``_hash``). Stores with subfolding configured insert
21+ additional path segments: ``{hash_prefix}/{schema}/{fold1}/{fold2}/{hash}``.
2022
2123 The database column stores JSON metadata: ``{hash, store, size}``.
2224 Duplicate content is automatically deduplicated across all tables.
Original file line number Diff line number Diff line change @@ -226,7 +226,9 @@ class NpyCodec(SchemaCodec):
226226 Schema-addressed storage for numpy arrays as .npy files.
227227
228228 The ``<npy@>`` codec stores numpy arrays as standard ``.npy`` files
229- using schema-addressed paths: ``{schema}/{table}/{pk}/{attribute}.npy``.
229+ using schema-addressed paths: ``{schema_prefix}/{schema}/{table}/{pk}/{attribute}_{token}.npy``
230+ (``schema_prefix`` defaults to ``_schema``; ``{token}`` is a random
231+ per-write suffix).
230232 Arrays are fetched lazily via ``NpyRef``, which provides metadata access
231233 without I/O and transparent numpy integration via ``__array__``.
232234
@@ -269,7 +271,7 @@ class Recording(dj.Manual):
269271
270272 Storage Details:
271273 - File format: NumPy .npy (version 1.0 or 2.0)
272- - Path: ``{schema}/{table}/{pk}/{attribute}.npy``
274+ - Path: ``{schema_prefix}/{ schema}/{table}/{pk}/{attribute}_{token }.npy``
273275 - Database column: JSON with ``{path, store, dtype, shape}``
274276
275277 Deletion: Requires garbage collection via ``dj.gc.GarbageCollector``.
Original file line number Diff line number Diff line change @@ -15,7 +15,9 @@ class ObjectCodec(SchemaCodec):
1515 Schema-addressed storage for files and folders.
1616
1717 The ``<object@>`` codec provides managed file/folder storage using
18- schema-addressed paths: ``{schema}/{table}/{pk}/{field}/``. This creates
18+ schema-addressed paths: ``{schema_prefix}/{schema}/{table}/{pk}/{field}_{token}/``
19+ (``schema_prefix`` defaults to ``_schema``; ``{token}`` is a random
20+ per-write suffix). This creates
1921 a browsable organization in object storage that mirrors the database schema.
2022
2123 Unlike hash-addressed storage (``<hash@>``), each row has its own unique path
@@ -51,7 +53,7 @@ def make(self, key):
5153 Storage Structure:
5254 Objects are stored at::
5355
54- {store_root}/{schema}/{table}/{pk}/{field}/
56+ {store_root}/{schema_prefix}/{ schema}/{table}/{pk}/{field}_{token }/
5557
5658 Deletion: Requires garbage collection via ``dj.gc.GarbageCollector``.
5759
You can’t perform that action at this time.
0 commit comments