longblob silently stores str(array) under DataJoint 2.x — 3 affected attributes, no error raised #48

longblob silently stores str(array) under DataJoint 2.x — the data is destroyed, nothing raises

element-event declares 3 longblob attributes. Under DataJoint 2.x every one of
them silently corrupts the data written to it. I lost some time to this and it
seems worth writing down, because the failure has no symptom at the point where
it happens.

DataJoint 2.0.0 shipped 2026-02-03 as a complete rewrite; 2.3.2 is current.
Because install_requires says datajoint>=0.13, a fresh pip install element-event
today resolves 2.x. (It then fails at import on dj.schema — that part is loud,
and is what the accompanying PR fixes. This issue is about what happens after
you get past it.)

What 2.x changed

In 0.14.x, longblob meant "a DataJoint blob": pack numpy on write, unpack on
read. In 2.x, blob handling moved into an extensible codec system (<blob>), and
a bare longblob is now treated as a raw native MySQL column. The write path
coerces whatever you hand it to a string; the read path returns the raw bytes.

Reproduction

import numpy as np, datajoint as dj          # datajoint 2.3.2, MySQL 9.7.1

schema = dj.Schema("probe")

@schema
class Legacy(dj.Manual):                     # how this element declares its blobs
    definition = "id : int\n---\ntrace : longblob\n"

@schema
class Codec(dj.Manual):                      # the 2.x codec
    definition = "id : int\n---\ntrace : <blob>\n"

a = np.arange(5000, dtype="float32") / 7.0   # 20,000 bytes
Legacy().insert1({"id": 1, "trace": a})
Codec().insert1({"id": 1, "trace": a})

longblob -> bytes      90 bytes
            b'[0.0000000e+00 1.4285715e-01 2.8571430e-01 ... 7.1385712e+02 7.1400000e+02\n 7.1414288e+02]'
<blob>   -> ndarray    dtype=float32 shape=(5000,)   array_equal: True

Read that stored value again: it is str(array). NumPy's own abbreviated text
repr, ellipsis and all. 4,994 of the 5,000 values no longer exist anywhere.
The insert did not raise. The fetch did not raise. information_schema reports
the column as longblob in both cases, so nothing about the schema looks wrong.

Smaller arrays are text too, just not yet truncated:

np.arange(4, dtype='float32')/7   ->  b'[0.         0.14285715 0.2857143  0.42857143]'
np.arange(9).reshape(3,3)         ->  b'[[0. 1. 2.]\n [3. 4. 5.]\n [6. 7. 8.]]'

dtype and shape are gone in every case; dj.blob.unpack() on the returned bytes
returns None rather than failing, because it was never a DataJoint blob.

There is one signal, at declaration time only, and it does not mention data
loss:

[WARNING]: Native type 'longblob' is used in attribute 'trace'.
           Consider using a core DataJoint type for better portability.

What is affected in element-event

event.py (1), trial.py (2) — the three attribute_blob attributes on Event.Attribute, Trial.Attribute and Block.Attribute

Fix

Declare <blob> instead of longblob. That restores 0.14.x round-trip
behaviour exactly, and the underlying MySQL column type is unchanged
(longblob either way), so it is a definition edit and not a table migration —
as long as it happens before real rows exist. Once a table has been populated
under 2.x with longblob, the original arrays are not recoverable from the
database.

I cannot send this one as a PR: 0.14.9 rejects <blob> with
Support for Adapted Attribute types is disabled, so it would break every
current user. It is on a branch instead, split so the backward-compatible part
can be taken separately:

• compat-fixes — dj.schema→dj.Schema, boolean→bool,
  enum("x")→enum('x'). Works on both 0.14.9 and 2.3.2. Sent as a PR.
• datajoint-2.x — the above + all 3 longblob → <blob>. 2.x only.
  https://github.com/akshay-jaggi/element-event/tree/datajoint-2.x

Every change on both branches was applied mechanically with regexes and reviewed
line by line; no element behaviour is altered, only attribute-type spellings.

Environment

datajoint 2.3.2 · MySQL 9.7.1 · Python 3.11 · element-event at main
(last commit 2025-05-20). Cross-checked against datajoint 0.14.9 on the same
server.

Related PR (the backward-compatible subset): #47