v0.14.10

What's Changed

Unofficial maintenance release. DataJoint no longer supports versions before 2.0. This release only fixes installation with setuptools 82 and later; the full test suite was not run.

Bug Fix

• Remove the setuptools pin and the otumat dependency (#1571, @tabedzki): plugin discovery now uses importlib.metadata instead of pkg_resources, which setuptools 82 removed, and otumat's two helper functions are copied into datajoint.plugin. setuptools and otumat are no longer runtime dependencies. Plugin behavior is unchanged.

Full Changelog: v0.14.9...v0.14.10

DataJoint Library 2.3.4

A patch release on the 2.3 line, focused on where data came from before it reached the pipeline.

Rows that enter the pipeline can now record where they came from. With provenance.capture enabled, a dj.Manual table carries a hidden _prov attribute, declared with the table and filled on insert. It is off by default, like jobs.add_job_metadata and for the same reason: it changes the DDL of every entry table declared afterwards, so that is a deployment's decision rather than a library default. Inside a pipeline provenance is structural — a computed row cannot exist without its declared upstream, so the foreign-key graph is the lineage — but at the boundary that runs out, and until now each pipeline invented its own record of external origin. _prov gives the boundary one shape, so that "which rows have no recorded origin" is a query rather than an audit. No author writes it: insert() takes no provenance argument, and the content comes from deployment configuration (provenance.source), from the connection, and — for an insert inside a make() — from the ingesting table and key, which is what makes the fan-out ingestion pattern traceable without a foreign key. A table declared before capture was enabled records nothing until dj.deploy.add_prov_column adds the slot. See Extrinsic Provenance at Entry Tables and Record Data Origin.

Three tiers gain a second name. dj.Entry, dj.Ingest and dj.Compute join dj.Manual, dj.Imported and dj.Computed. Each pair is one class rather than a subclass, so a table declared either way is identical — same SQL prefix, same tier detection, same describe() and diagram — and both names are permanent, with no deprecation and nothing to migrate. dj.Lookup and dj.Part are unchanged. The documentation keeps the original names for now; adopting the new ones as primary is slated for 2.4 (#1546).

Codecs receive connection context explicitly. encode() and decode() now take a context argument carrying schema, table, field and config, so key means what it means everywhere else — the primary key. Passing the calling connection's config is what matters: without it, path and backend resolution fall back to the global dj.config, which in a process holding connections for several users belongs to none of them and resolves a different store silently. Codecs written before 2.3.4 keep working unchanged; DataJoint passes context only to codecs whose signature declares it.

Two PostgreSQL defects in the hidden columns are fixed. Every platform-managed column — _job_start_time, _job_duration, _job_version, _prov, _singleton — is now declared in DataJoint notation and compiled through the type system, so each backend's mapping comes from one place. _job_start_time had been declared as a bare timestamp and so took microsecond precision on PostgreSQL where MySQL got datetime(3), and migrate.add_job_metadata_columns built its ALTER with backtick-quoted identifiers, which is a syntax error there — the retrofit had never run on PostgreSQL at all. Existing tables are untouched; only new declarations and retrofits change.

Upgrading from 2.3.3? No API breaks, and no DDL change: every new setting is off by default, so an unchanged schema declares under 2.3.4 exactly what it declared under 2.3.3. Reading a hidden attribute still requires SQL — to_arrays('_prov') and proj('_prov') raise, and a supported accessor arrives in 2.4 (#1562).

Full details in What's New in DataJoint 2.3.

---

What's Changed

• Pull MinIO from a registry that still serves it(#1559)@dimitri-yatsenko

⚡️ Enhancements

• Provenance capture defaults to off(#1568)@dimitri-yatsenko
• Codec context argument, additive to key(#1556)@dimitri-yatsenko
• Hidden _prov attribute for extrinsic provenance at Entry tables(#1555)@dimitri-yatsenko
• dj.Entry, dj.Ingest and dj.Compute as tier names on one axis(#1558)@dimitri-yatsenko

🐛 Bug Fixes

• Declare every platform-managed column through the DataJoint type system(#1565)@dimitri-yatsenko

Full Changelog: v2.3.3...v2.3.4

Release 2.3.3

A patch release on the 2.3 line, focused on storage credentials, diagram rendering, and type validation.

S3 stores no longer require static credentials. access_key and secret_key are now optional in an s3 store spec — omit both and the AWS credential chain resolves an ambient identity (EC2 instance profile, IRSA, ECS task role, SSO), matching how the gcs and azure protocols already behaved. This is what lets a pipeline run under an assumed role with no long-lived keys in its configuration. Setting exactly one of the two is now rejected at validation with a clear message instead of failing later inside the AWS client. See Configure Storage.

Diagrams were redesigned. dj.Diagram output moves to the DataJoint brand palette — Manual green, Lookup grey, Imported blue, Computed orange, with tier shapes unchanged — and renamed foreign keys are now drawn in amber, distinct from the Computed tier. Line weight now encodes cardinality and only cardinality: thick for one-to-one, thin for one-to-many. It previously also tried to signal the master-part relationship, which made the two unreadable together. A new display.diagram_theme setting takes auto (default), light, or dark; auto emits a single SVG that adapts to the viewer's light or dark mode. See Read Diagrams and Configuration.

Attribute types are validated at declaration and insert. Four related gaps closed, the most serious being that inserting a NumPy array into a native blob attribute silently stored the array's text representation, with no error on insert and none on fetch. decimal(M,D) unsigned is also accepted again after being rejected in 2.x, and misspelled native types are now caught at declaration rather than producing invalid DDL at the server.

Also: braces in a table or attribute comment no longer crash declaration, alter() no longer loses an added attribute's declared type, file-protocol stores are safe on Windows, and non-unique foreign-key columns are indexed on PostgreSQL.

Upgrading from 2.3.2? No API breaks. Two of the fixes above do reject input that previously passed silently — a misspelled native type at declaration, and a NumPy array inserted into a native blob. Both cases were already broken rather than working: the first produced invalid DDL at the server, the second corrupted the stored value. Anything affected was losing data.

Full details in What's New in DataJoint 2.3.

---

What's Changed

• docs: fix missing pixi -e flags in CONTRIBUTING.md(#1541)@gtouloumes
• ci: add Windows unit-test job to catch OS-specific path bugs(#1522)@dimitri-yatsenko

⚡️ Enhancements

• feat(#1543): adopt the DataJoint brand tier colors in diagram themes(#1544)@dimitri-yatsenko
• feat(#1532): modernize dj.Diagram rendering style(#1534)@dimitri-yatsenko
• fix(#1537): allow s3 stores without static credentials(#1538)@dimitri-yatsenko
• fix(#1512): index foreign-key columns on PostgreSQL (coverage-aware)(#1526)@dimitri-yatsenko

🐛 Bug Fixes

• fix: validate attribute types at declaration and insert (#1527, #1528, #1529, #1530)(#1531)@dimitri-yatsenko
• fix: braces in comments crash declaration, and alter() drops type metadata(#1548)@MilagrosMarin
• fix(diagram): collapsed-to-collapsed edges carry no per-foreign-key style(#1545)@dimitri-yatsenko
• fix(#1535): render diagrams with unresolved (raw table-name) nodes on PostgreSQL(#1536)@dimitri-yatsenko
• fix(#1532): diagram edge weight encodes cardinality, not master-part(#1533)@dimitri-yatsenko
• Use POSIX separators in file-protocol storage URLs and paths(#1521)@lochhh

📝 Documentation

• docs(codecs): thread the connection config in the SchemaCodec example(#1549)@MilagrosMarin
• docs(settings): correct the config-precedence docstring(#1542)@dimitri-yatsenko

Full Changelog: v2.3.2...v2.3.3

Release 2.3.2

Overview

v2.3.2 is a patch release — internal hardening and bug fixes on top of v2.3.1, with no breaking changes to the public API. The main items: the dependency graph moved to networkx.MultiDiGraph (correctly modeling multiple foreign keys between the same pair of tables), a ~jobs-table fix for UUID foreign-key-derived primary keys, removal of the process-wide sys.excepthook (uncaught exceptions now use Python's default handler, with tracebacks rendered by the log formatter), and a garbage-collection fix that avoids acting on partial storage listings. The package version is now derived from the git tag via hatch-vcs.

---

• test: cover parallel foreign-key edges to the same parent (#1492)(#1514)@dimitri-yatsenko
• tests: add coverage for deferred audit findings + isolate test_settings from ambient env(#1513)@MilagrosMarin
• chore: normalize stdlib copy imports to import copy(#1509)@dimitri-yatsenko
• docs(autopopulate): fix stale finally-comment; document upstream in class Attributes(#1510)@MilagrosMarin
• docs: post-2.3.1 stale-text sweep (3 small drift fixes)(#1511)@MilagrosMarin
• chore(release-flow): stale doc pointers, RELEASE_MEMO cleanup, and dead badge step(#1504)@MilagrosMarin
• docs(codecs): fix stale storage-path and hash-algorithm docstrings(#1505)@MilagrosMarin
• [github-actions]Update version.py to Release 2.3.1(#1502)@github-actions[bot]

⚡️ Enhancements

• refactor(graph): migrate Dependencies & Diagram to nx.MultiDiGraph (#1492)(#1508)@dimitri-yatsenko

🐛 Bug Fixes

• Fix jobs table generation for uuid FK-derived primary keys (#1515)(#1518)@dimitri-yatsenko
• Remove process-wide sys.excepthook; harden log formatter (#1516)(#1517)@dimitri-yatsenko
• fix(release): derive version from git tag via hatch-vcs (#1503)(#1507)@dimitri-yatsenko
• fix(gc): don't act on partial storage listings(#1506)@MilagrosMarin

Full Changelog: v2.3.1...v2.3.2

Release 2.3.1

Highlights

• populate() performance — self.upstream is now built lazily and dependencies are no longer reloaded per key, removing redundant work from populate() loops (#1499, #1493).
• Cascade / part-integrity fix — a cascaded delete now restricts the master from every restricted part and validates part integrity, fixing multi-part-master cascades (#1484).
• Garbage-collection correctness — codec-driven reference discovery with file-level, schema-addressed orphan matching (#1479, #1469).
• strict_provenance runtime guardrail retired — the best-effort runtime guard is removed; provenance remains an intrinsic structural property of the pipeline (#1489, #1488).
• Python 3.14 support — supported range extended through 3.14 with min/max exercised in CI; MySQL 8.4 LTS added to the test matrix (#1490, #1498).

⚡️ Enhancements

• fix(#1493): build self.upstream lazily; stop reloading deps per populate key(#1499)@dimitri-yatsenko
• ci: test MySQL 8.0 and 8.4 (LTS) in the matrix; parameterize MySQL image(#1498)@dimitri-yatsenko
• test(cascade): pin part-integrity paths (rollback, empty-materialization, U3 arm)(#1485)@dimitri-yatsenko
• Retire strict_provenance runtime guardrail (#1488)(#1489)@dimitri-yatsenko
• Extend Python support to 3.14; test min/max in CI(#1490)@dimitri-yatsenko

🐛 Bug Fixes

• fix(cascade): restrict master from every restricted Part; validate part_integrity(#1484)@dimitri-yatsenko
• ci(pre-commit): pin numpy<2.2 for mypy stub compatibility(#1491)@dimitri-yatsenko
• fix(#1469): codec-driven GC reference discovery + file-level orphan matching(#1479)@dimitri-yatsenko

📝 Documentation

• docs: clarify _propagate_part_to_master materialization is delete-order, not just MySQL 1093(#1495)@dimitri-yatsenko
• docs(#1484): document pair-keyed part-to-master invariant in _propagate_restrictions(#1494)@dimitri-yatsenko

Full Changelog: v2.3.0...v2.3.1

Release 2.3.0

Highlights

DataJoint 2.3 builds on the 2.2.x line with a set of features centered on provenance — knowing, and being able to trust, exactly which upstream data a computed result was derived from.

The headline is the provenance trinity. Diagram.trace constructs the upstream view of any result; self.upstream makes those declared ancestors ergonomic to read inside make(); and the opt-in strict_provenance flag turns the long-standing "read only from declared dependencies, write only to self" convention into something the framework actively checks. Together they move DataJoint's core provenance promise from a convention people hope to follow to one the framework helps construct and — when enabled — enforce.

Around that:

• SparkAdapter Codec Protocol — typed codecs can expose their decoded values as Spark-native types, opening lakehouse / Delta-Sharing consumers (e.g. Databricks) to columns that were previously opaque blobs.
• dj.deploy.set_replica_identity — a new dj.deploy module configures PostgreSQL REPLICA IDENTITY for change-data-capture pipelines.
• Cascade fix — part_integrity="cascade" is now correct across Part-of-Part and renamed-foreign-key chains (the same upward-propagation machinery Diagram.trace builds on).

No breaking changes. strict_provenance defaults off and everything else is additive, so existing pipelines are unaffected. See What's New in 2.3 for the full narrative.

🚀 Features

• feat(#1425): strict_provenance config flag for runtime enforcement(#1474)@dimitri-yatsenko
• feat(#1424): self.upstream property for pre-restricted ancestor access(#1473)@dimitri-yatsenko
• feat(#1423): Diagram.trace() for upstream restriction propagation(#1471)@dimitri-yatsenko
• feat(#1458): SparkAdapter Codec Protocol(#1472)@dimitri-yatsenko
• feat(deploy): set_replica_identity for PostgreSQL CDC (#1447)(#1466)@dimitri-yatsenko

🐛 Bug Fixes

• fix(#1429): cascade through FK chain for part_integrity="cascade"(#1468)@dimitri-yatsenko
• fix(#1454): refresh ~lineage on every @schema decoration(#1467)@dimitri-yatsenko
• fix(staged_insert): converge metadata shape with ObjectCodec.encode(#1465)@dimitri-yatsenko

Full Changelog: v2.2.4...v2.3.0

Release 2.2.4

Highlights

A small follow-up patch on 2.2.3 cleaning up rough edges discovered during the docs review:

• dj.StorageAdapter and dj.get_storage_adapter() exposed at top level. The Storage Adapter plugin contract documented in the spec used these names as if they were re-exported from the top-level datajoint namespace, but they were not. Plugin authors following the spec literally hit AttributeError. Both symbols are now in datajoint.__all__ and importable as dj.StorageAdapter / dj.get_storage_adapter.
• packaging declared as an explicit dependency. src/datajoint/migrate.py imports packaging.version.Version, but the package was never declared in pyproject.toml. With Python 3.12 venvs no longer shipping setuptools (which used to pull packaging transitively), pip install datajoint==2.2.3 into a clean Py3.12 venv hit ModuleNotFoundError: No module named 'packaging' on import datajoint. Fixed.
• Source-side version markers refreshed in settings.py docstrings and comments to match the public docs.

⚡️ Enhancements

• feat: export StorageAdapter and get_storage_adapter at top level (#1463)@dimitri-yatsenko

🐛 Bug Fixes

• fix: declare packaging as an explicit dependency (#1462)@dimitri-yatsenko

📝 Documentation

• docs: refresh settings.py version markers (#1461)@dimitri-yatsenko

Companion docs: datajoint/datajoint-docs#172 — env-var configuration of stores + Storage Adapter API spec.

Full Changelog: v2.2.3...v2.2.4

Release 2.2.3

Highlights

This patch release adds env-var configuration for object stores (closes the gap for env-var-only deployments such as the DataJoint platform, Kubernetes, and Lambda) and ships several minor fixes and documentation improvements.

• DJ_STORES — JSON-encoded copy of the stores dict, same shape as in datajoint.json. Replaces the file's stores block when set, with arbitrary adapter-defined attribute names (token, api_key, workspace_url…).
• DJ_IGNORE_CONFIG_FILE (default false) — when true, skips datajoint.json, the project .secrets/, and /run/secrets/datajoint/ entirely. Hard guarantee that no file on disk leaks into config.
• .secrets/stores.<name>.<attr> now accepts any attribute, not only access_key / secret_key — supports plugin-registered adapters (e.g. Databricks Unity Catalog Volumes) with non-S3-style credential fields.

Precedence (high → low): programmatic dj.config["stores"] > DJ_STORES > datajoint.json > .secrets/stores.<name>.<attr> (fills missing attrs only).

⚡️ Enhancements

• feat(config): DJ_STORES env var + DJ_IGNORE_CONFIG_FILE flag + arbitrary .secrets/ attrs (#1452)@dimitri-yatsenko
• chore: remove .vscode and .devcontainer (#1459)@dimitri-yatsenko

📝 Documentation

• docs: extend datajoint.migrate timeline to 2.4 or 2.5 (#1456)@dimitri-yatsenko
• docs: fix code formatting on rendered codecs API page (#1455)@dimitri-yatsenko
• docs: use relative path for README pipeline image (#1450)@MilagrosMarin
• docs: refresh README pipeline illustration (#1448)@dimitri-yatsenko
• docs: cite DataJoint 2.0 manuscript and add RRID (#1449)@dimitri-yatsenko

Companion docs: datajoint/datajoint-docs#172

Full Changelog: v2.2.2...v2.2.3

Release 2.2.2

⚡️ Enhancements

• feat: add StorageAdapter plugin system for third-party storage protocols (#1432) @kushalbakshi
• refactor: simplify index query, warn on MariaDB connections (#1439) @dimitri-yatsenko
• feat: export AutoPopulate and Job in public API (#1431) @dimitri-yatsenko

🐛 Bug Fixes

• fix(#1442): scan_*_references reads raw JSON metadata instead of decoded codec output (#1444) @kushalbakshi
• fix(#1438): preserve json flag for MariaDB longtext-aliased columns (#1443) @dimitri-yatsenko
• fix(#1433): clear error for leading-underscore attribute names (#1441) @dimitri-yatsenko
• fix: remove EXPRESSION column dependency in get_indexes_sql() (#1437) @dimitri-yatsenko
• fix: preserve NOT NULL constraint in migrate_columns() (#1435) @dimitri-yatsenko

Full Changelog: v2.2.1...v2.2.2

Release 2.2.1

Released: 2026-04-09

⚡️ Enhancements

• feat: add database.name setting for PostgreSQL connections (#1426) @kushalbakshi
  • Configurable database name via dj.config["database.name"], DJ_DATABASE_NAME env var, or database_name kwarg on Connection
  • Extracts _build_connect_kwargs() helper to eliminate duplicated connection parameter construction
  • __repr__ and logger show database name when set (e.g., user@host:5432/my_db)
  • Warns when database.name is set with MySQL backend
  • Deprecates database_prefix (warning emitted at Schema.activate())

📝 Documentation

• Fix CONTRIBUTING.md link for PyPI (#1422) @dimitri-yatsenko
• Add PostgreSQL configuration to CONTRIBUTING.md
• Document database.name setting and database_prefix deprecation (datajoint-docs#159)
• Fix dead links in llms.txt — now auto-generated from mkdocs nav (datajoint-docs#160)

Full Changelog: v2.2.0...v2.2.1