Proposal: Unify Resource Management On Texera

[bobbairobot]

How Texera manages resources today

Texera manages six kinds of resources: workflows, datasets, models, computing units, environments, and warehouses. By managing a resource I mean three things: creating, reading, updating, and deleting it (CRUD), keeping its versions, and controlling who can access it. Today each kind is managed by its own code: workflows and warehouses in the Dashboard Service, datasets and models in the File Service, computing units in the Computing Unit Managing Service, and environments in the computing unit itself.

Major Limitation: Resources Are Managed Ad-hoc

I think Texera's resource management has one major limitation: resources are managed ad-hoc. Each kind was built on its own, with its own tables, endpoints, and rules, and the codebase has no single definition of a resource that every kind follows. As a result, the same three things are done differently for each kind:

• CRUD is defined per kind: each kind has its own endpoints with its own conventions. For example, a workflow is created with POST /workflow/create, but an environment with POST /pve/db. A dataset is deleted with DELETE /dataset/{did}, but a workflow with POST /workflow/delete.
• Versions are defined per kind: each versioned kind has its own version logic. For example, the Dashboard Service stores a workflow version as a JSON patch on every save that changes the workflow. The File Service instead stores a dataset version as a lakeFS commit when the user asks for one, and keeps a second copy of that code for model versions.
• Access control is defined per kind: each kind has its own way to record who owns a resource and who can access it. For example, ownership is recorded in a workflow_of_user table for workflows, in an owner_uid column for datasets and models, and in a uid column for the other kinds. The rules differ too: a user with read access can remove themselves from a shared workflow, but not from a shared dataset, and environments and warehouses cannot be shared at all.

The codebase itself shows how far the kinds have drifted apart. ResourceTables.scala was added so that the access rules could be written once, but its own comment says that it fits only datasets and models, because workflows record ownership in a separate table and computing units have no public flag.

Consequences: Backend Code Is Duplicated, Hard to Maintain, and Hard to Extend

The current backend codebase suffers from three problems:

• Duplicate code: the same logic is written once per kind. For example, ModelResource.scala repeats 31 of the 32 endpoints of DatasetResource.scala, and the access rules are implemented three times: for workflows, for datasets and models, and for computing units. The database repeats itself in the same way: 17 of its 42 tables are per-kind copies.
• Hard to add new resource kinds: a new kind has to repeat the tables, the endpoints, the version logic, and the access rules. For example, adding models took 12 backend pull requests and 5 database migrations, and 5 of those pull requests were refactors that made dataset code reusable for models. Down the road I expect Texera to add more kinds, such as agents, conversations, and operators, and each of them will pay this cost again.
• Hard to develop new features on top of resources: a feature that works across kinds has to be connected to each kind separately. For example, search maps the columns of each searchable kind by hand into one schema of 28 columns, and the agent service can read and save only workflows, because that is the only kind it has a client for. A new feature that works with every kind, such as an agent or an MCP server, would need the same boilerplate for each kind.

Proposal: Unify the Resource Management

I propose to unify resource management. By unify I mean: designing and implementing common primitives that are shared across all kinds of resources.

Based on current Texera's position, I see three major primitives:

P1: Resource CRUD: every kind has an id, a name, and an owner, and every kind needs the same operations: create, list, read, update, and delete.

P2: Resource version CRUD: a workflow version, a dataset version, and a model version are the same concept, which is an immutable snapshot of a resource. The dataset_version and model_version tables already have the same columns.

P3: Resource access control: every kind needs the same decision, which is whether a user may read or write a resource. The four *_user_access tables already have the same three columns: the user, the resource, and the privilege.

[image: Screenshot 2026-11-09 at 4.51.03 PM.png]

What stays specific to a kind is its own fields and actions, such as executing a workflow or uploading a file to a dataset. Once the common parts exist in one copy, a new kind defines only what is specific to it, and a feature built on the common parts works for every kind.

Implementation Proposal

I see at least two layers whose implementation needs to be refactored: the database schema layer and the API definition layer. Each layer needs its own detailed discussion, so below I only give my proposal for each.

Layer 1: Database Schema Layer

At this layer, my proposal is to store the common parts once, under two supertypes: Resource and Resource Version.

Before: each kind has its own tables for the common parts. Each kind keeps its own relationships to User, ten own and access relationships in total, and each versioned kind keeps its own version table.

[image: Animation of today's database schema: the plain diagram first, then redundant access tables, redundant version tables, and what adding one kind costs, highlighted one at a time]

After: every kind is a subtype of Resource, which has a single access relationship to User with the levels own, read, and write. Every version is a subtype of Resource Version, keyed by (rid, vno), so one lineage table can link a version to the versions it was derived from. A new kind adds only its own fields under Resource.

[image: Animation of the proposed database schema: the plain diagram first, then one access relationship, one version model, easy to record resource lineage, and easy to add new resource kinds, highlighted one at a time]

Layer 2: API Definition Layer

My proposal at this layer is to give every kind the same endpoints for the three common parts. For each part, the table below lists the endpoints before and after. In the After column, {kind} is the plural name of a kind, such as workflows or datasets, {rid} is the id of a resource, {vno} is the number of a version, and {uid} is the id of a user.

Resource CRUD

Before, each kind spells the same operation in its own way, and some kinds lack an operation. After, every kind has the same five operations.

Operation	Before	After
Create	POST /workflow/create, POST /dataset/create, POST /model/create, POST /computing-unit/create, POST /pve/db, POST /warehouse	POST /{kind}
List	GET /workflow/list, GET /dataset/list, GET /model/list, GET /computing-unit, GET /pve/db, GET /warehouse/status	GET /{kind}
Read	GET /workflow/{wid}, GET /dataset/{did}, GET /model/{mid}, GET /computing-unit/{cuid}; none for environments and warehouses	GET /{kind}/{rid}
Update (rename)	POST /workflow/update/name, POST /dataset/update/name, POST /model/update/name, PUT /computing-unit/{cuid}/rename/{name}, PUT /pve/db/{veid}; none for warehouses	PATCH /{kind}/{rid}
Delete	POST /workflow/delete, DELETE /dataset/{did}, DELETE /model/{mid}, DELETE /computing-unit/{cuid}/terminate, DELETE /pve/db/{veid}, DELETE /warehouse/{whid}	DELETE /{kind}/{rid}

Resource Version CRUD

Before, workflow versions have their own root path and no endpoint to create one, because a version is made whenever a workflow is saved, while dataset and model versions put the operation in the path. After, every versioned kind has the same three operations. A version is numbered within its resource and immutable, so it has no update or delete.

Operation	Before	After
Create	POST /dataset/{did}/version/create, POST /model/{mid}/version/create; none for workflows, where POST /workflow/persist makes a version as a side effect	POST /{kind}/{rid}/versions
List	GET /version/{wid}, GET /dataset/{did}/version/list, GET /model/{mid}/version/list	GET /{kind}/{rid}/versions
Read	GET /version/{wid}/{vid}, GET /dataset/{did}/version/{dvid}/rootFileNodes, GET /model/{mid}/version/{mvid}/rootFileNodes	GET /{kind}/{rid}/versions/{vno}

Resource Access Control

Before, the same sharing endpoints are defined four times, in three services. Environments and warehouses have none, and no endpoint checks access, so each service checks it in its own code. After, every kind is shared and checked in the same way.

Operation	Before	After
List	GET /access/workflow/list/{wid}, plus a copy each for datasets, models, and computing units	GET /{kind}/{rid}/access
Grant or change	PUT /access/workflow/grant/{wid}/{email}/{privilege}, plus the same three copies	PUT /{kind}/{rid}/access/{uid}
Revoke	DELETE /access/workflow/revoke/{wid}/{email}, plus the same three copies	DELETE /{kind}/{rid}/access/{uid}
Check	No endpoint: each service checks access in its own code	POST /access/evaluation