[FEAT] DAGs as Durable Tasks

[mrkaye97]

Is your feature request related to a problem? Please describe.

The internals of DAGs are extremely costly for the components of the Hatchet Engine, especially the database, largely because of how much coordination needs to occur (locking, e.g.) to support replays, cancellations, status updates, analytics, and so on.

Now that durable tasks are a genuinely first-class feature of Hatchet, we can build off of the existing durable execution tooling internally to rewrite DAGs as an internal durable task, powered by an operator. The idea would be that instead of having a genuine notion of a "DAG" in the codebase and database schema, we'd perform the orchestration dynamically at runtime on the engine. This has the obvious benefit of allowing us to simplify the database schema significantly by removing all of the DAG-related tables (v1_dag, v1_dag_to_task, v1_dags_olap, v1_dag_to_task_olap, v1_workflow_concurrency, and so on).

Describe the solution you'd like

See above - DAGs are orchestrated at runtime by a durable task via an operator.

Known issues

There are a number of known issues still to fix in the current implementation:

• Ensure the APIs that return DAGs all support the new durable task backend, in addition to the legacy backend for rendering DAGs
• Ensure all permutations of skip / cancel / wait works with each type (and combination of types) of conditions
• Ensure that cancellation behavior works as intended (when an upstream task is cancelled, all downstream tasks must also cancel)
• Ensure skipping behavior works as intended (when all upstream tasks have been skipped, the downstream task(s) is also skipped)
• Ensure replay behavior works, allowing for replaying from a specific failed task in the DAG, and consider how "later" tasks in the event log should behave in this case (maybe use idempotency keys to ensure they don't run multiple times?)

---

🤖 AI Disclosure

No AI was used for this issue