Should I use jobs or AutomationConditions?

[marcinsiewiera]

Hi,

I played with dagster and dbt integration for a while and I got stuck.

Basically I need 4 types of batches

1. Full load - every night at midnight all the models need to be processed according to DBT dag
2. 1 data source X (and all its downstream models) need to be fully processed at noon as well
3. 1 data source Y (and all its downstream models) need to be fully processed ad hoc
4. Data which comes from Change Data Capture source need to be processed between 8AM and 22PM every 15 minutes

Apart from this there are 2 additional requirements

1. During full load subset of the source need to wait for the availability of the source system.
2. When any asset (either data import or dbt model) fails it should be automatically retried 3 times after x minutes breaks

I've managed to load all the dbt assets and integrate it with data imports.
I could run entire E2E batch using job definition.
However, then came challanges:

1. I was able to implement data source availability polling, however entire dbt execution waited for all the sources to complete
2. dagster was issuing simple dbt build, which basically executed entire DAG in one command, however, when any model failed, it was not able retry failed model.

I tried playing with AutoMaterization and then AutomationCondition, but I did not got far.
I was able to:

• implement source polling, only dependent dbt models waited, all others run asynchronously
• implement retry policy (3 times every 5 minute after failure)

I failed with:

• implementing proper CDC flow (next run was overlapping previous one)
• putting it all together 4 cases above

Can you please point me in the right direction? Are there any code samples that would help with above use cases?

Regards,
Marcin