Is there a Celery-specific RunLauncher (without k8s)?

[skeller88]

I'm not seeing a Celery RunLauncher, but I'm seeing a celery executor, which I guess you can launch with any of the run launchers? I'm confused as to why there's a CeleryK8sRunLauncher in the first place, because it seems like that run launcher uses the celery broker as the queue, and then k8s as the worker/executor. It seems more accurate to think of the Celery broker/backend as the RunCoordinator?

[gibsondan]

Using the CeleryK8sRunLauncher is required if you're using the celery_k8s_job_executor, which is discussed here and lets you use celery to maintain global concurrency limits across multiple runs: https://docs.dagster.io/deployment/guides/kubernetes/deploying-with-helm-advanced#per-op-limits-in-kubernetes

I think if we were redoing the celery+k8s integration from scratch in 2023 we probably would not have a separate run launcher for this, and would instead just use the K8sRunLauncher there as well, as in practice they do the same thing (launch the run worker for a given run in a k8s pod)

[XBeg9]

Can CeleryK8sRunLauncher and celery_executor be used together? celery_k8s_job_executor with CeleryK8sRunLauncher works, but it spins up steps for each op... that's too granular.

Reported this as an issue #22027

[Radiergummi]

This is really confusing; I'm specifically not using K8s, and the docs just don't really say anything on that. Should I use the default run launcher? How do I do this? Does it need configuration? Nobody knows.

[guitcastro]

Ok, there is a CeleryRunLauncher not mentioned in the docs. But there is a example of it.

run_launcher:
  module: dagster_celery.launcher
  class: CeleryRunLauncher
  config:
    broker:
      env: CELERY_BROKER_URL
    backend:
      env: CELERY_RESULT_BACKEND
    default_queue: long-queue

[lmicverm]

We are having the same problem, did you manage to get things working using the CeleryRunLauncher?