[BUG] OOM error with long-running eager workflows

[LanaMorkva]

Flyte & Flytekit version

flyteidl 1.16.6
flytekit 1.16.14

Describe the bug

When using long-running eager workflows, the flytekit Controller loops inside the _poll function. Every two seconds, it creates a new Deck object and appends it to the global Flyte context's decks list. This list grows indefinitely, inevitably leading to OOM error if the workflow runs long enough.

This is a critical bug because it cannot be resolved by simply increasing the RAM of the workflow pod, at a larger execution scale, the issue will eventually recur. This limitation significantly restricts the potential use cases of the Flyte framework for long-orchestration tasks.

Location of the bug:
flytekit/core/worker_queue.py, line 329

Proposed Solution:
The issue can be resolved by setting auto_add_to_deck=False during deck creation. However, I am uncertain about the potential side effects of this change:

if len(self.entries) > 0:
    with self.entries_lock:
        html = self.render_html()
        FlyteContextManager.push_context(self.remote._ctx)
        # Setting auto_add_to_deck=False prevents the object from being appended to the global context
        Deck("Eager Executions", html, auto_add_to_deck=False).publish()
        FlyteContextManager.pop_context()

Expected behavior

The RAM requirements of the pod running the Eager workflow orchestration task should remain stable. Memory usage should not scale with the number of tasks spawned in the workflow or the total execution time.

Additional context to reproduce

No response

Screenshots

No response

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes

[welcome]

Thank you for opening your first issue here! 🛠

[runllm]

It could be possible that this OOM issue with long-running eager workflows is due to the global Flyte context's decks list growing without bound, as each poll appends a new Deck object. The data sources do not explicitly mention this bug, but they do note that in eager workflows, all task/subworkflow outputs are materialized as Python values in the pod, which can lead to OOM if large or many objects accumulate in memory. There is no mention in the data sources of a built-in mechanism to limit or prune the decks list in eager workflows, nor is there documentation on the auto_add_to_deck parameter or its side effects in this context.

Though I'm not entirely certain, it might be that setting auto_add_to_deck=False could prevent the Deck from being appended to the global context and thus mitigate the OOM, but the data sources do not confirm whether this is safe or recommended. The sources do highlight that eager workflows are still in beta and have limitations, including all outputs being materialized in memory, which aligns with your observed issue. Would you like a more detailed breakdown of the relevant documentation and possible workarounds?

Sources:

• https://flyte.org/blog/flyte-1-6-runtime-metrics-ui-imagespec-external-backend-plugins-and-more#starCount
• https://github.com/flyteorg/flyte-sdk/blob/main/examples/higher_order_patterns/README.md
  Note: I wasn’t able to find highly relevant data sources, but above are a few potentially relevant links.

---

This is an AI-generated response and your feedback is appreciated! Please leave a 👍 if this is helpful and 👎 if it is not.

Mention @RunLLM to ask follow-up questions. See runllm.com for more!

[muskan-creates352]

Hi @LanaMorkva, I’d like to work on this issue.

From the description, it seems the memory leak is caused by repeatedly adding Deck objects to the global Flyte context inside the _poll loop.

Would setting auto_add_to_deck=False during Deck creation be the correct approach to prevent this, or are there any potential side effects I should be aware of?

Thanks!

[muskan-creates352]

Hi @LanaMorkva,

I’d like to work on this issue.

From my understanding, the memory leak is caused by repeatedly adding Deck objects to the global context inside the polling loop.

I’m thinking of fixing this by setting auto_add_to_deck=False during Deck creation to prevent accumulation.

Please let me know if this approach looks correct or if you suggest any improvements.

Thanks!

[LanaMorkva]

Hi @muskan-creates352,

Thank you for your interest in this issue! Yes, I believe setting auto_add_to_deck=False or at least making it configurable would solve the issue. However I'm not sure if there are any consequences to this solution, given I'm just a user of the library. I quickly skimmed over the source code and didn't see any places where it's essential for the global context to populate this deck list, though.