Fix/multi inheritance #2639
Open
Fix/multi inheritance #2639
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
romain-intel
changed the base branch from
master
to
fix/support_editable_pkgs_more
romain-intel
force-pushed
the
fix/multi_inheritance
branch
from
ad214ed to
cc637df
Compare
Flow decorators were not properly merged in the case of multiple inheritance. For example: @project(name="foobar") class A(FlowSpec): pass class B(A): pass class C(B): pass Would result in the `project` decorator being included twice. This PR fixes this behavior and should support any arbitrary inheritance pattern. It also cleans up _flow_state (moving _flow_decorators into it) and fixes some other corner cases (I think). In particular, configs are now no longer merged as they are naturally inherited.
romain-intel
force-pushed
the
fix/multi_inheritance
branch
from
cc637df to
bd13ddf
Compare
valayDave
requested changes
Nov 10, 2025
Collaborator
valayDave
left a comment
valayDave
left a comment
There was a problem hiding this comment.
Okay I went and I test drove this. There are some changes needed since this currently actively breaks current the inheritance. The bug was very subtle but the suggestion I have added in the PR explains the only line of code we need to add to fix it!
I have currently tested the basic inheritance model. I will add one more review on thorougher testing with deployer / runner APIS alongside inheritance.
Comment on lines
421
to
454
| if cls._flow_state[FlowStateItems.CACHED_PARAMETERS] is not None: | ||
| cls._flow_state[FlowStateItems.CACHED_PARAMETERS] = None | ||
| else: | ||
| raise MetaflowInternalError( | ||
| "A non FlowMutator found in flow custom decorators" | ||
| ) | ||
|
|
||
| for step in cls._steps: | ||
| for deco in step.config_decorators: | ||
| if isinstance(deco, StepMutator): | ||
| inserted_by_value = [deco.decorator_name] + (deco.inserted_by or []) | ||
| debug.userconf_exec( | ||
| "Evaluating step level decorator %s for %s (pre-mutate)" | ||
| % (deco.__class__.__name__, step.name) | ||
| ) | ||
| deco.pre_mutate( | ||
| MutableStep( | ||
| cls, | ||
| step, | ||
| pre_mutate=True, | ||
| statically_defined=deco.statically_defined, | ||
| inserted_by=inserted_by_value, | ||
| ) | ||
| ) | ||
| else: | ||
| raise MetaflowInternalError( | ||
| "A non StepMutator found in step custom decorators" | ||
| ) | ||
|
|
||
| # Process parameters to allow them to also use config values easily | ||
| for var, param in cls._get_parameters(): | ||
| if param.IS_CONFIG_PARAMETER: | ||
| continue | ||
| param.init() |
Collaborator
There was a problem hiding this comment.
Simple inheritance based flows crashing in the _process_config_decorators. trivial Example to repro:
# inheritted.py
from metaflow import (
FlowSpec,
Config,
pyproject_toml_parser,
FlowMutator,
pypi_base,
)
class project_pypi(FlowMutator):
def pre_mutate(self, mutable_flow):
project_deps = dict(mutable_flow.configs).get("project_deps")
if project_deps["packages"]:
mutable_flow.add_decorator(
pypi_base, deco_kwargs=project_deps, duplicates=mutable_flow.ERROR
)
@project_pypi
class FBBase(FlowSpec):
project_deps = Config(
"project_deps", default="pyproject.toml", parser=pyproject_toml_parser
)# flow.py
from inheritted import FBBase
from metaflow import step
class MyFlow(FBBase):
@step
def start(self):
print("Hello World!")
self.next(self.end)
@step
def end(self):
print("Flow completed successfully!")
if __name__ == "__main__":
MyFlow()Crashing with stack trace:
Traceback (most recent call last):
File "/home/ubuntu/metaflow/metaflow/cli.py", line 723, in main
start(auto_envvar_prefix="METAFLOW", obj=state)
File "/home/ubuntu/metaflow/metaflow/_vendor/click/core.py", line 829, in __call__
return self.main(args, kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/metaflow/metaflow/_vendor/click/core.py", line 782, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "/home/ubuntu/metaflow/metaflow/cli_components/utils.py", line 65, in invoke
click.Command.invoke(self, ctx)
File "/home/ubuntu/metaflow/metaflow/_vendor/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/metaflow/metaflow/_vendor/click/core.py", line 610, in invoke
return callback(args, kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/metaflow/metaflow/_vendor/click/decorators.py", line 21, in new_func
return f(get_current_context(), args, kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/metaflow/metaflow/cli.py", line 477, in start
new_cls = ctx.obj.flow._process_config_decorators(config_options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/metaflow/metaflow/flowspec.py", line 452, in _process_config_decorators
if param.IS_CONFIG_PARAMETER:
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/metaflow/metaflow/user_configs/config_parameters.py", line 164, in __getattr__
raise AttributeError(key)
AttributeError: IS_CONFIG_PARAMETER
| setattr(cls, var, val) | ||
|
|
||
| cls._flow_state[_FlowState.SET_CONFIG_PARAMETERS] = to_save_configs | ||
| cls._flow_state[FlowStateItems.SET_CONFIG_PARAMETERS] = to_save_configs |
Collaborator
There was a problem hiding this comment.
Somehow cls._flow_state[FlowStateItems.SET_CONFIG_PARAMETERS] is returning [] even when to_save_configs has a list of tuples. Weird!
Comment on lines
+101
to
+118
| def __getitem__(self, key): | ||
| # ORDER IS IMPORTANT: we use inherited first and extend by whatever is in | ||
| # the flowspec | ||
| if key in self._merged_data: | ||
| return self._merged_data[key] | ||
|
|
||
| # We haven't accessed this yet so compute it for the first time | ||
| self_value = self._self_data.get(key) | ||
| inherited_value = self._inherited.get(key) | ||
|
|
||
| if self_value is not None: | ||
| self._merged_data[key] = self._merge_value(inherited_value, self_value) | ||
| return self._merged_data[key] | ||
| elif key in self._self_data: | ||
| # Case of CACHED_PARAMETERS; a valid value is None. It is never inherited | ||
| self._merged_data[key] = None | ||
| return None | ||
| raise KeyError(key) |
Collaborator
There was a problem hiding this comment.
The root cause of this bug is here . This bug is triggered because self._merged_data can be set when the user does something like in L545 (ie doing a getter like cls._flow_state[_FlowState.SET_CONFIG_PARAMETERS] ).
This pattern is a little funky because based on how __getitem__ is written, if a key is accessed once then it will be set in the self._merge_data and once it is set in the self._merge_data then even if we call __setitem__, the __getitem__ will ONLY return the older value since it's in self._merge_data
| setattr(cls, var, val) | ||
|
|
||
| cls._flow_state[_FlowState.SET_CONFIG_PARAMETERS] = to_save_configs | ||
| cls._flow_state[FlowStateItems.SET_CONFIG_PARAMETERS] = to_save_configs |
Collaborator
There was a problem hiding this comment.
Suggested change
| cls._flow_state[FlowStateItems.SET_CONFIG_PARAMETERS] = to_save_configs | |
| del cls._flow_state[FlowStateItems.SET_CONFIG_PARAMETERS] | |
| cls._flow_state[FlowStateItems.SET_CONFIG_PARAMETERS] = to_save_configs |
This suggestion fixes the bug I have mentioned here. The root cause of the bug is explained here.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.