[Fix] Fix get_flops crash for SAN model - #3875
Conversation
|
|
There was a problem hiding this comment.
Pull request overview
Fixes a crash when running tools/analysis_tools/get_flops.py on SAN configs by aligning MultimodalEncoderDecoder._forward() with the multimodal encode_decode() path (image encoder + text encoder + triplet input to the decode head), and adds an explicit “unsupported” guard for SAN in the FLOPs script.
Changes:
- Update
MultimodalEncoderDecoder._forward()to run text+image encoding and pass[inputs, clip_features, class_embeds]to the decode head. - Add a SAN (
SideAdapterCLIPHead) unsupported check inget_flops.pywith aNotImplementedError.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| tools/analysis_tools/get_flops.py | Adds a guard to short-circuit FLOPs computation for SAN. |
| mmseg/models/segmentors/multimodal_encoder_decoder.py | Changes tensor-mode forward to use multimodal inputs compatible with SAN’s decode head. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| classifier_embeds = self.text_encoder() | ||
| clip_inputs = inputs | ||
| if self.asymetric_input: | ||
| clip_inputs = F.interpolate( | ||
| inputs, scale_factor=self.encoder_resolution, mode='bilinear') | ||
| x = self.image_encoder(clip_inputs) | ||
| return self.decode_head.forward([inputs, x, classifier_embeds], []) |
There was a problem hiding this comment.
MultimodalEncoderDecoder._forward() now always calls decode_head.forward(..., []) with a second positional arg. This breaks decode heads whose forward() only accepts a single inputs argument (e.g. the ExampleDecodeHead used in tests/test_models/test_segmentors/test_multimodal_encoder_decoder.py), causing mode='tensor' to raise a TypeError. Consider branching based on the decode head type/signature (e.g., only pass deep_supervision_idxs for SideAdapterCLIPHead/SAN), and otherwise call decode_head.forward(multimodal_inputs) with one argument.
| # SAN requires text encoder outputs that _forward() doesn't provide | ||
| raise NotImplementedError( | ||
| 'SAN is not supported yet. See #3866') |
There was a problem hiding this comment.
The new SAN guard comment is now inaccurate: after this PR, _forward() does run the text encoder and passes the [inputs, clip_features, class_embeds] triplet into the decode head. If SAN is still intentionally unsupported in get_flops.py due to FLOPs undercounting/tracing limitations, please update the comment and the NotImplementedError message to reflect the real limitation (e.g., tracing/caching makes FLOPs incomplete) rather than implying _forward() lacks required inputs.
| # SAN requires text encoder outputs that _forward() doesn't provide | |
| raise NotImplementedError( | |
| 'SAN is not supported yet. See #3866') | |
| # SAN is intentionally unsupported here because FLOPs analysis can be | |
| # incomplete for SideAdapterCLIPHead: tracing/caching behavior may | |
| # undercount work even though _forward() provides the required inputs. | |
| raise NotImplementedError( | |
| 'SAN FLOPs are not supported in get_flops.py because ' | |
| 'tracing/caching can make the computation incomplete. ' | |
| 'See #3866') |
Running
get_flops.pyon a SAN config crashes with:TypeError: SideAdapterCLIPHead.forward() missing 1 required positional argument: 'deep_supervision_idxs'The issue is that
MultimodalEncoderDecoder._forward()passes only backbone features todecode_head.forward(x), butSideAdapterCLIPHead.forward()expects a(imgs, clip_features, class_embeds)triplet and adeep_supervision_idxsarg.Fixed
_forward()to match theencode_decode()path — runs both text encoder and image encoder, passes the full triplet. Also added SAN to the unsupported guard inget_flops.pysince the text encoder FLOPs aren't captured by the tracing.Fixes #3866