Bbox problem

[dreichCSL]

Thanks for this repo!

I'm trying to train BUTD with GQA, but running into several issues (fixing one causes the next etc). It seems that there is an issue with the bounding box calculation. Could you explain why the expected size of bboxes is 5 values and not 4? This is defined in openvqa/core/base_cfgs.py.
I'm getting the following error when running as is:

File "openvqa/openvqa/core/base_dataset.py", line 87, in forward
return self.gqa_forward(feat_dict)
File "openvqa/openvqa/models/butd/adapter.py", line 55, in gqa_forward
bbox_feat = self.bbox_linear(bbox_feat)
File "lib/python3.9/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
return forward_call(*input, **kwargs)
File "lib/python3.9/site-packages/torch/nn/modules/linear.py", line 103, in forward
return F.linear(input, self.weight, self.bias)
RuntimeError: mat1 and mat2 shapes cannot be multiplied (51200x4 and 5x1024)
(Note: batch size is 512)

The bbox layer expects 5 input values, but the data preprocessing code only produces 4 (which makes perfect sense to me, but the 5 is hard-coded, which seems weird).
If I change the code to 4, I'm getting a gradient/backprop error and it crashes, as well.

Any ideas here? Thanks!

[dreichCSL]

Update:

I modified the following files to be able to train BUTD without errors:
(Using torch 1.11.0, spacy 2.3.7, en_vectors_web_lg 2.3.0)

1. Reverted changes from commit Fixing the dimension mismatch bug for gqa_loader #75 (increases bbox feature dim to 5 again, like it used to be before it was "fixed" in openvqa/datasets/gqa/gqa_loader.py)
2. Added USE_AUX_FEAT = False in openvqa/configs/gqa/butd.yml
3. Modified Line 50 in openvqa/openvqa/models/butd/net.py from inplace=True -> inplace=False

Point 3 fixes issues in gradient calculation during training because of in-place operations.

Disclaimer: I've so far only attempted to train a BUTD model with GQA from scratch for which these changes were necessary. I can't say if any of the listed changes would cause issues to already existing models or even other models ((1) could affect non-BUTD models that use GQA)