SenseBridge

Official code release for SenseBridge, a reliability-aware framework for Vision-Audio-Tactile (VAT) Cross-Modal Retrieval (CMR) in robotics.

SenseBridge is designed for cross-modal retrieval under multi-sensory heterogeneity and scenario-dependent modality quality variation. Given a query from one modality, the model retrieves semantically corresponding samples from the remaining modality or fused retrieval space, supporting robot perception, material understanding, and embodied interaction in incomplete or noisy environments.

This repository contains the main training and evaluation code used for the SenseBridge pipeline. Datasets and pretrained checkpoints are not bundled in the release.

Overview

Cross-modal retrieval is a core capability for embodied agents. A robot that hears an impact sound should be able to retrieve the corresponding object in vision or touch; a robot that sees an object should be able to anticipate tactile or acoustic evidence aligned with the same semantic entity.

Existing VAT-CMR systems still face two fundamental difficulties:

• Modality heterogeneity: vision, audio, and touch have substantially different feature structures, dimensions, and statistical properties. Naive projection or concatenation often introduces semantic interference and unstable class boundaries in the shared space.
• Modality quality variability: the most reliable supervisory modality is not fixed. In real scenarios, perceptual quality changes with environment, object category, and interaction dynamics. A fixed dominant modality can therefore become mismatched to the actual retrieval condition.

SenseBridge addresses these issues with a unified reliability-aware VAT-CMR framework.

Main Contributions

• Unified VAT-CMR framework: SenseBridge provides an end-to-end training pipeline for tri-modal retrieval, avoiding fragmented pairwise retrieval spaces and improving semantic consistency across modalities.
• Reliability-aware supervision: a pre-training modality reliability assessment module adaptively selects the dominant supervisory modality instead of relying on a predefined one.
• Distribution-level cross-modal alignment: class-conditional prototype optimal transport aligns modality-specific substructures beyond instance-level matching, improving intra-class compactness and alignment stability.
• Release-oriented codebase: this repository reorganizes the main research code into a cleaner public version centered on training, evaluation, and reproducible experiment entry points.

Method Components

SenseBridge in this release is organized around three stages:

1. Stage 1: joint tri-branch representation learning across vision, audio, and touch
2. Stage 2: PMRA-based dominant-modality selection and reliability-aware supervision
3. Stage 3: triplet training with class-conditional prototype OT

The corresponding code modules are:

• main.py: end-to-end pipeline entry
• train_c_entropy.py: stage-2 training
• PMQA.py: PMRA implementation in the current code release
• train_triplet_loss.py: stage-3 retrieval training
• Optimal_Transport.py: prototype OT implementation
• performance_eval.py: final test evaluation

Repository Highlights

This release focuses on the parts required to run the SenseBridge pipeline:

• model definitions
• data loading
• PMRA-based adaptive supervision
• prototype OT training
• retrieval evaluation
• example experiment launcher scripts

The release intentionally excludes:

• large datasets
• pretrained checkpoints
• cached outputs and experiment artifacts

Environment

Tested environment:

• Linux
• Python 3.8.10
• PyTorch 2.0.1
• torchvision 0.15.2

Installation

git clone https://github.com/lvkailin0118/SenseBridge.git
cd SenseBridge
conda create -n sensebridge python=3.8 -y
conda activate sensebridge
pip install -r requirements-open-source.txt

Notes:

• requirements-open-source.txt is the recommended release dependency list.
• requirements.txt is the original full environment export from a Linux CUDA machine and is kept for reference.
• If your local machine cannot install the pinned PyTorch build directly, install a compatible torch and torchvision pair first, then install the remaining dependencies.

Dataset Preparation

The code searches the dataset in the following order:

• ./data
• ../data
• the path specified by VAT_CMR_DATASET_DIR

Expected structure:

data/
  audio/
    train/<object_id>/*.wav
    val/<object_id>/*.wav
    test/<object_id>/*.wav
  touch/
    train/<object_id>/*.png
    val/<object_id>/*.png
    test/<object_id>/*.png
  vision/
    train/<object_id>/*.png
    val/<object_id>/*.png
    test/<object_id>/*.png

Current release assumptions:

• The released configuration targets the 20-object setup defined by OBJECT_NUMBERS in load_data.py.
• If you use different object IDs or a different split, update OBJECT_NUMBERS accordingly.
• Audio is padded or truncated to length 132300.
• Vision and touch images are resized to 246 x 246.

Quick Start

SenseBridge defaults are already aligned with the recommended full framework configuration used in scripts/PMQA_OT.sh:

• adaptive dominant-modality selection enabled through the legacy code flag --ce_auto_dominant pmqa
• PMRA warm-up steps = 10000
• similarity metric = cosine
• stage-3 prototype OT enabled by default
• prototype OT defaults: K=4, classes_per_step=8, samples_per_class=8, lambda_ot=0.1, lambda_local=0.3

Recommended single-run example:

python main.py \
  --query_modality audio \
  --retrieval_modality fused \
  --results_name runs/sensebridge_audio_fused

Run all 9 query-retrieval settings with the provided script:

bash scripts/PMQA_OT.sh

Run only the PMRA stage:

python main.py \
  --query_modality audio \
  --retrieval_modality fused \
  --pmqa_warmup_only \
  --results_name runs/pmqa_probe_only

Naming Note

In the paper, the reliability assessment module is referred to as PMRA.

In the current code release, the implementation keeps the legacy identifiers:

• file: PMQA.py
• function: run_pmqa
• CLI option family: --pmqa_*
• mode flag: --ce_auto_dominant pmqa

These names are preserved for backward compatibility with earlier experiment scripts and checkpoints. Conceptually, they correspond to the PMRA module described in the paper.

Running the Full Pipeline

main.py executes the complete SenseBridge workflow in sequence:

1. joint feature learning over the three modality branches
2. PMRA-guided cross-entropy training
3. prototype-OT-enhanced triplet training
4. final test-set evaluation

Implementation note:

• the current public training entry still contains an optional tactile warm-start path inherited from the original codebase
• the SenseBridge method view exposed in this README follows the paper formulation, where the first major training stage is the joint learning of the three modality branches

Useful control flags:

• --skip_triplet: run only the pretraining and stage-2 parts
• --skip_perf: skip the final test evaluation
• --resume: reuse an existing results directory and skip completed stages when possible

Common Overrides

If you run main.py with only the required query setting, the released code already uses the full SenseBridge configuration by default. The following arguments are the most useful overrides.

• --query_modality {audio,tactile,visual}
• --retrieval_modality {fused,audio,tactile,visual}

PMRA-related overrides:

• --ce_auto_dominant {none,pmqa}: default is pmqa
• --pmqa_force_dominant {none,audio,tactile,visual}
• --pmqa_joint_loss [true|false]: default is False
• --pmqa_warmup_steps: default is 10000
• --pmqa_similarity_metric {cka,cosine,euclidean}: default is cosine
• --pmqa_warmup_only

Prototype OT overrides:

• --triplet_ot_mode {none,proto}: default is proto
• --proto_num_classes: default is 20
• --proto_k: default is 4
• --proto_classes_per_step: default is 8
• --proto_samples_per_class: default is 8
• --proto_ot_weight: default is 0.1
• --proto_local_weight: default is 0.3
• --proto_ot_reg: default is 0.1
• --proto_ot_iters: default is 50
• --proto_warmup_epochs: default is 1000
• --proto_learn_pi

Output control:

• --results_dir
• --results_name

Full argument list:

python main.py --help

Output Structure

Each experiment creates a results directory that typically contains:

• c-entropy-results-<query>-query/
• triplet-results-<query>-query/
• performance-evaluation-results/
• result.txt

Typical artifacts include:

• stage-2 checkpoint: c-entropy-model.pth
• stage-2 embeddings: *_embeddings_{train,test}.npy
• stage-2 metrics: c_entropy_stage2_metrics.json
• triplet checkpoint: triplet_model_best.pth
• prototype OT checkpoint: proto_ot_best.pth
• training curves and retrieval metrics

Nine-Run Script

scripts/PMQA_OT.sh launches the complete SenseBridge setting over 9 query-retrieval combinations:

• query = audio, retrieval = tactile, visual, fused
• query = tactile, retrieval = audio, visual, fused
• query = visual, retrieval = audio, tactile, fused

The script uses the same default SenseBridge configuration now embedded in the codebase, so the script and the CLI defaults are consistent.

Reproducibility Notes

• In PMRA mode, do not pass --dominating_modality; the dominant modality is selected automatically.
• --retrieval_modality must be different from --query_modality.
• Final evaluation depends on the triplet-stage checkpoint. If triplet training is skipped, final evaluation is also skipped.
• This release does not include datasets or checkpoints, so a fresh run requires local dataset preparation.
• The current public version is centered on the main training and evaluation scripts rather than the full internal experiment archive.

Acknowledgement

SenseBridge is released on top of the VAT-CMR task setting and extends it with reliability-aware dominant-modality selection and class-conditional prototype OT alignment.

Citation

If you use SenseBridge in your research, please cite the paper associated with this repository.

Citation template:

@article{sensebridge2026,
  title   = {SenseBridge: A Reliability-Aware Framework for Vision-Audio-Tactile Cross-Modal Retrieval},
  author  = {Author1 and Author2 and Author3 and ...},
  journal = {arXiv preprint arXiv:XXXX.XXXXX},
  year    = {2026}
}

BibTeX template:

@article{sensebridge2026,
  title   = {SenseBridge: A Reliability-Aware Framework for Vision-Audio-Tactile Cross-Modal Retrieval},
  author  = {Author1 and Author2 and Author3 and ...},
  journal = {arXiv preprint arXiv:XXXX.XXXXX},
  year    = {2026},
  url     = {https://github.com/lvkailin0118/SenseBridge}
}

Please replace the placeholder metadata with the final publication information once the paper is public.

Contact

For questions about the code release, implementation details, or reproduction issues, please use one of the following:

• open a GitHub issue in this repository
• contact the corresponding author by email

Contact template:

Name: <Corresponding Author Name>
Email: <your-email@example.com>
Affiliation: <Your Institution>

TODO

Planned release items:

• add final publication metadata and complete BibTeX
• provide checkpoint download links
• provide dataset access instructions or external links
• release more experiment scripts for additional query-retrieval settings
• add qualitative retrieval visualizations and pipeline figure

License

This project is released under the MIT License. See LICENSE for details.