FairFactor Temporal FER

Robust video facial expression recognition with:

fairness-aware modeling,
heavy teacher and fast student architectures,
teacher-student distillation,
ablation and comparison suites,
paper artifact generation,
checkpoint publishing utilities.

This README is a complete runbook for everything implemented in this repository.

1) What this repository does

Main objective: optimize both overall FER quality and subgroup robustness (worst_group_f1) on diverse data.

Implemented capabilities:

Metadata building from generic datasets and from coded Recordings datasets (labels.csv + Coding.txt).
Video preprocessing (face-centric clips + quality features).
Training:

Stage 1 SSL warmup,
Stage 2 supervised training,
Stage 3 GroupDRO fine-tuning,
teacher-student distillation.

Evaluation with publication metrics:

accuracy, macro_f1, weighted_f1, balanced_accuracy, worst_group_f1, ece, brier, per-group F1, confusion matrix.

Automated experiment suites:

ablation suite,
model comparison suite with speed/efficiency benchmarking.

Paper output generation:

markdown + LaTeX tables,
metric and tradeoff figures.

Publish interface for checkpoints:

package folder + model card + manifest + zip.

2) Repository structure

fairfactor-temporal-fer/
  apps/
    preprocess_ui.py
    publish_ui.py
  configs/
    fairfactor_stage1_ssl.yaml
    fairfactor_stage2_sup.yaml
    fairfactor_stage3_groupdro.yaml
    teacher_heavy_stage2.yaml
    student_fast_stage2.yaml
    distill_teacher_student.yaml
    suites/
      ablation_suite.yaml
      comparison_suite.yaml
  data/
    annotations_template.csv
  scripts/
    build_metadata.py
    preprocess_dataset.py
    train_stage1_ssl.py
    train_stage2.py
    train_stage3.py
    train_distill.py
    eval.py
    run_ablation_suite.py
    run_comparison_suite.py
    build_paper_artifacts.py
    publish_checkpoint.py
    run_train.sh
    launch_preprocess_ui.sh
    launch_publish_ui.sh
  src/fairfactor/
    data/
    losses/
    models/
    train/
    utils/

3) One-time setup

cd /home/ans353/Documents/myproject/fairfactor-temporal-fer
python -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install -e .
export PYTHONPATH=src

Optional sanity checks:

python -c "import torch; print('torch', torch.__version__)"
python -c "import cv2; print('cv2 ok')"

4) Data formats and label mapping

4.1 Required metadata columns for training

video_id, path, subject_id, emotion_label, ethnicity, gender, age_bucket, split

4.2 Extra columns generated by this repo

emotion_code, ethnicity_code, cohort_code, cohort_label

After preprocessing:

processed_path, frames_in, frames_out, face_detected_ratio, quality_blur, quality_brightness, status

4.3 Recordings dataset support (your case)

If your videos are in data/Recordings with:

data/Recordings/labels.csv
data/Recordings/Coding.txt

then metadata builder automatically:

parses XX-YY-ZZ-WW naming convention,
maps emotion codes to text labels (happy/sad/angry/surprised/neutral),
maps ethnicity codes to text labels,
fills required training columns,
handles slight filename irregularities where possible.

4.4 Parametric fairness group definition (`group_cols`)

All training/eval configs support:

data:
  group_cols: [ethnicity]

This controls which metadata columns define fairness groups.

Ethnicity only: group_cols: [ethnicity]
Gender only: group_cols: [gender]
Age only: group_cols: [age_bucket]
Intersectional: group_cols: [ethnicity, gender, age_bucket]

Current configs in this repo are set to ethnicity-only.

5) Data preparation commands

5.1 Build metadata for Recordings (recommended)

source .venv/bin/activate
export PYTHONPATH=src

python scripts/build_metadata.py \
  --video-root data/Recordings \
  --output data/metadata_recordings.csv

Optional explicit paths:

python scripts/build_metadata.py \
  --video-root data/Recordings \
  --labels-csv data/Recordings/labels.csv \
  --coding-txt data/Recordings/Coding.txt \
  --output data/metadata_recordings.csv

5.2 Build metadata for generic datasets

python scripts/build_metadata.py \
  --video-root data/raw_videos \
  --annotations data/annotations.csv \
  --output data/metadata.csv

5.3 Preprocess videos into face clips

python scripts/preprocess_dataset.py \
  --metadata-csv data/metadata_recordings.csv \
  --output-dir data/processed_clips \
  --output-metadata data/metadata_recordings_processed.csv \
  --target-size 224 \
  --target-fps 8 \
  --min-face-size 40 \
  --bbox-margin 0.2

5.4 Point configs to your processed metadata

sed -i 's#data/metadata_processed.csv#data/metadata_recordings_processed.csv#g' configs/*.yaml

5.5 Data prep UI

bash scripts/launch_preprocess_ui.sh

6) Training commands

Before training, ensure your config data.metadata_csv points to the processed metadata CSV.

6.1 Stage 1 SSL warmup (optional)

python scripts/train_stage1_ssl.py --config configs/fairfactor_stage1_ssl.yaml

6.2 Stage 2 baseline FairFactor

python scripts/train_stage2.py --config configs/fairfactor_stage2_sup.yaml

6.3 Stage 3 GroupDRO fine-tuning

python scripts/train_stage3.py --config configs/fairfactor_stage3_groupdro.yaml

6.4 Train heavy teacher

python scripts/train_stage2.py --config configs/teacher_heavy_stage2.yaml

Expected artifact:

outputs/teacher_stage2/best_stage2.pt

6.5 Train fast student (non-distilled baseline)

python scripts/train_stage2.py --config configs/student_fast_stage2.yaml

Expected artifact:

outputs/student_stage2/best_stage2.pt

6.6 Distill student from teacher

Set teacher.checkpoint in configs/distill_teacher_student.yaml to your trained teacher checkpoint, then run:

python scripts/train_distill.py --config configs/distill_teacher_student.yaml

Expected artifacts:

outputs/student_distill/best_distill.pt
outputs/student_distill/result_distill.json

6.7 Train SOTA baselines used for comparison

# R3D-18 baseline
python scripts/train_stage2.py --config configs/r3d18_stage2.yaml

# R(2+1)D-18 baseline
python scripts/train_stage2.py --config configs/r2plus1d18_stage2.yaml

Expected artifacts:

outputs/r3d18_stage2/best_stage2.pt
outputs/r2plus1d18_stage2/best_stage2.pt

6.8 Simple baseline train script

bash scripts/run_train.sh

7) Evaluation commands

7.1 Single model evaluation

python scripts/eval.py \
  --config configs/student_fast_stage2.yaml \
  --checkpoint outputs/student_stage2/best_stage2.pt \
  --split test \
  --output outputs/student_stage2/eval_test.json

7.2 Evaluate distilled student

python scripts/eval.py \
  --config configs/distill_teacher_student.yaml \
  --checkpoint outputs/student_distill/best_distill.pt \
  --split test \
  --output outputs/student_distill/eval_test.json

8) Ablation suite

Config: configs/suites/ablation_suite.yaml

Generate experiment configs only:

python scripts/run_ablation_suite.py \
  --suite-config configs/suites/ablation_suite.yaml

Execute full ablation training/evaluation suite:

python scripts/run_ablation_suite.py \
  --suite-config configs/suites/ablation_suite.yaml \
  --execute

Outputs:

outputs/ablation/ablation_summary.csv
outputs/ablation/ablation_summary.json

9) Comparison suite (quality + efficiency)

Config: configs/suites/comparison_suite.yaml

This suite compares:

teacher_full,
r3d18_baseline_full,
r2plus1d18_baseline_full,
student_plain,
student_distilled.

Before running comparison, make sure all corresponding checkpoints exist:

outputs/teacher_stage2/best_stage2.pt
outputs/r3d18_stage2/best_stage2.pt
outputs/r2plus1d18_stage2/best_stage2.pt
outputs/student_stage2/best_stage2.pt
outputs/student_distill/best_distill.pt

python scripts/run_comparison_suite.py \
  --suite-config configs/suites/comparison_suite.yaml

Outputs:

outputs/comparison/comparison_summary.csv
outputs/comparison/comparison_summary.json

10) Paper artifacts (tables + figures)

python scripts/build_paper_artifacts.py \
  --comparison-csv outputs/comparison/comparison_summary.csv \
  --ablation-csv outputs/ablation/ablation_summary.csv \
  --output-dir outputs/paper

Generates:

outputs/paper/table_main.md
outputs/paper/table_main.tex
outputs/paper/table_main.csv
outputs/paper/figure_main_metrics.png
outputs/paper/figure_efficiency_tradeoff.png
outputs/paper/figure_ablation_delta.png (if ablation CSV provided)
outputs/paper/paper_artifacts_summary.json

11) Publish checkpoints

11.1 CLI packaging

python scripts/publish_checkpoint.py \
  --checkpoint outputs/student_distill/best_distill.pt \
  --model-name fairfactor_student_distilled_v1 \
  --config configs/distill_teacher_student.yaml \
  --metrics outputs/student_distill/result_distill.json \
  --output-root outputs/published \
  --license MIT \
  --contact you@example.com

Optional extras:

add multiple --extra-file path/to/file
disable zip with --no-zip

11.2 Publish UI

bash scripts/launch_publish_ui.sh

Packaging outputs include:

packaged artifacts directory,
MODEL_CARD.md,
manifest.json with SHA256 hashes,
optional .zip archive.

12) Complete command checklist (all functionalities)

Use these in order for a full experiment cycle on Recordings, including SOTA baseline comparison and ablation.

# 0) setup
cd /home/ans353/Documents/myproject/fairfactor-temporal-fer
source .venv/bin/activate
export PYTHONPATH=src

# 1) metadata from Recordings labels/coding
python scripts/build_metadata.py \
  --video-root data/Recordings \
  --output data/metadata_recordings.csv

# 2) preprocessing
python scripts/preprocess_dataset.py \
  --metadata-csv data/metadata_recordings.csv \
  --output-dir data/processed_clips \
  --output-metadata data/metadata_recordings_processed.csv

# 3) (optional) ssl warmup
python scripts/train_stage1_ssl.py --config configs/fairfactor_stage1_ssl.yaml

# 4) FairFactor stage2 (required for stage3 init checkpoint)
python scripts/train_stage2.py --config configs/fairfactor_stage2_sup.yaml

# 5) FairFactor stage3 GroupDRO
python scripts/train_stage3.py --config configs/fairfactor_stage3_groupdro.yaml

# 6) teacher
python scripts/train_stage2.py --config configs/teacher_heavy_stage2.yaml

# 7) student baseline
python scripts/train_stage2.py --config configs/student_fast_stage2.yaml

# 8) distillation
python scripts/train_distill.py --config configs/distill_teacher_student.yaml

# 9) SOTA baselines for comparison
python scripts/train_stage2.py --config configs/r3d18_stage2.yaml
python scripts/train_stage2.py --config configs/r2plus1d18_stage2.yaml

# 10) evaluate distilled model
python scripts/eval.py \
  --config configs/distill_teacher_student.yaml \
  --checkpoint outputs/student_distill/best_distill.pt \
  --split test \
  --output outputs/student_distill/eval_test.json

# 11) ablation suite
python scripts/run_ablation_suite.py \
  --suite-config configs/suites/ablation_suite.yaml \
  --execute

# 12) comparison suite (includes teacher, student, distilled, R3D-18, R(2+1)D-18)
python scripts/run_comparison_suite.py \
  --suite-config configs/suites/comparison_suite.yaml

# 13) paper artifacts
python scripts/build_paper_artifacts.py \
  --comparison-csv outputs/comparison/comparison_summary.csv \
  --ablation-csv outputs/ablation/ablation_summary.csv \
  --output-dir outputs/paper

# 14) publish packaged checkpoint
python scripts/publish_checkpoint.py \
  --checkpoint outputs/student_distill/best_distill.pt \
  --model-name fairfactor_student_distilled_v1 \
  --config configs/distill_teacher_student.yaml \
  --metrics outputs/student_distill/result_distill.json \
  --output-root outputs/published

13) Troubleshooting

ModuleNotFoundError: fairfactor

Run: export PYTHONPATH=src (or pip install -e .).

ModuleNotFoundError: cv2

Install deps: pip install -e ..

OOM during training

Reduce clip_len, image_size, batch_size in config.

Pretrained backbone download issues

The backbone loader falls back to random init automatically.

Unknown labels in metadata

Typically caused by non-coded filenames in Recordings.
Inspect rows with emotion_label == unknown in metadata CSV.

14) Citation / release notes

If you publish results from this repository, release:

config files used,
split metadata CSV,
best checkpoints,
result_*.json files,
subgroup metrics and model card.

Name		Name	Last commit message	Last commit date
apps		apps
configs		configs
data		data
scripts		scripts
src/fairfactor		src/fairfactor
.gitignore		.gitignore
README.md		README.md
pyproject.toml		pyproject.toml
requirements.txt		requirements.txt

Latest commit

History

Folders and files

Repository files navigation