Phoenix

Learning from Adversity: Semantic-Aware Mask Refinement through Adversarial Perturbation
ECCV 2026

Beomyoung Kim^1,2 · Sung Ju Hwang^1,3
1KAIST   ²NAVER Cloud, Image Vision   ³DeepAuto.ai

Project Page · Paper · Code · Demo ·

Keep your segmenter. Add Phoenix. Phoenix is a model-agnostic refinement layer that turns coarse or noisy segmentation masks into precise object masks. It can sit behind a deployed segmenter, clean annotations and pseudo-labels, or act as an on-demand refinement tool in a visual-agent pipeline.

Phoenix learns from realistic segmentation failures through Adversarial Mask Perturbation (AMP) and learns the relationship between noisy, target, and refined masks through Contrastive Mask Refinement Learning (CMRL). At inference, the image encoder runs once and only the lightweight decoder is repeated across masks and refinement steps.

Phoenix at a glance

Use Phoenix as...	Input	Output
A deployment add-on	Existing segmenter's image + mask	Cleaner mask, without retraining the base model
A data-quality step	Human annotation or pseudo-label	Refined label for training or evaluation
A visual-agent tool	Mask flagged by a quality gate	Task-specific refinement on demand

How Phoenix learns refinement

• Adversarial Mask Perturbation (AMP) injects learnable perturbation embeddings into a frozen decoder and creates controllable, semantic-aware errors that resemble real segmentation failures.
• Contrastive Mask Refinement Learning (CMRL) jointly structures target, noisy, and refined mask features, preserving same-region consistency while separating foreground and background.

Installation

git clone https://github.com/naver-ai/Phoenix.git && cd Phoenix
pip install -r requirements.txt

Checkpoints

Each benchmark family is trained separately, so it has its own encoder and checkpoint — there is no single model that covers every table. Download from the Phoenix model repository into ckpt/:

File	Encoder	Used for
phoenix_vit_h_instseg.pt	ViT-H	Instance segmentation (Tables 1 & 2) and Cityscapes (Table S3)
phoenix_efficientvit_xl1.pt	EfficientViT-XL1	Lightweight instance segmentation and the demo
phoenix_vit_h_dis.pt	ViT-H	DIS fine-grained segmentation (Table 3)
phoenix_efficientvit_xl1_dis.pt	EfficientViT-XL1	Lightweight DIS model used by the demo
phoenix_vit_h_voc.pt	ViT-H	VOC semantic segmentation (Table S2)

The instance-segmentation tables are reported with both the ViT-H and the lightweight EfficientViT-XL1 backbones; pass the matching --encoder.

Download all checkpoints with the Hugging Face CLI:

hf download naver-iv/phoenix-weights --include "*.pt" --local-dir ckpt

import phoenix
model = phoenix.build_phoenix("ckpt/phoenix_efficientvit_xl1.pt",
                              encoder="efficientvit_xl1", device="cuda")

Quick start

import numpy as np, phoenix
from PIL import Image

model   = phoenix.build_phoenix("ckpt/phoenix_efficientvit_xl1.pt")
refiner = phoenix.PhoenixRefiner(model)

image = np.array(Image.open("image.jpg").convert("RGB"))
noisy = np.array(Image.open("noisy_mask.png").convert("L"))
refined = refiner.refine(image, noisy, refine_iters=5)   # -> bool HxW mask

Command line:

python infer.py --checkpoint ckpt/phoenix_efficientvit_xl1.pt \
    --image assets/examples/instance/00_image.jpg \
    --mask  assets/examples/instance/00_noisy.png \
    --output refined.png

Generate adversarial noise from a clean mask (AMP):

ng = phoenix.NoiseGenerator(model)
ng.set_image(image)
noisy = ng.generate(clean_mask, iou_thresh=0.7, guidance="dilate")  # expansion / erode / inversion

Gradio demo

PHOENIX_CKPT=ckpt/phoenix_efficientvit_xl1.pt python app.py

Three tabs (this is the same script used for the HuggingFace Space):

1. Instance Refinement — refine a noisy instance mask (EfficientViT-XL1).
2. Fine-grained Refinement — refine a coarse fine-segmentation mask on the DIS task (EfficientViT-XL1, phoenix_efficientvit_xl1_dis.pt).
3. Noise Generator (AMP) — from a clean target mask, set the IoU threshold τ and visualise all three guidance directions (expansion / contraction / inversion) at once, each as overlay + binary mask.

Each tab ships an example gallery (assets/examples/{instance,dis,noise}/) with a # index column so samples are easy to refer to. The fine-seg tab uses the EfficientViT-XL1 DIS checkpoint; override with PHOENIX_DIS_CKPT / PHOENIX_DIS_ENCODER.

Required data construction

Download the coarse predictions and manifests (~8.4 GiB) from naver-iv/phoenix-eval-data:

hf download naver-iv/phoenix-eval-data \
    --repo-type dataset --local-dir data

Original images and GT are not redistributed. Download them from COCO / LVIS, Cityscapes, DIS5K, and PASCAL VOC 2012, then copy or symlink them into the downloaded tree:

data/
├── coco/
│   ├── {train2017,val2017}/
│   ├── {lvis_v1_train_5K_cocofied,lvis_v1_val_cocofied}.json  # local GT
│   ├── coarse_gt_coco_val2017.json                              # local GT-derived input
│   └── *.json                                                   # HF predictions
├── cityscapes/
│   ├── leftImg8bit/val/{frankfurt,lindau,munster}/...
│   ├── cityscapes_fine_instance_seg_val_coco_format.json       # local GT
│   └── cityscapes_mask2former_r50.json                          # HF prediction
├── dis/
│   ├── _shared/{VD,TE1,TE2,TE3,TE4}/{im,gt}/...
│   └── DIS-<split>-<model>/{input,val_matte_list.txt}
└── voc/
    ├── _shared/{JPEGImages,SegmentationClassAug}/...
    └── VOC2012-<model>/{input_<model>,val_matte_list.txt}

The four GT/GT-derived JSONs marked local must retain these exact filenames. For DIS/VOC, each val_matte_list.txt already points to _shared relatively:

../_shared/VD/im/sample.jpg ../_shared/VD/gt/sample.png input/sample.png
../_shared/JPEGImages/sample.jpg ../_shared/SegmentationClassAug/sample.png input_clipes/sample.png

Evaluation

scripts/eval.sh runs every table with the correct per-benchmark encoder + checkpoint (see above). Edit the roots / CKPT_DIR inside it, then:

bash scripts/eval.sh             # single GPU
NGPU=8 bash scripts/eval.sh      # 8-GPU sharded evaluation

A single setting, single GPU:

python eval.py --checkpoint ckpt/phoenix_vit_h_instseg.pt --encoder vit_h \
    --data-root data/coco --refine-iters 5 \
    --valset lvis_wssis_1p_5k,lvis_mrcnn50_val5k

Multi-GPU — just launch with torchrun; the images are sharded across ranks and the predictions/metrics are merged on rank 0:

torchrun --nproc_per_node=8 eval.py --checkpoint ckpt/phoenix_efficientvit_xl1.pt \
    --encoder efficientvit_xl1 --data-root data/coco --refine-iters 5 \
    --valset lvis_wssis_1p_5k

Use --refine-iters 1 for DIS and Cityscapes, 5 otherwise. The evaluator reports the "Refine" configuration (point + box + mask prompts) from the paper.

Every benchmark reports both the noisy (un-refined input) and the refined scores, so the gain from refinement is visible directly.

Efficiency. Phoenix runs the heavy encoder once per image and then refines every mask of that image by running only the lightweight decoder (set_image → many refine_current), sharded across GPUs with prefetching data workers (--workers, default 8). Profiling the ViT-H pipeline (8× V100) shows the time split is ≈ 68% encoder / 16% prompt-sampling / 16% decoder — i.e. the encode-once design already targets the dominant cost, and the encoder is the thing to speed up:

• --amp runs the forward passes in fp16 (≈1.4× faster on V100, since the ViT-H encoder benefits most from fp16 tensor cores). Off by default as it can shift metrics by ~0.1; enable it when exact reproduction isn't required.
• The EfficientViT-XL1 backbone is ≈2.6× faster end-to-end than ViT-H at a small accuracy cost — prefer it for large sweeps.
• Batched decoding: all masks of an image are refined in one batched decoder pass (PhoenixRefiner.refine_batch, chunked by max_batch), amortising Python/kernel-launch overhead — ≈1.6× on many-instance images, bit-equivalent to per-mask refinement.
• Multi-GPU load balancing: images are assigned to ranks by a cost-aware (LPT) split (# instances for LVIS/COCO, image area for DIS/VOC) so no single rank straggles; results are identical to plain striping.

Repository structure

phoenix/
├── build.py              # build_phoenix(): construct + load checkpoint
├── refiner.py            # PhoenixRefiner: high-level refine(image, noisy_mask)
├── predictor.py          # low-level image-conditioned predictor
├── noise_generator.py    # NoiseGenerator: Adversarial Mask Perturbation (AMP)
├── modeling/             # encoder (EfficientViT / ViT) + SAM decoder
├── data/                 # evaluation datasets (LVIS/COCO/Cityscapes/DIS/VOC)
└── eval/                 # evaluators + metrics
app.py    infer.py    eval.py    scripts/eval.sh

Experimental results

Show all benchmark tables

Phoenix-ViT-H

benchmark	Noisy AP	Noisy AP50	Noisy Boundary-AP	Refine AP	Refine AP50	Refine Boundary-AP
lvis_wssis_1p_5k	0.1264	0.2619	0.0628	0.2874	0.3894	0.2362
lvis_wssis_5p_5k	0.2565	0.4227	0.1637	0.3627	0.4891	0.3025
lvis_wssis_10p_5k	0.3016	0.4773	0.2039	0.3891	0.5240	0.3258
lvis_nb_1p_5k	0.0506	0.1065	0.0192	0.0984	0.1204	0.0813
lvis_nb_5p_5k	0.1845	0.3111	0.0967	0.2671	0.3375	0.2221
lvis_nb_10p_5k	0.2264	0.3668	0.1266	0.3085	0.3951	0.2571
lvis_mrcnn50_val5k	0.3980	0.6137	0.2734	0.4696	0.6305	0.3883
lvis_mrcnn101_val5k	0.4158	0.6332	0.2900	0.4812	0.6461	0.3977
lvis_solo_val5k	0.3743	0.5857	0.2465	0.4627	0.6144	0.3767
lvis_condinst_val5k	0.3977	0.6037	0.2917	0.4671	0.6284	0.3863
lvis_mask2former_val5k	0.4677	0.6780	0.3701	0.5066	0.6837	0.4214
lvis_pointrend_val5k	0.4147	0.6188	0.3065	0.4706	0.6321	0.3889
lvis_queryinst_val5k	0.4237	0.6334	0.2991	0.4770	0.6473	0.3935
lvis_refinemask_val5k	0.4117	0.6077	0.3049	0.4604	0.6181	0.3791
lvis_transfiner_val5k	0.4315	0.6359	0.3232	0.4769	0.6435	0.3913
lvis_vitdet_h_coco	0.5464	0.7555	0.4247	0.5497	0.7456	0.4545
lvis_maskdino_swinl_coco	0.5684	0.7819	0.4649	0.5597	0.7622	0.4651
lvis_coco_val	0.3834	0.6079	0.2734	0.4373	0.6115	0.3612

Phoenix-EfficientViT-XL1

benchmark	Noisy AP	Noisy AP50	Noisy Boundary-AP	Refine AP	Refine AP50	Refine Boundary-AP
lvis_wssis_1p_5k	0.1264	0.2619	0.0628	0.2855	0.3934	0.2331
lvis_wssis_5p_5k	0.2565	0.4227	0.1637	0.3572	0.4890	0.2966
lvis_wssis_10p_5k	0.3016	0.4773	0.2039	0.3804	0.5231	0.3179
lvis_nb_1p_5k	0.0506	0.1065	0.0192	0.0975	0.1206	0.0794
lvis_nb_5p_5k	0.1845	0.3111	0.0967	0.2593	0.3336	0.2143
lvis_nb_10p_5k	0.2264	0.3668	0.1266	0.3012	0.3892	0.2499
lvis_mrcnn50_val5k	0.3980	0.6137	0.2734	0.4608	0.6313	0.3819
lvis_mrcnn101_val5k	0.4158	0.6332	0.2900	0.4714	0.6434	0.3915
lvis_solo_val5k	0.3743	0.5857	0.2465	0.4548	0.6112	0.3722
lvis_condinst_val5k	0.3977	0.6037	0.2917	0.4600	0.6277	0.3821
lvis_mask2former_val5k	0.4677	0.6780	0.3701	0.4992	0.6852	0.4177
lvis_pointrend_val5k	0.4147	0.6188	0.3065	0.4620	0.6290	0.3829
lvis_queryinst_val5k	0.4237	0.6334	0.2991	0.4688	0.6451	0.3880
lvis_refinemask_val5k	0.4117	0.6077	0.3049	0.4536	0.6165	0.3754
lvis_transfiner_val5k	0.4315	0.6359	0.3232	0.4709	0.6431	0.3882
lvis_vitdet_h_coco	0.5464	0.7555	0.4247	0.5422	0.7441	0.4496
lvis_maskdino_swinl_coco	0.5684	0.7819	0.4649	0.5521	0.7629	0.4595
lvis_coco_val	0.3834	0.6079	0.2734	0.4284	0.6087	0.3557

Phoenix ViT-H (DIS)

benchmark	Noisy IoU	Noisy Boundary-F	Refine IoU	Refine Boundary-F
DIS-VD-UNet	0.5477	0.6698	0.7480	0.8409
DIS-TE1-UNet	0.4411	0.5919	0.6962	0.8329
DIS-TE2-UNet	0.5439	0.6612	0.7933	0.8584
DIS-TE3-UNet	0.5929	0.7195	0.7944	0.8575
DIS-TE4-UNet	0.6114	0.7779	0.7524	0.8288
DIS-VD-ISNet	0.6710	0.7984	0.7664	0.8609
DIS-TE1-ISNet	0.5617	0.7493	0.7143	0.8531
DIS-TE2-ISNet	0.6705	0.7835	0.7926	0.8667
DIS-TE3-ISNet	0.6994	0.8122	0.8018	0.8703
DIS-TE4-ISNet	0.7012	0.8377	0.7533	0.8319
DIS-VD-HRNet	0.6102	0.7418	0.7644	0.8572
DIS-TE1-HRNet	0.5098	0.6705	0.6961	0.8340
DIS-TE2-HRNet	0.6122	0.7361	0.7814	0.8635
DIS-TE3-HRNet	0.6443	0.7786	0.8070	0.8657
DIS-TE4-HRNet	0.6467	0.8196	0.7493	0.8337
DIS-VD-PSPNet	0.5641	0.6926	0.7564	0.8474
DIS-TE1-PSPNet	0.4757	0.6684	0.7025	0.8397
DIS-TE2-PSPNet	0.5771	0.7045	0.7951	0.8601
DIS-TE3-PSPNet	0.5990	0.7184	0.7982	0.8624
DIS-TE4-PSPNet	0.5761	0.7039	0.7443	0.8282
DIS-VD-PFNet	0.5620	0.7010	0.7558	0.8530
DIS-TE1-PFNet	0.4697	0.6511	0.6803	0.8309
DIS-TE2-PFNet	0.5704	0.7031	0.7825	0.8613
DIS-TE3-PFNet	0.5946	0.7346	0.7977	0.8632
DIS-TE4-PFNet	0.5911	0.7478	0.7418	0.8325
DIS-VD-ICNet	0.5647	0.6907	0.7636	0.8525
DIS-TE1-ICNet	0.4622	0.6270	0.7005	0.8369
DIS-TE2-ICNet	0.5650	0.6828	0.7834	0.8552
DIS-TE3-ICNet	0.5983	0.7263	0.7938	0.8648
DIS-TE4-ICNet	0.6053	0.7603	0.7534	0.8294

Phoenix - VOC 2012 Semantic Seg

benchmark	Noisy mIoU	Noisy mBA	Refine mIoU	Refine mBA
VOC2012-CLIPES	0.7084	0.5168	0.8056	0.5481
VOC2012-BECO	0.6633	0.5323	0.7498	0.5450
VOC2012-MaskCLIP	0.5731	0.5320	0.5831	0.5301

Phoenix - Cityscapes Instance Seg

benchmark	Noisy AP	Noisy AP50	Noisy Boundary-AP	Refine AP	Refine AP50	Refine Boundary-AP
cityscapes_mask2former_r50	0.3683	0.6024	0.3488	0.3681	0.6034	0.3465

Citation

@inproceedings{kim2026phoenix,
  title     = {Learning from Adversity: Semantic-Aware Mask Refinement
               through Adversarial Perturbation},
  author    = {Kim, Beomyoung and Hwang, Sung Ju},
  booktitle = {European Conference on Computer Vision (ECCV)},
  year      = {2026},
}

Acknowledgements

Built on Segment Anything, EfficientViT and prompt-sampling from SAMRefiner.

License

Phoenix
Copyright (c) 2026-present NAVER Cloud Corp.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.