MARS: a Multimodal Alignment and Ranking System for Few-Shot Segmentation

Pipeline overview

The MARS framework enhances few-shot segmentation by integrating multimodal information into a unified pipeline. It begins with a module that retrieves the essential textual cues from the support images. Using ViP-LLaVA, the system extracts the class name and obtains a detailed description via WordNet. In scenarios with multiple support images, a majority voting mechanism ensures that the consolidated textual information is both robust and reliable.

Following this, the system aligns visual and textual modalities. A pre-trained CLIP model processes the query image along with two specialized text prompts—one emphasizing the presence of the object and one its absence—to generate an initial saliency map that highlights regions of interest. This map is refined by incorporating prior information, leading to a more precise localization of the target region.

To address situations where visual features may not fully capture semantic nuances, the AlphaCLIP module is employed. This module fuses the class name with its detailed textual description to create a robust global conceptual score. By comparing the normalized image and text embeddings, it provides a semantic measure that verifies whether the mask proposals are aligned with the expected class characteristics.

In parallel, the framework processes visual information exclusively. A Vision Transformer extracts detailed features from both the support and query images, producing metrics that capture overall visual similarity as well as fine-grained local correspondences. This results in the computation of both a global visual score and a local visual score, ensuring that the segmentation captures both holistic appearance and detailed structure.

Finally, all the computed scores are integrated in a filtering and merging stage. The Filtering-Merging Module combines the contributions of the local and global conceptual and visual scores into a single MARS score for each mask proposal. Through a dual-threshold strategy, low-confidence proposals are discarded and the remaining ones are merged to produce the final, refined segmentation mask.

Quantitative results

MARS, being a plug-and-play module, has been tested when applied to a variety of state-of-the-art few-shot segmentation methods, both under the one-shot and five-shot settings.

The results are shown in the tables below. The evaluation suite is composed of the $\text{COCO-20}^i$, $\text{Pascal-5}^i$, $\text{LVIS-92}^i$, and $\text{FSS-100}$ datasets. The results are reported in terms of the mean mean Intersection over Union (mIoU) metric.

Qualitative Results on the $\text{COCO-20}^i$ dataset

◀▶

1 / 3

BibTeX citation

    @misc{catalano2025marsmultimodalalignmentranking,
      title={MARS: a Multimodal Alignment and Ranking System for Few-Shot Segmentation}, 
      author={Nico Catalano and Stefano Samele and Paolo Pertino and Matteo Matteucci},
      year={2025},
      eprint={2504.07942},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2504.07942}, 
}