FoundYou: A Unified Model for Personalized Segmentation and Retrieval

ECCV 2026

Gabriele Trivigno · Marcos Alfaro1,2 · Claudia Cuttano1 · Gabriele Berton · Luis Payá2,3 · Carlo Masone1
1Politecnico di Torino 2Miguel Hernández University of Elche 3Valencian Graduate School of AI
Paper Code Poster Gradio Demo Colab

Give FoundYou an example of your object. It can segment the same physical instance in a new image or retrieve it from a large gallery with one compact model.

FoundYou overview

FoundYou segments a prompted handbag and retrieves the same handbag, with performance and speed comparisons.
One model, two personalized tasks. A visual prompt identifies an object; FoundYou then segments that exact instance in a target image or ranks gallery images by instance-level similarity.
Interactive Demo

Try FoundYou with Gradio

Explore the interactive FoundYou demo powered by Gradio. Upload reference images and see segmentation and retrieval results.

Launch Gradio Demo
At a glance

The same instance alignment drives both tasks

Unified

Segment and retrieve

A shared SAM 2 representation produces either a dense segmentation mask or an image-level retrieval score.

Promptable

Mask, box, or point

The frozen SAM 2 decoder preserves flexible prompting, reducing the annotation required to identify a reference object.

Few-shot

More views, better retrieval

Multiple references are stored in the memory bank and jointly improve instance matching without retraining.

Efficient

Built on SAM 2-small

Only 5.9 M parameters are trained, yielding a 52 M-parameter model that runs at 90.2 images/s on an RTX 4090.

Motivation

Personalized segmentation and retrieval are the same matching problem

Personalized segmentation localizes a particular physical object in a target image, while personalized retrieval ranks images that contain it. Although their outputs differ, both tasks require the same core ability: preserving object identity across unrelated images, backgrounds, viewpoints, and scales.

FoundYou starts from the instance-aware representations learned by Segment Anything 2 for video tracking. Lightweight feature adapters remove the model’s spatio-temporal continuity bias, while self-distillation preserves the spatial structure needed for accurate masks. The result is a unified framework that improves over PDM, the prior unified approach, as well as task-specific methods, while keeping SAM 2 entirely frozen.

Method

Repurposing SAM 2 memory across independent images

FoundYou stores a prompted reference object in the SAM 2 memory bank, aligns a target image to that memory, and decodes the shared representation for segmentation or retrieval.

FoundYou method with a frozen SAM 2 backbone, AdaptFormer layers, a memory bank, and segmentation and retrieval decoders.

Swipe horizontally to inspect the full diagram.

FoundYou overview. The prompted reference is encoded into the memory bank. Target features attend to that representation through frozen memory attention. The original SAM 2 segmentation decoder predicts a mask, while a lightweight retrieval decoder aggregates dense correspondences into a similarity score.
01

Adapt identity features

AdaptFormer layers in the final image-encoder stages strengthen instance discrimination across unrelated scenes.

02

Store the reference

A point, box, or mask identifies the object; its features are encoded and stored in the memory bank.

03

Decode either task

The same memory-conditioned target features produce a segmentation mask or a retrieval score.

Qualitative results

Retrieve first, segment the top result

FoundYou distinguishes the requested instance from visually similar hard negatives, then localizes that same instance using the shared representation.

References, top-two retrieved examples with scores, hard negatives, and top-one personalized segmentation results.

Swipe horizontally to inspect all examples.

For each reference image, the top two retrieved candidates receive high similarity scores, while visually related but non-matching objects are rejected. The final row segments the top-ranked image using the same reference prompt.
Few-shot personalized retrieval

Additional views provide complementary evidence

Each reference is stored in memory, allowing candidate images to attend jointly to multiple views of the same instance without retraining the model.

Examples comparing one-shot and four-shot personalized retrieval rankings.

Swipe horizontally to inspect all examples.

With one reference, a hard negative can occupy the first rank. Adding three views of the same instance moves the true positive to rank one across all shown examples.
1 reference 30.1 mAP@1k on ILIAS
4 references 34.2 mAP@1k on ILIAS
+4.1 mAP from additional views
Feature analysis · Object-Placement

Adaptation removes SAM 2’s continuity bias

Video pretraining encourages SAM 2 features to depend on a stable scene and limited object motion. FoundYou adapts those features for identity matching when position and context change independently.

Object-Placement analysis comparing frozen SAM 2 and FoundYou feature similarity under object translation and background changes.

Swipe horizontally to inspect all three analyses.

Same-instance feature similarity on Object-Placement. Frozen SAM 2 features degrade under object translation, background replacement, and their combination. FoundYou’s adapted features remain substantially more invariant across all three changes.
Main results

State of the art across both personalized tasks

One 52 M-parameter model improves over the prior unified solution while matching or surpassing specialized methods.

+18.4 mIoU over PDM on PerMIS
+17.8 mAP over PDM on ILIAS
90.2 images/s on an RTX 4090
5.9 M trainable parameters
Task Benchmark Metric FoundYou
Personalized segmentation PerSeg mIoU / bIoU 96.4 / 85.6
Personalized segmentation PerMIS mIoU / bIoU 62.6 / 57.4
Personalized retrieval PerMIR mAP 92.1
Personalized retrieval ILIAS mAP@1k 32.5

FoundYou is over 75× faster and 20× smaller than PDM, the only prior unified personalized segmentation and retrieval solution.

Flexible prompting

Strong masks from lightweight annotations

On PerSeg, performance changes only slightly from a full mask to a single point. On the harder PerMIS benchmark, FoundYou remains substantially stronger than prior personalized methods across prompt types.

Point 95.2 PerSeg 58.6 PerMIS
Box 95.9 PerSeg 61.5 PerMIS
Mask 96.4 PerSeg 62.6 PerMIS
Citation

BibTeX

@inproceedings{trivigno2026foundyou,
  title     = {{FoundYou}: A Unified Model for Personalized Segmentation and Retrieval},
  author    = {Gabriele Trivigno and Marcos Alfaro and Claudia Cuttano and Gabriele Berton and Luis Pay{\'a} and Carlo Masone},
  booktitle = {Proceedings of the European Conference on Computer Vision (ECCV)},
  year      = {2026}
}

Acknowledgments

FoundYou builds on the open-source Segment Anything 2 project and evaluates large-scale personalized retrieval using ILIAS. We thank the authors and contributors of these projects.