When Trackers Fail: VLM-Guided Verification and Recovery for Robust Video Object Segmentation

The problem

Propagation feeds its own errors back into memory

Every recent tracker improves propagation: better memory, motion cues, distractor-aware memory. All of them still trust their own last prediction. One wrong mask in memory, and every frame after it inherits the mistake. Nothing in the loop notices, so nothing undoes it.

occlusionreappearancelook-alike distractorsscene cutspersistent identity drift

Ground truth, SAM 3 and Dual-Agent on a sequence where SAM 3 fails

Rows: ground truth, SAM 3, Dual-Agent. SAM 3 loses the violinist and never gets her back. Dual-Agent does.

The idea

Robust tracking needs more than better propagation: it needs failure detection and recovery

Dual-Agent adds a failure-aware layer above a frozen SAM 3. A verifier checks, from the tracker's own history, whether the track is still right. A corrector re-localizes the target when it is not. A committed correction re-prompts SAM 3, so it continues from a clean memory instead of a corrupted one.

Dual-Agent adds a verification and recovery layer above frozen SAM 3

Sparse triggers call the verifier. An incorrect verdict calls the corrector. A commit gate decides whether the fix reaches SAM 3's memory.

Principle 1

Asymmetric decomposition

Whether the track is wrong and where the target is are different problems. A large model verifies; a small specialist corrects.

Principle 2

Tracker-aware verification

The verifier sees trusted exemplars beside SAM 3's own, possibly wrong, recent boxes. It judges the real trajectory, not an oracle one.

Principle 3

State-safe intervention

A wrong correction rewrites every future frame. Sparse triggers and a commit gate make interventions rare and deliberate.

Principle 4

Failure-oriented supervision

The corrector learns from SAM 3's real failures, with identity-switch and distractor perturbations that expose it to exactly the ambiguities that break propagation.

How it works

One correction, from trigger to memory reset

Step 1

Trigger sparsely

The verifier is called only when SAM 3's own signals look wrong: during occlusion, after a reappearance, or when its confidence starts to oscillate. A confident SAM 3 is left alone.

Step 2

Verify against the tracker's own history

Qwen3-VL-8B-Thinking, zero-shot, sees two exemplars, four context frames with SAM 3's own boxes, and the query. It must find the target before judging the box, so the verdict is about identity. Three votes at t, t+4 and t+8 decide.

Step 3

Correct only on a detected failure

A Qwen3-VL-4B-Instruct corrector, fine-tuned on 79,457 of SAM 3's own failures, answers two things: is the target visible, and where. The verifier re-checks the proposal; a rejected box gets up to three tries.

Step 4

Commit conservatively

The fix overwrites SAM 3's memory only if it differs substantially from the current box (IoU below 0.5). SAM 3 is re-prompted and carries on by itself. One reset recovers a whole segment.

Results

The harder the identity problem, the bigger the gain

Failure detection and recovery pay most where propagation is weakest. On SecVOS, built around scene cuts, the gain grows from +1.0 on single-shot videos to +4.1 on videos with multiple scene changes. On LVOS-v2 it improves an already strong SAM 3 and posts the best unseen-object scores.

SecVOS-160, J&F by scene complexity

Method	No cut	One cut	Multi cut	Overall
Cutie-base	72.5	53.0	48.3	52.7
SAM 2.1	79.4	58.5	52.4	58.2
SAMURAI	81.8	60.6	59.3	62.2
SAM2.1Long	81.3	61.8	58.5	62.3
SeC	84.2	69.6	67.5	70.0
SAM 3	87.1	74.6	69.3	74.2
Dual-Agent	88.1+1.0	76.2+1.6	73.4+4.1	76.9+2.7

LVOS-v2 validation

Method	J&F	J_s	F_s	J_u	F_u
XMem	64.5	62.6	69.1	60.6	65.6
SAM 2.1	84.1	80.7	87.4	80.6	87.7
SAM2.1Long	85.9	81.7	88.6	83.0	90.5
SeC	86.5	—	—	—	—
SAM 3	88.2	85.4	92.9	83.3	91.5
Dual-Agent	89.4+1.2	84.9	92.7	85.8	94.1

s / u: seen / unseen object classes.

Where the SecVOS gain comes from

The gain does not come from rewriting the tracker often. Dual-Agent touches 4.8% of frames and lands where propagation had failed: 88 commits across 160 videos, 67 better than SAM 3, 19 worse, 2 unchanged.

Commits by SAM 3 state

State at commit	Commits	SAM 3 IoU	Dual-Agent IoU
Target missed	39	0.000	0.856
Drifted to the wrong object	32	0.035	0.903
Was already correct	17	0.961	0.026

IoU at the committed frame. The last row is what a wrong verdict costs.

Trigger ablation

Triggers	J&F
Occlusion trigger only	75.4
+ reappearance trigger	75.6
+ confidence-variance trigger (full)	76.9

Occlusion and reappearance cues alone recover little. The confidence-variance trigger catches drift before SAM 3 admits a loss and supplies most of the gain.

See it

Where SAM 3 fails and Dual-Agent recovers

Left: SAM 3. Right: Dual-Agent. Green outline: ground truth. Each clip runs through the moment the two diverge; everywhere else, Dual-Agent is SAM 3.

Scene cuts (SecVOS)

A cut wipes out what the memory knew. SAM 3 drops to zero; Dual-Agent re-identifies the target in the new shot.

Car, interior shots

J&F 64.2 → 95.9

The video cuts to the car's interior. SAM 3 paints seats and dashboard for 59 frames; Dual-Agent declares the target absent and clears the memory until the car returns.

Man in a kitchen, several shots

J&F 49.3 → 75.3

Across the cuts SAM 3 switches to the wrong person, then loses the man. Dual-Agent catches the switch and re-localizes him.

Cook, alternating close-ups

J&F 57.3 → 84.5

Close-ups of hands alternate with wide shots. Dual-Agent re-anchors after each reappearance; SAM 3 alone loses the hands.

Fish across four shots

J&F 18.4 → 69.1

Underwater, at the surface, on a trailer. SAM 3 loses the fish at the first cut; Dual-Agent follows it through every change.

Red car, changing camera

J&F 64.2 → 90.3

Wide, in-car, chase. SAM 3 drops to zero at each cut; Dual-Agent re-identifies the car every time.

Earbud

J&F 44.7 → 83.7

A small object across product shots. SAM 3 cannot bridge the cut; the corrector recovers it from the exemplars.

Car with a person in the way

J&F 64.9 → 90.8

After the cut a person partly covers the car. SAM 3 loses it; Dual-Agent recovers the car and keeps the person out of the mask.

Long videos (LVOS-v2)

Targets leave and return minutes later. Miss the reappearance once and SAM 3 rarely recovers. A black card marks a skipped absence.

Zebra among zebras

J&F 33.7 → 92.5

The target zebra leaves and returns among look-alikes. SAM 3 produces nothing; Dual-Agent re-acquires the right animal.

Highway car

J&F 48.4 → 83.0

The camera cuts away and back. SAM 3 never re-acquires the car; Dual-Agent does.

Young gorilla

J&F 63.8 → 87.4

After an occlusion SAM 3 latches onto the adult gorilla. Dual-Agent catches the switch and returns to the young one.

Koi in a crowded pond

J&F 78.8 → 93.6

The fish re-enters among a dozen similar fish. SAM 3 misses the reappearance; Dual-Agent recovers the right one.

Skateboard in a parking garage

J&F 50.1 → 87.6

Small, low-contrast, out of view for a minute. SAM 3 stays empty; Dual-Agent re-acquires it on the first verifier call that sees it.

Person on a segway

J&F 86.8 → 92.8

SAM 3 misses him for several seconds after he re-enters; Dual-Agent recovers him at once.

Billiard ball

J&F 81.6 → 88.2

A hand and cue cover the ball. SAM 3 recovers much later; Dual-Agent as soon as it is visible.

Spotted fish

J&F 72.8 → 80.8

The fish swims behind a rock. On reappearance SAM 3 stays empty; Dual-Agent re-acquires it.

Transfer

Dual-Agent transfers: other trackers, other tasks

Verification and recovery are not tied to SAM 3. Swap the propagator, retrain nothing: every tracker gains, the weaker ones most, and SeC, which already reasons semantically, still gains.

Other frozen propagators, SecVOS-160 J&F

Propagator	Overall		No scene change		Single change		Multiple changes
Propagator	alone	+ Dual-Agent	alone	+ Dual-Agent	alone	+ Dual-Agent	alone	+ Dual-Agent
Cutie	52.7	59.2+6.5	72.5	75.9+3.4	53.0	59.5+6.5	48.3	56.6+8.3
SAM 2.1	58.2	65.1+6.9	79.4	83.4+4.0	58.5	61.6+3.1	52.4	61.3+8.9
SeC	70.0	72.0+2.0	84.2	84.6+0.4	69.6	72.2+2.6	67.5	70.5+3.0
SAM 3	74.2	76.9+2.7	87.1	88.1+1.0	74.6	76.2+1.6	69.3	73.4+4.1

Long-term single-object tracking, box-initialized

LaSOT-ext	AUC	P	P_norm
SAM 3	63.7	74.8	72.9
Dual-Agent	64.3	75.5	73.5

TNL2K	AUC	P	P_norm
SAM 3	65.2	72.8	74.3
Dual-Agent	66.5	74.1	75.7

VOT-LT2022	Pr	Re	F
SAM 3	63.2	57.4	60.1
Dual-Agent	61.0	61.3	61.2

The largest gain is on TNL2K, where long occlusions most often need recovery. On VOT-LT, higher recall lifts the F-score.

Limits

When Dual-Agent makes it worse

19 of the 88 commits on SecVOS lowered the score. Three of them, SAM 3 left and Dual-Agent right:

Hallucinated recovery during an absence

J&F 82.9 → 46.7

The car leaves for the interior shots and SAM 3 correctly outputs nothing. At frame 57 the verifier votes INCORRECT and the corrector returns a box for a target that is not there; Dual-Agent paints the dashboard for 80 frames.

Overriding a correct track with the wrong object

J&F 85.0 → 65.8

The red car returns and SAM 3 is back on it. At frame 204 the verifier rejects the correct box; the corrector lands on the black car next to it, propagated for 36 frames.

Wrong absence at a reappearance

J&F 95.8 → 87.8

Both track the singer. At frame 168 the verifier flags the track and the corrector declares her absent; Dual-Agent outputs nothing for 13 frames.

Cost and limitations

SecVOS-160, one H100	s / video	J&F
SAM 3	12.2	74.2
Dual-Agent, bf16	71.8	76.9
Dual-Agent, FP8 verifier	56.2	76.8
FP8, occlusion poll every 24 frames	47.7	76.4

Not real-time. 71.8 s per SecVOS video on one H100 against 12.2 s for SAM 3; FP8 and sparser polling bring it to 47.7 s.

Verifier errors are expensive. A missed failure stays; a wrong commit propagates.

Cite

BibTeX

@inproceedings{bundele2026whentrackersfail,
  title     = {When Trackers Fail: VLM-Guided Verification and Recovery for Robust Video Object Segmentation},
  author    = {Bundele, Valay and Agrawal, Susmit and Hosseinzadeh, Mehran and Nguyen, The Nam and Lensch, Hendrik P. A.},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  year      = {2026}
}