ReMeDI: Refined Memory for Disambiguation of Identities with SAM3 in Surgical Segmentation

We propose ReMeDI-SAM3: Refined Memory for Disambiguation of Identities with SAM3, a training-free extension of SAM3 to enhance temporal consistency and identity preservation in surgical videos. The pipeline is shown above. Our approach (1) restructures SAM3 memory into two components: (i) a relevance-aware memory that admits only high-confidence frames to stabilize temporal propagation, and (ii) an occlusion–aware memory that selectively retains pre-occlusion appearance cues to facilitate identity recovery. Building upon this design, we further introduce (2) a novel memory expansion scheme based on piecewise interpolation of temporal positional encodings, enabling larger memory without retraining, and (3) a feature-based re-identification module with temporal voting for robust post-occlusion identity verification and correction. Together, these components improve robustness to occlusions, tracking stability, and reliable instrument re-entry handling in surgical videos.

Among these components, the memory expansion scheme is the least intuitive, so we illustrate it in more detail below. Long surgical procedures require long-horizon temporal context, yet SAM3 is constrained by a fixed set of temporal positional encodings. Our piecewise interpolation scheme preserves the boundary embeddings, which carry distinct temporal priors, and samples new positions only in the interior region, enabling substantially larger memory capacity without retraining. Uniform interpolation, in contrast, distorts the learned boundary temporal priors.

We analyze the contribution of each component of ReMeDI-SAM3, with all ablation studies conducted on EndoVis17. Relevance-aware memory filtering alone provides a 3.5% mcIoU gain over vanilla SAM3 by suppressing noisy memory updates, and the occlusion-aware memory partition adds a further 0.5% mcIoU. The feature-based re-identification module contributes 1.4% IoU by correcting post-occlusion identity drift, while expanding the memory adds another 0.8% IoU, for a total improvement of 6.0% IoU and 5.9% mcIoU — confirming that the components are jointly essential for robust long-horizon segmentation. Expanding the memory to 15 frames improves all metrics, whereas further expansion slightly degrades performance, suggesting that an overly large memory introduces less informative context. Finally, replacing the proposed piecewise interpolation with uniform interpolation causes a drop of about 1.2% Challenge IoU and 1% IoU, since uniform resampling distorts the learned boundary temporal priors that piecewise interpolation preserves.