Segment Anything Model: From OR Automation to Zero-Shot Plant Segmentation—The Latest Breakthroughs

Latest 50 papers on segment anything model: Nov. 10, 2025

The Segment Anything Model (SAM), and its subsequent versions like SAM2, have fundamentally reshaped the landscape of computer vision, transitioning image segmentation from a specialized, heavily annotated task into a promptable, generalized skill. The current wave of research isn’t just about using SAM; it’s about hyper-specializing, adapting, and efficiently fine-tuning these colossal foundation models to solve complex, domain-specific problems that demand high precision, minimal data, or real-time performance. This digest explores the latest advancements, revealing how researchers are unlocking SAM’s potential across diverse fields, from surgery to space, often relying on clever prompting and parameter-efficient techniques.

The Big Idea(s) & Core Innovations

The central theme across recent research is the strategic adaptation of SAM for robustness and efficiency under constraints—be it limited labels, complex 3D structures, or noisy, multi-modal data.

1. Zero-Shot Generalization and Domain-Specific Prompts

Several papers demonstrate remarkable success in achieving zero-shot or few-shot segmentation by integrating SAM with domain-specific knowledge or leveraging optimized prompting strategies. The work from the University of Angers and Inria in their paper, Unlocking Zero-Shot Plant Segmentation with Pl@ntNet Intelligence, successfully leverages Pl@ntNet’s specialized plant representations to guide SAM, achieving IoU improvements of 60–70% in agricultural scenarios without explicit training. Similarly, the University of Göttingen’s zero-shot approach in Zero-Shot Multi-Animal Tracking in the Wild combines SAM 2 with Grounding Dino and adaptive detection thresholds to robustly track diverse animal species without retraining. For multi-modal tasks, Nanjing University of Science and Technology introduced HyPSAM: Hybrid Prompt-driven Segment Anything Model for RGB-Thermal Salient Object Detection, which uses dynamic convolution and hybrid prompts to fuse RGB and thermal data, boosting salient object detection accuracy.

2. Parameter Efficiency and Specialized Adaptation

To make SAM usable in resource-constrained environments (like mobile devices or clinical workstations), researchers are focusing on minimal parameter updates. University of Waterloo’s EMA-SAM: Exponential Moving-average for SAM-based PTMC Segmentation uses an exponential moving average pointer mechanism to stabilize real-time tumor tracking during radio-frequency ablation with minimal computational overhead. Even more resource-efficient adaptations, like BALR-SAM: Boundary-Aware Low-Rank Adaptation of SAM for Resource-Efficient Medical Image Segmentation, introduced by Shanghai Jiao Tong University, reduce SAM’s parameters by 94% using low-rank decomposition adapters while enhancing boundary delineation using a Complementary Detail Enhancement Network (CDEN). A similar spirit drives Subsampled Randomized Fourier GaLore for Adapting Foundation Models in Depth-Driven Liver Landmark Segmentation, which proposes SRFT-GaLore to replace computationally heavy SVD with a randomized Fourier transform for efficient surgical fine-tuning.

3. Bridging Modality Gaps and Contextual Integration

A significant body of work is dedicated to integrating SAM with other modalities or models for complex tasks:

• 3D Medical Segmentation: SAM2-3dMed: Empowering SAM2 for 3D Medical Image Segmentation (Beijing Jiaotong University) systematically adapts SAM2 for volumetric data by introducing modules (SRPP and BD) to model crucial spatial dependencies.
• Vision-Language Integration: SpinalSAM-R1: A Vision-Language Multimodal Interactive System for Spine CT Segmentation from Nanjing University of Aeronautics and Astronautics enables natural language-guided refinement by integrating SAM with DeepSeek-R1, achieving high parsing accuracy for clinical commands. The HFUT and MBZUAI collaboration in SimToken: A Simple Baseline for Referring Audio-Visual Segmentation uses Multimodal LLMs (MLLM) to generate semantic tokens, guiding SAM for accurate audio-visual segmentation.

Under the Hood: Models, Datasets, & Benchmarks

The advancements are heavily dependent on customizing and leveraging powerful models and introducing new high-quality datasets to challenge the state-of-the-art.

• Model Architectures: The core innovation often lies in the modular additions to SAM/SAM2. Examples include the Memory-View MoE module and dual-memory bank system in LM-EEC (Robust Ego-Exo Correspondence with Long-Term Memory) for cross-view tracking, and the Semantic Visual Projector (SVP) in Zhejiang University’s work, Re-purposing SAM into Efficient Visual Projectors for MLLM-Based Referring Image Segmentation, which dramatically cuts visual token redundancy (by ~93%).
• Prompt Optimization: Attack for Defense: Adversarial Agents for Point Prompt Optimization Empowering Segment Anything Model creatively uses adversarial techniques to optimize point prompts, while VRP-SAM: SAM with Visual Reference Prompt uses annotated reference images as prompts to boost generalization.
• Key Datasets & Resources: New resources are crucial for domain growth:
  • LLSD (Liver Landmark Segmentation Dataset): Introduced in Subsampled Randomized Fourier GaLore… for robust cross-dataset generalization in surgical settings.
  • UCIS4K Dataset: Introduced in Expose Camouflage in the Water… to benchmark underwater camouflaged instance segmentation.
  • Annotated Erosion Dataset: Created for From Pixels to People: Satellite-Based Mapping and Quantification of Riverbank Erosion and Lost Villages in Bangladesh, enabling precise quantification of land loss using SAM.
  • Public Code: Many projects encourage reproducibility; readers can explore the real-time segmentation toolkit SAM-EM at github.com/JamaliLab/SAM-EM and the few-shot segmentation framework CMaP-SAM at https://github.com/Chenfan0206/CMaP-SAM.

Impact & The Road Ahead

These collective advances are driving SAM beyond mere object segmentation into integrated, intelligent systems across critical domains. In healthcare, frameworks like SAMRI: Segment Anything Model for MRI (focused on fine-tuning the mask decoder) and the privacy-preserving pFedSAM: Personalized Federated Learning of Segment Anything Model for Medical Image Segmentation are making high-accuracy segmentation efficient and scalable, even for small, clinically relevant structures. For complex structural analysis, KG-SAM: Injecting Anatomical Knowledge into Segment Anything Models via Conditional Random Fields leverages Conditional Random Fields (CRF) and knowledge graphs to enforce anatomical consistency, leading to significant Dice score improvements in prostate segmentation.

In the broader industrial and environmental space, AD-SAM: Fine-Tuning the Segment Anything Vision Foundation Model for Autonomous Driving Perception shows how SAM can be adapted for robustness against domain shifts in self-driving, while remote sensing applications like TASAM: Terrain-and-Aware Segment Anything Model for Temporal-Scale Remote Sensing Segmentation enhance large-scale environmental monitoring.

The next frontier is clearly about seamless multimodal fusion (vision-language and vision-depth), increasing temporal consistency for video analysis, and perfecting parameter-efficient methods that allow foundation models to be deployed ubiquitously. The challenge of feature universality, highlighted in How Universal Are SAM2 Features?, confirms that while SAM is powerful, task-specific adaptation is indispensable. We are entering an exciting era where the Segment Anything Model is not just a tool, but a highly customizable architectural backbone for domain-aware AI assistants.