Segment Anything Model: Unleashing its Power Across Diverse Domains – A Research Digest

Latest 50 papers on segment anything model: Nov. 2, 2025

The Segment Anything Model (SAM), and its successor SAM2, have rapidly become cornerstone technologies in computer vision, offering powerful zero-shot and few-shot segmentation capabilities. This remarkable ability to segment novel objects with minimal prompting has opened doors across numerous applications, from intricate medical imaging to broad remote sensing. However, adapting these generalist models to highly specialized, noisy, or resource-constrained environments remains an active area of research. This digest delves into recent breakthroughs that fine-tune, extend, and optimize SAM/SAM2, showcasing how researchers are pushing the boundaries of what these foundation models can achieve.

The Big Idea(s) & Core Innovations

The central challenge these papers tackle is harnessing SAM’s immense power while addressing its limitations in specific, often critical, contexts. A recurring theme is the need for domain-specific adaptation and efficiency. For instance, in medical imaging, researchers are striving for precision on minute anatomical structures and efficiency for real-time applications.

Medical Imaging: Several papers focus on enhancing SAM’s capabilities for healthcare. From the University of Science and Technology and Shanghai General Hospital, the paper SAMRI: Segment Anything Model for MRI introduces an MRI-specific SAM that achieves state-of-the-art accuracy on small, clinically important structures like cartilage, efficiently by only fine-tuning the mask decoder. Similarly, Maryland Dialameh et al. from the University of Waterloo in EMA-SAM: Exponential Moving-average for SAM-based PTMC Segmentation improve temporal stability in tumor segmentation during radio-frequency ablation, even with occlusions. This is achieved by maintaining a stable latent prototype of the lesion with a confidence-weighted exponential moving average pointer.

Addressing the critical need for label-free segmentation, G. Comas and Bjoern H Menze present an unsupervised approach in Towards Label-Free Brain Tumor Segmentation: Unsupervised Learning with Multimodal MRI, using Vision Transformer Autoencoders and SAM-based postprocessing for anomaly detection. In a similar vein of reducing annotation burden, Wenxiang Chen et al. from the University of Science and Technology, China introduce Uncertainty-Aware Extreme Point Tracing for Weakly Supervised Ultrasound Image Segmentation, leveraging extreme points to generate high-quality pseudo labels for ultrasound images.

More advanced SAM2 adaptations for 3D medical images are also emerging. From Beijing Jiaotong University, SAM2-3dMed: Empowering SAM2 for 3D Medical Image Segmentation by Yeqing Yang et al. tackles the domain gap between video-based models and 3D medical data through novel modules for spatial dependencies and boundary precision. The innovative ABS-Mamba from Anonymized Author in ABS-Mamba: SAM2-Driven Bidirectional Spiral Mamba Network for Medical Image Translation combines SAM2’s global semantic modeling with Mamba’s efficient contextual understanding for high-fidelity medical image translation. For efficient fine-tuning, Y. Zhang et al. from the University of Science and Technology of China propose SAM2LoRA: Composite Loss-Guided, Parameter-Efficient Finetuning of SAM2 for Retinal Fundus Segmentation, significantly boosting retinal fundus segmentation accuracy with minimal parameter updates. Zelin Liu et al. from Shanghai Jiao Tong University further this with BALR-SAM: Boundary-Aware Low-Rank Adaptation of SAM for Resource-Efficient Medical Image Segmentation, introducing a Complementary Detail Enhancement Network and low-rank tensor attention for fine-grained boundaries with reduced memory usage. Yu Li et al. from The George Washington University present KG-SAM: Injecting Anatomical Knowledge into Segment Anything Models via Conditional Random Fields, a groundbreaking framework that unifies anatomical priors and uncertainty quantification for superior medical image segmentation.

Beyond Healthcare: Diverse Applications: SAM’s adaptability extends far beyond medicine. For environmental monitoring, M. Saifuzzaman Rafat et al. in From Pixels to People: Satellite-Based Mapping and Quantification of Riverbank Erosion and Lost Villages in Bangladesh leverage fine-tuned SAM to map riverbank erosion with high accuracy. In hazardous environments, Promptable Fire Segmentation: Unleashing SAM2 s Potential for Real-Time Mobile Deployment with Strategic Bounding Box Guidance by UEmmanuel5 optimizes SAM2 for real-time fire segmentation on mobile devices using bounding box prompts. From Shanghai Jiao Tong University, Expose Camouflage in the Water: Underwater Camouflaged Instance Segmentation and Dataset introduces UCIS-SAM to expose camouflaged objects in challenging underwater environments.

For improved real-world robot perception, Yijun Hu et al. from the University of Chinese Academy of Sciences introduce Robust Ego-Exo Correspondence with Long-Term Memory, enhancing SAM2 with dual-memory architecture for better object correspondence in long videos. Shuai Chen et al. from the University of Electronic Science and Technology of China propose CMaP-SAM: Contraction Mapping Prior for SAM-driven Few-shot Segmentation, optimizing position priors using contraction mapping theory for state-of-the-art few-shot segmentation. In a fascinating twist, Attack for Defense: Adversarial Agents for Point Prompt Optimization Empowering Segment Anything Model by Xiao Li et al. demonstrates how adversarial agents can be repurposed to improve SAM’s performance through prompt optimization, enhancing robustness and accuracy.

Addressing the multi-modal future, milotic233 presents HyPSAM: Hybrid Prompt-driven Segment Anything Model for RGB-Thermal Salient Object Detection, leveraging dynamic convolution and prompt engineering for robust salient object detection. Dian Jin et al. introduce SimToken: A Simple Baseline for Referring Audio-Visual Segmentation, combining MLLMs with SAM for instruction-guided video segmentation. Further integrating multi-modal data, Iacopo Curti et al. from the University of Bologna introduce Multimodal SAM-adapter for Semantic Segmentation, using an adapter network to inject fused multimodal features into SAM’s RGB stream for robust scene understanding. The paper MirrorSAM2: Segment Mirror in Videos with Depth Perception by Mingchen Xu et al. from Cardiff University tackles the complex problem of video mirror segmentation by integrating depth perception with SAM2.

From a foundational perspective, Chaitanya Ryali et al. from Meta AI investigate How Universal Are SAM2 Features?, highlighting the necessity of domain adaptation and fine-tuning. Yingzhen Hu et al. from Mohamed bin Zayed University of AI introduce SAM-DCE: Addressing Token Uniformity and Semantic Over-Smoothing in Medical Segmentation, a prompt-free framework for medical segmentation that enhances class separability. Suzhe Xu et al. from Huaqiao University offer BiPrompt-SAM: Enhancing Image Segmentation via Explicit Selection between Point and Text Prompts, showcasing that explicit prompt selection can outperform complex feature fusion. Finally, in remote sensing, PolSAM: Polarimetric Scattering Mechanism Informed Segment Anything Model by Yuqing Wang et al. integrates physical scattering characteristics for enhanced PolSAR data segmentation.

Under the Hood: Models, Datasets, & Benchmarks

These advancements are driven by clever architectural modifications, specialized training strategies, and the introduction of new data resources:

• SAMRI: Fine-tunes only SAM’s mask decoder on precomputed embeddings, utilizing a novel loss combining focal and Dice loss for small object segmentation. Code: https://github.com/wangzhaomxy/SAMRI
• EMA-SAM: Extends SAM-2 with a confidence-weighted exponential moving average pointer for temporal stability. Code: https://github.com/mdialameh/EMA-SAM
• Promptable Fire Segmentation: Leverages SAM2 with bounding box prompts for real-time mobile deployment. Code: https://github.com/UEmmanuel5/ProFSAM
• UCIS-SAM: Introduced alongside the UCIS4K dataset for underwater camouflaged instance segmentation. Code: https://github.com/wchchw/UCIS4K
• LM-EEC: Enhances SAM2 with a dual-memory bank system and Memory-View MoE module for robust ego-exo correspondence on the EgoExo4D benchmark. Code: https://github.com/juneyeeHu/LM-EEC
• SAM2LoRA: Applies Low-Rank Adaptation (LoRA) and composite loss functions for parameter-efficient fine-tuning of SAM2 for retinal fundus segmentation.
• SAM2-3dMed: Adapts SAM2 for 3D medical images with Slice Relative Position Prediction (SRPP) and Boundary Detection (BD) modules.
• SAMSOD: Optimizes SAM for RGB-T salient object detection, performing well on scribble-supervised and fully supervised datasets. Code: https://github.com/liuzywen/SAMSOD
• CMaP-SAM: Integrates contraction mapping theory for optimizing position priors in few-shot segmentation, achieving SOTA on PASCAL-5i and COCO-20i. Code: https://github.com/Chenfan0206/CMaP-SAM
• SOHES: A self-supervised approach generating over 100 high-quality pseudo-labels per image for open-world hierarchical entity segmentation.
• PlantNet Integration: Combines Pl@ntNet’s specialized plant representations with SAM for zero-shot agricultural segmentation.
• ABS-Mamba: U-shaped network with a hybrid encoder (SAM2-Hiera + CNN), spiral-scanned bidirectional Mamba blocks, and uncertainty-aware hierarchical skip-connections. Code: https://github.com/gatina-yone/ABS-Mamba
• PolSAM: Uses Microwave Vision Data (MVD) representation and FFP/SFP modules for PolSAR data segmentation on the PhySAR-Seg dataset. Code: https://github.com/XAI4SAR/PolSAM
• BALR-SAM: Introduces Complementary Detail Enhancement Network (CDEN), low-rank decomposition adapters, and a low-rank tensor attention mechanism for resource-efficient medical image segmentation.
• KG-SAM: Integrates a Conditional Random Field (CRF) module with a medical knowledge graph to enforce anatomical consistency with SAM features.
• SAMIR: A feature-driven medical image registration framework leveraging SAM’s structure-aware properties with a novel feature-level loss. (Code to be released post-acceptance)
• ReCOT: Reformulates cross-view object geo-localization as a recurrent problem, using SAM-based knowledge distillation and a Reference Feature Enhancement Module (RFEM). Code: https://github.com/zju-icst/ReCOT
• FS-SAM2: Repurposes SAM2’s video-based design for few-shot semantic segmentation via Low-Rank Adaptation (LoRA) on PASCAL-5i, COCO-20i, and FSS-1000 datasets. Code: https://github.com/fornib/FS-SAM2
• SAM-TTT: Addresses SAM’s semantic deficiency in camouflaged object detection using Reverse SAM Parameter Configuration and T-Visioner Module for Test-Time Training. Code: https://github.com/guobaoxiao/SAM-TTT
• EMeRALDS: Integrates SAM2 with text prompts, radiomic features, and synthetic electronic medical records for zero-shot lung nodule segmentation and classification.
• Organoid Tracker: A SAM2-powered platform for zero-shot cyst analysis in human kidney organoid videos, using an inverse temporal tracking strategy. Code: https://github.com/hrlblab/OrganoidTracker
• Multimodal SAM-adapter: Extends SAM for multimodal semantic segmentation using an adapter network for fusing RGB and auxiliary sensor data on DeLiVER, FMB, and MUSES benchmarks.
• pFedSAM: Personalizes federated learning of SAM for medical image segmentation using LoRA and L-MoE components for efficient cross-domain adaptation.
• ENSAM: An efficient SAM-based model for interactive 3D medical image segmentation using relative positional encoding and the Muon optimizer.
• TASAM: Enhances SAM with terrain-aware features for temporal-scale remote sensing segmentation.
• FloorSAM: Combines semantic and geometric information for SAM-guided floorplan reconstruction, supporting zero-shot learning. Code: https://github.com/Silentbarber/FloorSAM
• CLAPS: Leverages CLIP to unify auto-prompt segmentation in multi-modal retinal imaging.
• SAM*: Introduces a physics-aware reward function for task-adaptive hyperparameter tuning of SAM in microscopy imaging.

Impact & The Road Ahead

The research highlighted here paints a vibrant picture of SAM/SAM2’s transformative potential. By systematically addressing challenges related to computational efficiency, domain adaptation, and data scarcity, these papers demonstrate how foundation models can be fine-tuned and augmented to achieve remarkable results in specialized, high-stakes applications. The ability to perform zero-shot segmentation in areas like medical diagnostics (e.g., lung nodule detection, brain tumor segmentation) and environmental monitoring (riverbank erosion) is particularly impactful, drastically reducing the annotation burden and accelerating scientific discovery.

The trend towards parameter-efficient fine-tuning (PEFT), exemplified by methods like LoRA, signals a move towards more accessible and sustainable AI. This is crucial for deploying powerful models on edge devices and in resource-constrained settings. The integration of multi-modal data (RGB-T, depth, audio-visual) with SAM also underscores a growing understanding that real-world perception often benefits from diverse information streams.

The road ahead will likely involve further refinement of these adaptation strategies, with a greater emphasis on explainability and robustness in critical domains like healthcare. Continued innovation in unsupervised and weakly supervised learning will further democratize access to advanced segmentation capabilities. As researchers continue to unlock SAM’s full potential, we can expect to see an even wider array of applications that were once thought impossible, pushing the boundaries of AI-driven perception and understanding.