Segment Anything Model: The Latest Leaps in Generalization, Efficiency, and Specialized Applications

Latest 50 papers on segment anything model: Dec. 27, 2025

The Segment Anything Model (SAM) has captivated the AI/ML community, promising a new era of generalizable image segmentation. Born from Meta AI, SAM and its successors (SAM2, SAM3) have demonstrated remarkable zero-shot capabilities, transforming how we approach pixel-level understanding. However, the path from a foundational model to real-world, efficient, and specialized applications is paved with unique challenges. This digest dives into recent breakthroughs, exploring how researchers are pushing SAM’s boundaries, from enhancing its efficiency to tailoring it for complex domains like medical imaging, remote sensing, and even safeguarding against AI-generated forgeries.

The Big Idea(s) & Core Innovations

The core challenge addressed across these papers is enhancing SAM’s practicality. While powerful, vanilla SAM variants can be computationally intensive and may lack domain-specific nuance. Researchers are tackling this from multiple angles:

• Bridging the SAM2-SAM3 Divide: A pivotal insight comes from Ranjan Sapkota et al. from Cornell University and University of the Peloponnese in their paper, “The SAM2-to-SAM3 Gap in the Segment Anything Model Family: Why Prompt-Based Expertise Fails in Concept-Driven Image Segmentation”. They highlight SAM3’s fundamental shift from prompt-based to concept-driven segmentation, integrating vision-language fusion for open-vocabulary reasoning. This means SAM3 can segment all instances of a concept (e.g., “all cars”) from a text prompt, a significant leap from SAM2’s prompt-specific segmentation.

• Efficiency for Real-world Deployment: Making SAM practical for edge devices and real-time applications is crucial. Avilasha Mandala et al. from the University of Electronic Science and Technology of China introduce “Fast SAM2 with Text-Driven Token Pruning”, significantly reducing GPU memory and inference latency for video object segmentation through text-guided token pruning. Similarly, Nicola Farronato et al. from IBM Research Zurich and ETH Zurich propose “Q-SAM2: Accurate Quantization for Segment Anything Model 2”, achieving high accuracy at ultra-low bit-widths, making SAM2 8x smaller. For SAM3, Chengxi Simon Zeng et al. from the University of Bristol present “EfficientSAM3: Progressive Hierarchical Distillation for Video Concept Segmentation from SAM1, 2, and 3”, a family of distilled models enabling on-device video concept segmentation with flexible accuracy-latency trade-offs.

• Domain Adaptation & Specialization: Many papers focus on adapting SAM to specific, challenging domains:

  • Medical Imaging: From Z. Gong and X. Chen at the University of Nottingham with “SSL-MedSAM2: A Semi-supervised Medical Image Segmentation Framework Powered by Few-shot Learning of SAM2” for limited annotated data, to Anglin Liu et al. with “MedSAM3: Delving into Segment Anything with Medical Concepts” enabling text-promptable anatomical segmentation, SAM is being tailored for clinical workflows. Zhiguo Lu et al. from Zhejiang University demonstrate “3DTeethSAM: Taming SAM2 for 3D Teeth Segmentation” achieving state-of-the-art 3D results. Meanwhile, Cecilia Diana-Albelda et al. from Universidad Autónoma de Madrid present “GBT-SAM: A Parameter-Efficient Depth-Aware Model for Generalizable Brain tumour Segmentation on mp-MRI”, showcasing high accuracy with minimal parameters. The concept of continual learning in medical imaging is also addressed by Jiayi Wang et al. from Xi’an Jiaotong University in “Continual Alignment for SAM: Rethinking Foundation Models for Medical Image Segmentation in Continual Learning”, proposing an Alignment Layer for efficient domain adaptation.
  • Remote Sensing: For geospatial understanding, Xu Zhang et al. from Xidian University introduce “Bridging Semantics and Geometry: A Decoupled LVLM-SAM Framework for Reasoning Segmentation in Remote Sensing”, decoupling semantic reasoning from pixel prediction. Kaiyu Li et al. from Xi’an Jiaotong University and Chinese Academy of Sciences explore “SegEarth-OV3: Exploring SAM 3 for Open-Vocabulary Semantic Segmentation in Remote Sensing Images” for training-free open-vocabulary segmentation, while M.Naseer Subhani introduces “ReSAM: Refine, Requery, and Reinforce: Self-Prompting Point-Supervised Segmentation for Remote Sensing Images” using sparse point annotations.
  • Security & Safety: In a proactive measure against deepfakes, Qi Song et al. from Hong Kong Baptist University propose “Creating Blank Canvas Against AI-enabled Image Forgery”, which leverages adversarial perturbations to make AI-forged content more detectable. For industrial safety, Hang Chen et al. from Wuhan University adapt SAM for “Power Transmission Corridor Hazard Segmentation” with ELE-SAM.

Under the Hood: Models, Datasets, & Benchmarks

These advancements are powered by innovative models, novel datasets, and robust benchmarking strategies:

• SAM/SAM2/SAM3 and their Variants: The foundational Segment Anything Model series (SAM, SAM2, SAM3) serves as the backbone, with many papers focusing on optimizing or extending its capabilities. Variants like Fast SAM2, Q-SAM2, EfficientSAM3, and SAM3-UNet are developed for efficiency, while domain-specific adaptations include 3DTeethSAM, SSL-MedSAM2, GBT-SAM, UniUltra, ELE-SAM, and MedSAM3.
• New Architectures & Modules: Innovations include:
  • Text-driven token pruning in Fast SAM2 for video efficiency.
  • Decoupled LVLM-SAM framework (Think2Seg-RS) for semantic reasoning.
  • Lightweight modules (Prompt Embedding Generator, Mask Refiner, Mask Classifier) and Deformable Global Attention Plugins (DGAP) in 3DTeethSAM.
  • Variance-Reduced Calibration (VRC) and Learnable Statistical Clipping (LSC) in Q-SAM2 for quantization.
  • Context-Aware Prompt Adapter (CAPA) and High-Fidelity Mask Decoder (HFMD) in ELE-SAM.
  • Alignment Layer in CA-SAM for continual learning in medical segmentation.
  • AugModule and Module Selector in SAMCL for storage-efficient continual learning.
  • Mask Fusion Strategy and Presence Score Filtering in SegEarth-OV3 for open-vocabulary segmentation.
  • Perceptual-Consistency Clipping and Prompt-Aware Reconstruction in SAQ-SAM for semantically-aligned quantization.
  • Memory-guided Gated Fusion Module in BoxPromptIML for weakly supervised image manipulation localization.
• Novel Datasets & Benchmarks: To evaluate these specialized models, researchers are creating new resources:
  • The SA-Co benchmark for Promptable Concept Segmentation (PCS) introduced with SAM 3: Segment Anything with Concepts.
  • The FMOX dataset for benchmarking SAM2-based trackers on fast-moving objects, as seen in “Benchmarking SAM2-based Trackers on FMOX” from Aktas et al. at Maynooth University.
  • ELE-40K, the first large-scale real-world dataset for Power Transmission Corridor Hazard Segmentation, introduced by Hang Chen et al. from Wuhan University.
  • SA-SV Benchmark, the largest surgical iVOS benchmark with instance-level spatio-temporal annotations, presented with SAM2S: Segment Anything in Surgical Videos via Semantic Long-term Tracking from Haofeng Liu et al. at National University of Singapore.
  • SurgBlood dataset for laparoscopic bleeding detection, built by Jialun Pei et al. from The Chinese University of Hong Kong.
  • A large-scale dataset with over 27 million triplets of images, region annotations, and text descriptions across ten biomedical imaging modalities, contributing to UniBiomed: A Universal Foundation Model for Grounded Biomedical Image Interpretation from Linshan Wu et al..
  • The CellFMCount dataset for fluorescence microscopy cell counting, introduced by the NRT-D4 Team.
• Code Availability: Many projects offer open-source code for wider adoption and further research:
  • Think2Seg-RS
  • Automated Mosaic Tesserae Segmentation
  • OW-Rep
  • A Unified Framework with Multimodal Fine-tuning for Remote Sensing Semantic Segmentation
  • 3DTeethSAM
  • SSL-MedSAM2
  • UniBiomed
  • A Distributed Framework for Privacy-Enhanced Vision Transformers on the Edge (code link for framework, not SAM)
  • SegEarth-OV3
  • Team-Aware Football Player Tracking with SAM
  • SAMCL
  • SAM3-I
  • NAS-LoRA
  • On Efficient Variants of Segment Anything Model: A Survey
  • SAM3-UNet
  • GBT-SAM
  • Creating Blank Canvas Against AI-enabled Image Forgery
  • BoxPromptIML
  • SAM Guided Semantic and Motion Changed Region Mining for Remote Sensing Change Captioning
  • ELE-SAM
  • Supervise Less, See More
  • SD-MVS (code link for SD-MVS)
  • CellFMCount
  • MedSAM3
  • SCALER
  • Attention Guided Alignment in Efficient Vision-Language Models
  • Continual Alignment for SAM
  • UniUltra
  • Unbiased Semantic Decoding
  • LithoSeg
  • SAQ-SAM

Impact & The Road Ahead

The research summarized here paints a vibrant picture of SAM’s evolving role in AI. These advancements are not merely incremental; they represent a concerted effort to make powerful foundation models more versatile, efficient, and reliable for specialized tasks. From streamlining surgical procedures with precise instrument tracking (SAM2S from National University of Singapore), to enabling real-time detection of bleeding in laparoscopic surgery (BlooDet from The Chinese University of Hong Kong), to automating tedious tasks like mosaic tesserae segmentation (Automated Mosaic Tesserae Segmentation via Deep Learning Techniques), the potential for real-world impact is immense.

The shift from prompt-based to concept-driven segmentation (SAM3), alongside the development of parameter-efficient fine-tuning strategies (e.g., NAS-LoRA from Fudan University, UniUltra from HKUST), signals a move towards more intelligent, adaptive, and deployable AI systems. Open-world object detection (OW-Rep from KAIST) and continual learning methods (SAMCL from Xidian University) promise models that can learn and adapt continuously, addressing the dynamic nature of real-world data.

Looking forward, the integration of Multimodal Large Language Models (MLLMs) with SAM, as seen in UniBiomed (Harvard University) and uLLSAM (Fudan University), is particularly exciting. This fusion enables a deeper, more contextual understanding of images, bridging the gap between pixel-level analysis and semantic reasoning. The ongoing focus on privacy-enhanced frameworks (A Distributed Framework for Privacy-Enhanced Vision Transformers on the Edge from Rutgers University) also highlights a critical direction for responsible AI deployment. The Segment Anything Model family continues to evolve rapidly, promising a future where AI can truly understand and segment anything, anytime, anywhere, with unprecedented efficiency and precision.