MC-NuSeg: Multi-Contour Aware Nuclei Instance Segmentation with Segment Anything Model

Abstract

Accurate nuclei instance segmentation is critical in digital pathology image analysis, facilitating disease diagnosis and advancing medical research. While various methods have been proposed, recent approaches leverage foundation models like the Segment Anything Model (SAM) for their robust representational power. However, existing models face challenges in handling the unique characteristics of histopathology images, particularly dense nuclei clusters, and complex morphological and staining variations. To address these issues, we propose a novel method, Multi-Contour Aware Nuclei Instance Segmentation (MC-NuSeg) framework, which incorporates the hierarchical boundary structure of nuclei for precise segmentation. MC-NuSeg predicts multiple segmentation maps corresponding to different contour layers, allowing for accurate separation of densely clustered nuclei and those with high morphological variance. Furthermore, we introduce an auxiliary instance counting loss that directly supervises the number of nuclei, significantly enhancing segmentation accuracy by reducing false positives and missed cases. Extensive evaluations on four public pathology datasets demonstrate that MC-NuSeg achieves state-of-the-art performance, effectively addressing the challenges of nuclei instance segmentation. © 2025 Elsevier B.V., All rights reserved.