HoloFluoNet: Live cell imaging intelligence based on fused holography and fluorescence for virtual staining, cell segmentation, classification, and viability analysis

Abstract

Fluorescence microscopy enables detailed visualization of subcellular structures but is limited by phototoxicity, photobleaching, and labor-intensive labeling. In contrast, digital holographic microscopy (DHM) offers label-free, quantitative phase imaging but lacks biochemical specificity. To integrate the strengths of both modalities, we propose HoloFluoNet, a deep learning-based framework that generates virtual fluorescence information from phase images acquired by DHM. Using a dual-mode imaging system, we simultaneously captured phase and fluorescence images of live cancer cells. The fluorescence data were used to construct biologically grounded supervision signals, including nuclei masks and multi-class cell viability labels. From a single-phase image, HoloFluoNet predicts a virtual nuclei mask, a distance map for boundary refinement, and a viability mask. The architecture incorporates multi-scale feature extractor, and attention mechanisms, optimized with novel inclusion and exclusion loss functions. Post-processing using the watershed algorithm ensures accurate segmentation of overlapping cells. The final registered image provides label-free virtual staining for nuclei localization, viability assessment, and class-specific morphological profiling. HoloFluoNet achieved a Dice score of 0.8685 and an Aggregated Jaccard Index (AJI) of 0.7883, outperforming conventional deep learning models such as U-Net, Fully Convolutional Network (FCN), Pyramid Scene Parsing Network (PSPNet), and DeepLab v3+. These improvements were statistically validated using one-way analysis of variance (p < 0.01). Ablation experiments confirmed the complementary roles of architectural modules and novel loss functions, while robustness tests under noisy and low-quality conditions revealed high stability to low contrast and moderate noise. With an inference speed of 0.123 s per image, the model enables real-time cellular analysis. By bridging structural and molecular imaging, HoloFluoNet provides an efficient, label-free alternative to fluorescence microscopy, with promising applications in artificial intelligence (AI)-assisted drug screening, cancer research, and live-cell monitoring.