SIGNIFICANCE: Digital holographic microscopy (DHM) is a promising technique for the study of semitransparent biological specimen such as red blood cells (RBCs). It is important and meaningful to detect and count biological cells at the single cell level in biomedical images for biomarker discovery and disease diagnostics. However, the biological cell analysis based on phase information of images is inefficient due to the complexity of numerical phase reconstruction algorithm applied to raw hologram images. New cell study methods based on diffraction pattern directly are desirable. AIM: Deep fully convolutional networks (FCNs) were developed on raw hologram images directly for high-throughput label-free cell detection and counting to assist the biological cell analysis in the future. APPROACH: The raw diffraction patterns of RBCs were recorded by use of DHM. Ground-truth mask images were labeled based on phase images reconstructed from RBC holograms using numerical reconstruction algorithm. A deep FCN, which is UNet, was trained on the diffraction pattern images to achieve the label-free cell detection and counting. RESULTS: The implemented deep FCNs provide a promising way to high-throughput and label-free counting of RBCs with a counting accuracy of 99% at a throughput rate of greater than 288 cells per second and 200 μm × 200 μm field of view at the single cell level. Compared to convolutional neural networks, the FCNs can get much better results in terms of accuracy and throughput rate. CONCLUSIONS: High-throughput label-free cell detection and counting were successfully achieved from diffraction patterns with deep FCNs. It is a promising approach for biological specimen analysis based on raw hologram directly.