Overlapping cell nuclei segmentation in digital histology images using intensity-based contours

Document Type

Conference Proceeding

Publication Title

2021 Digital Image Computing: Techniques and Applications (DICTA)




School of Science / School of Medical and Health Sciences / Centre for Precision Health / Graduate Research




Hossain, M. S., Armstrong, L. J., Abu-Khalaf, J., Cook, D. M., & Zaenker, P. (2021, November-December). Overlapping cell nuclei segmentation in digital histology images using intensity-based contours [Paper presentation]. 2021 Digital Image Computing: Techniques and Applications (DICTA), Gold Coast, Australia.



Automated nuclei segmentation techniques in histopathological image analysis continue to improve. The machine learning model requires the annotation of large data sets which is a time-consuming, expensive, and laborious process. This segmentation is also limited in detecting touching or overlapping nuclei and considers any overlapping nuclei as a single nucleus. This is due to low contrast images, occultation, and diversity of cell nuclei. This work proposes an automated overlapping nuclei segmentation model with a U-net and an intensity-based contour technique in order to address these issues. In a previous study, a U-net segmentation model was trained with synthetic data, which was generated using a GAN model, where a small number of histopathology data was used to generate the synthetic data. This reduced the data limitation and need for nuclei annotation in the deep learning model. Initially in this study, the overlapping nuclei regions were not considered for segmentation by the network. Hence, an intensity-based contour line is proposed to separate overlapping nuclei regions. The distance transformation is utilized to define the center of each nucleus. The identification of local minima followed by intensity-based gradient weights is applied to obtain the final segmentation line of overlapping nuclei. The boundary of the overlapping nuclei is refined, and noise is removed in order to clearly describe each nuclei region. The proposed method results in 91.6% accuracy in separating the overlapping nuclei compared to other existing methods.



Access Rights

subscription content

Research Themes


Priority Areas

Prevention, detection and management of cancer and other chronic diseases