Precise separation of adjacent nuclei using a siamese neural network

Nuclei segmentation in digital histopathology images plays an important role in distinguishing stages of cancer. Recently, deep learning based methods for segmenting the nuclei have been proposed, but precise boundary delineation of adjacent nuclei is still challenging. To address this problem, we propose a post processing method which can accurately separate the adjacent nuclei when an image and a predicted nuclei segmentation are given. Specifically, we propose a novel deep neural network which can predict whether adjacent two instances belong to a single nuclei or separate nuclei. By borrowing the idea of decision making with Siamese networks, the proposed network learns the affinity between two adjacent instances and surrounding features from a large amount of adjacent nuclei even though the training data is limited. Furthermore, we estimate the segmentation of instances through a decoding network and then use their overlapping Dice score for class prediction to improve the classification accuracy. The proposed method effectively alleviates the over-fitting problem and compatible with any cell segmentation algorithms. Experimental results show that our proposed method significantly improves the cell separation accuracy.