To achieve accurate lesion classification, features from the lesion attention maps can be extracted by an encoder, such that high-level lesion features can be captured for the classifier module. Thus, in each branch, an encoder is incorporated after the segmentor to extract each domain’s specific features. Besides, we propose to fuse the lesion features and the prostate features to boost the classification accuracy. Skip connection and concatenation operations are introduced to reuse prostate features from the segmentors.
We design a domain transfer module (in Figure 4 ) without requiring target labels in the training process. The semantics features from both the prostate region and attention map are fused, such that deep coral features from fully connected (FC) layers can be captured for feature affinity. Deep Coral loss [25] is employed to minimize cross-domain feature distribution discrepancy, owing to its generality, transferability, and ease of implementation. It is defined as the difference of second-order covariances between domains. Our domain transfer loss \({\mathcal{L}_{Coral}}\) is defined as: