This study tackled the problem of investigating the evidence for a model measuring the similarity between two related sentences. In each of the given sentences, there would be phrases that are semantically corresponding to each other. Some of them are phrases that increase the similarity of the sentences and, on the other hand, some of them are phrases that decrease the similarity of the sentences. We propose a method to extract such “similar” and “dissimilar” phrase pairs. In the proposed method, we first train a BERT model that can measure sentence similarity. Next, we find semantically corresponding phrases in a given pair of sentences. When we replace a certain corresponding phrase with a BERT [MASK] token if the trained BERT model determines that the sentences are less similar, we consider the phrase pair to be a “similar” phrase pair. On the other hand, the model determines that the sentences are more similar, we consider the phrase pair to be a “dissimilar” phrase pair. We have implemented the proposed method and conducted a case study.