In recent years, dense retrieval approaches have brought significant improvements in information retrieval effectiveness. However, these approaches are significantly more complex than traditional retrieval techniques, decreasing their transparency and explainability. ColBERT, a popular dense retrieval approach, uses a late interaction mechanism (MaxSim) to match the embeddings of tokens in queries and documents. Compared to single-representation dense retrieval approaches, this late interaction offers transparency, in that it allows to observe token-level MaxSim matches between the query and document, and even measure the extent that tokens were matched in exact “lexical” matches, or “semantic” matches. However, previous measures for semantic matching do not take into account how semantic matching influences the model’s effectiveness, i.e. whether it is beneficial or harmful. We propose S-nDCG, which weights the relevance gains in nDCG by the semantic or lexical proportion of matching. In combination with nDCG, this allows to quantify the extent to which a (ColBERT-based) model’s effectiveness can be attributed to semantic or lexical matching, respectively. Through experiments on TREC Deep Learning track queries, we examine how ColBERT’s effectiveness is obtained over training iterations, and find that generally ColBERT learns to rely more heavily on semantic matching throughout training. Surprisingly though, there is a significant decrease of semantic matching during the first 5k training iterations, indicating that even though BERT’s pre-trained token embeddings provide a base for the semantic matching of several different token types, they likely lead to a lot of inaccurate semantic matches at the beginning of training that are addressed by fine-tuning for retrieval.