Writer recognition involves analyzing handwritten documents with respect to the identity of the writer. Although automated methods can achieve strong benchmark results, their lack of interpretability limits practical adoption, particularly in settings where trust and verifiability are critical. To address this challenge, we propose a novel framework grounded in character-level analysis. At its core is Vectors of Locally Aggregated Characters (VLAC), a feature aggregation method that fuses the aligned outputs of a feature extractor and a feature annotator network. By aggregating local features on a per-character basis, VLAC provides the backbone for computing verifiable character-wise distances, thereby enhancing interpretability and trustworthiness. We extensively evaluate the proposed framework on two contemporary datasets (CVL and IAM) – achieving new state-of-the-art retrieval results on CVL – as well as a historical dataset (Hist-WI). Our method does not only perform well, but also facilitates interpretable insights into the decision process, paving the way for broader acceptance in practical and forensic applications.