Although the Transformer architecture has become the predominant framework for NLP tasks, as of today it is difficult to pre-train one of these models for a low-resource language, due to the large amount of texts required to successfully train one. Recent work, however, has started to uncover a series of supposed linguistic capabilities achieved by these models –which they seemingly acquire simply by their exposure to raw text during pre-training. In this thesis I propose to i) examine the mechanisms of these models for achieving this linguistic competence and ii) explore how to modify them. This will be done to attempt to adapt one of these models, trained for one language, to a minority language from the same family. This is proposed via the use of Linguistic Linked Data (LLD) resources, encoding expert-defined lexical and grammatical knowledge of a specific language, to serve as input to custom solutions that inject linguistic knowledge into the parameters of this architecture. I propose small-scale modifications of models regarding their abilities at different linguistic levels –e.g. syntax, morphology, etc.–, in the context of Aragonese, a minority Romance language, as a testbed to evaluate the validity of this hypothesis. The outreach of this thesis is intended to help develop cross-lingual transfer technologies that are more interpretable, for related languages sharing similar features.