Question Answering is held as one of the most applicable natural language generation tasks, assisting users in search of objective answers to specific questions, such as in chatbots or search engines. Large Language Models have been shown to surpass previous state of the art performance in the task, and yet, progress in the field is hard to track due to a lack of reliable automated model evaluation methods. The traditional evaluation metrics are known to produce results that don’t correlate well to human judgement and lack interpretability. In this work, we push for more interpretable evaluation by means of adversarial attacks that test answerers against stressful inputs, which may confuse a model but not a human being. These include the insertion of typographical mistakes, word swaps, and insertion of additional unrelated context. We employ the Q3AE method to do so, and test it on models of the LLaMA and Gemma architectures, showcasing the additional feedback attacks can provide when compared to the use of traditional metrics. We find that the models can be surprisingly brittle when exposed to character level attacks.