Current deep learning models for knee osteoarthritis (OA) grading often lack anatomical guidance, limiting their accuracy and explainability. This work proposes a novel framework centered on anatomically-focused patches to overcome these limitations. Our method extracts a set of small image patches from clinically-relevant locations along the joint line, identified by automated landmark detection. These patches are then processed as a bag of instances within an attention-based multiple instance learning (MIL) framework. The MIL model learns to identify and weight the most salient pathological features for an accurate, patient-level diagnosis. Our approach is evaluated on the OAI dataset and achieves state-of-the-art performance with a quadratic weighted Cohen’s Kappa of 0.807. This result outperforms larger baselines such as ResNet-34 while using over 85 times fewer parameters. Furthermore, our attention-weighted visualization method produces sharp, clinically meaningful saliency maps that precisely localize features such as osteophytes and joint space narrowing, in contrast to the diffuse heatmaps of prior work. By combining anatomical guidance with an MIL framework, our work presents a lightweight, accurate and trustworthy solution for automated knee OA grading. The code is available at: https://github.com/tien-endotchang/focused-patch-KOA .