Objectives <p>Magnetic resonance imaging (MRI) is a complex medical imaging method where multiple technical and physiological factors may lead to undesired changes in image quality. The quality control methods utilizing test objects are useful in measuring technical performance, but they may not fully detect all factors present in clinical imaging. In this study, we developed methodologies to quantify observer-based image quality and to compare these observations with technical quality control (QC) parameters.</p> Materials and methods <p>We analysed 150 brain MRI 3D-FLAIR volumes from 15 scanners, measuring image quality both quantitatively and by visually ranking the images using forced-choice comparison.</p> Results <p>Significant differences were found between different scanners based on the forced choice comparison. In imaging study-specific analysis, a weak correlation was observed with contrast-to-noise ratio (CNR) (<i>R</i>² = 0.17) and brain white matter–gray matter (WM/GM) contrast (<i>R</i>² = 0.14). With device-specific median correlation, the CNR and WM/GM contrast <i>R</i>² were 0.21 and 0.34, respectively. Additionally, using device-specific median values, a correlation was found with image quality index (QI) (<i>R</i>² = 0.21) and some modulation transfer function (MTF) based resolution-specific parameters (MTF10 FH, <i>R</i>² = 0.19; MTF10 AP, <i>R</i>² = 0.20; MTF50 AP, <i>R</i>² = 0.17).</p> Discussion <p>The forced choice comparison can be effectively utilized to rank image quality across multiple MRI scanners. Technical image quality parameters, directly analysed from anatomical image volumes, can offer prospective maintenance value. Additionally, the quality of clinical image volumes can be assessed using both forced choice comparison and calculational image analysis methods.</p>