Background <p>Clear communication is critical for diagnosis and disease management, especially for patients with a non-English language preference (NELP). Health systems use various strategies to certify clinicians in non-English languages, and patient ratings have been proposed as one assessment method.</p> Objective <p>Evaluate the reliability of patient ratings as well as the validity of patient and clinician self-assessments of clinician non-English language skills.</p> Design <p>Observational study leveraging data from the Language Access System Improvement (LASI) study at the University of California San Francisco (UCSF).</p> Participants <p>The sample included 323 patients with NELP (Cantonese, Mandarin, or Spanish) and 32 academic primary care clinicians.</p> Main Measures <p>Patient ratings and clinician self-assessments of clinician non-English language skills (5-category); algorithm-based indicator of clinician bilingual verification status (binary, validating measure).</p> Key Results <p>The estimated reliability/dependability coefficient (<i>Φ</i>) of a single patient rating equaled 0.29. Averaging six patient ratings yielded acceptable dependability for research (<i>Φ</i> = 0.707). Higher reliability required averaging 10 (<i>Φ</i> = 0.801), 23 (<i>Φ</i> = 0.902), or 48 (<i>Φ</i> = 0.951) patient ratings. Clinician-averaged patient ratings were weakly correlated with bilingual verification status, suggesting low validity (<i>r</i>_b = 0.28, <i>p</i> &lt; 0.0001). Clinician self-assessments had a substantially higher validity coefficient when measured against verification status (<i>r</i>_b = 0.75, <i>p</i> &lt; 0.0001).</p> Conclusions <p>Patient ratings of clinician non-English language proficiency had low reliability and validity. Clinician self-assessments showed stronger validity and may be sufficient for research use but should be verified with standardized assessments for clinical certification. Patient feedback may still inform communication training and quality improvement.</p>