The increasing use of Large Language Models (LLMs) for document analysis in decision support systems calls for a deeper understanding of their consistency and evaluative robustness. This paper examines these aspects through the lens of supply chain risk and disruption management as a representative high-stakes application. We analyze how LLMs interpret and assess textual risk indicators in news articles, using a standardized risk assessment task to benchmark 22 state-of-the-art LLMs from commercial and open-source providers. Our findings indicate an acceptable degree of ensemble agreement (Krippendorff’s alpha up to 68.3%) while also highlighting distinct evaluative clusters, often aligned with provider ecosystems, and several consistent outliers. Open-source models perform comparably to proprietary alternatives. The study underscores the potential of ensemble-based strategies that leverage model diversity to synthesize both consensus and divergent perspectives in complex document analysis tasks.