As modern IT infrastructures grow in complexity, root cause analysis (RCA) is becoming increasingly crucial for Site Reliability Engineering (SRE). Traditional RCA relies heavily on human expertise, making incident resolution time-consuming and error-prone. With the rise of AIOps (Artificial Intelligence for IT Operations), Large Language Models (LLMs) have emerged as potential tools for automating incident detection and diagnosis. This study evaluates the capability of GPT-4o, Gemini-1.5, and Mistral-small in diagnosing system failures purely from observability metrics within a chaos engineering framework. We simulate eight real-world failure scenarios in a controlled e-commerce environment and assess LLMs’ performance in zero-shot and few-shot settings compared with Site Reliability Engineers. While LLMs can identify common failure patterns, their accuracy is highly dependent on prompt engineering. In zero-shot settings, models achieve moderate accuracy (44–58%), often misattributing harmless load spikes as security threats. However, few-shot prompting improves performance (60–74% accuracy), suggesting that LLMs require structured guidance for reliable RCA. Despite their potential, LLMs are not yet ready to replace human SREs, who achieved over 80% accuracy due to hallucinations, misclassification biases, and lack of explainability. The findings highlight that LLMs can be co-pilots in incident response, but human oversight remains essential.GitHub with code and dataset: https://github.com/szandala/llms-chaos-engineering