Accurate and interpretable evaluation of vision language models (VLMs) is crucial for applications where accuracy and transparency are essential, such as automotive damage assessment, where structured outputs support reliable decision making. This paper presents CarDamageEval, a dual-layer evaluation framework designed to measure both the structural accuracy and semantic quality of VLM outputs. The framework enforces a predefined structured output format comprising explicit tuples of damage type, vehicle body part and severity level, enabling rigorous quantitative assessment through pair-matching metrics such as precision, recall and F1 score. Complementing this, semantic quality is assessed using the Holistic Description Score (HDS), which captures correctness, completeness, coherence and relevance. These two perspectives are unified through the hybrid CarDD Score, providing a balanced metric that rewards factual accuracy and descriptive clarity. To support structured evaluation, we compiled a dataset integrating public vehicle images, each annotated with bounding boxes and linked damage type, body part and severity labels. This baseline comparison further demonstrates the framework’s ability to distinguish model performance across configurations, highlighting the effectiveness of fine-tuning in generating accurate and structured damage descriptions. While developed for automotive assessment, the principles underlying CarDamageEval also apply to other structured vision language tasks, such as lesion detection in medical imaging or defect localisation in industrial inspection, making it a versatile and reusable evaluation standard.