The Interim Experiment Report: A Systematic Account of The Experimental Design of Large Language Model-Based Text-to-Model Approaches
摘要
The increasing advent of approaches that adopt Large Language Models for text-to-model generation is accompanied by a variety of experimental designs for their evaluation. Inspired by “The Prompt Report”, which provides a systematic account of prompting techniques for prompt engineering, we present a structured synthesis of experimental designs used to evaluate LLM-based text-to-model approaches. Through a systematic literature review, we compile a compendium of experimental designs, organized into six tree-structured categories. This compendium integrates both experimental design principles from traditional empirical research and challenges unique to LLM usage. For example, specific to LLMs, we identify studies that conduct an ablation study to isolate the effect of individual prompting techniques, as well as those that address cascading errors introduced by intermediate LLM outputs. The compendium is intended to inform the design of future evaluations in this emerging field, offering researchers structured support and guidance.