Research in machine learning (ML) for structural health monitoring (SHM) has increased in recent years due to ML’s potential to extract information from large quantities of data collected under varying conditions to make predictions, such as detection, identification, and characterization. Current ML implementation consists of training a model for narrow, task-specific needs, such as training a neural network to detect the presence of structural faults. However, advancements in ML and artificial intelligence (AI) have contributed foundation models—large models trained on broad data in a self-supervised fashion that can be quickly fine-tuned to accomplish specific downstream tasks different from the task it was trained on. For example, a foundation model that is trained for damage detection may be prompted to perform identification and characterization tasks given future data. Because the training and tasking of foundation models are inherently different from current ML implementation, metrics that capture performance while also considering these differences are needed for fair evaluation and comparison between approaches. This research focuses on the testing, evaluation, and benchmarking for foundation models trained on vibration data from ground sensors. We will present metrics that incorporate a broad series of tasks to quantify the performance of foundation models against other traditional ML models. Since the ground vibration data is employed to train and test foundation models for seismic events, the tasks of these models are well aligned with SHM: detecting, identifying, and characterizing acceleration signals or images. The comprehensive evaluation approach and metrics we present will be a step toward holistically quantifying advanced ML/AI success as it inevitably permeates the field of SHM.