Background Determining an adequate sample size is essential for developing reliable and generalisable clinical prediction models, yet practical guidance on selecting appropriate sample size calculation methods remains limited. Existing analytical and simulation-based tools impose restrictive assumptions and focus on mean-based criteria. We present and validate <Emphasis FontCategory="NonProportional">pmsims</Emphasis>, an <Emphasis FontCategory="SansSerif">R</Emphasis> package that uses Gaussian process (GP) surrogate modelling to provide a flexible and efficient simulation-based framework for sample size determination, validated here across binary, continuous, and survival prediction modelling contexts. Methods We conducted a comprehensive simulation study with two aims. Aim 1 compared three search engines implemented in <Emphasis FontCategory="NonProportional">pmsims</Emphasis>, a GP surrogate-based adaptive procedure (<Emphasis FontCategory="NonProportional">gp</Emphasis>), a deterministic bisection method (<Emphasis FontCategory="NonProportional">bisection</Emphasis>), and a hybrid GP-bisection approach (<Emphasis FontCategory="NonProportional">gp-bs</Emphasis>), across binary, continuous, and survival outcomes. Scenarios varied outcome prevalence or event rate, predictor dimensionality (<InlineEquation ID="IEq1"><EquationSource Format="TEX">\(p = \{10, 100\}\)</EquationSource></InlineEquation>), target performance metric (discrimination and calibration slope), aggregation criterion (mean vs 80% assurance), and total simulation budget (<InlineEquation ID="IEq2"><EquationSource Format="TEX">\(B = 200-2000\)</EquationSource></InlineEquation>). Each scenario was replicated 100 times; estimator stability was assessed via the coefficient of variation (CV). Aim 2 benchmarked the best-performing <Emphasis FontCategory="NonProportional">pmsims</Emphasis> engine against <Emphasis FontCategory="NonProportional">pmsampsize</Emphasis> (analytical) and <Emphasis FontCategory="NonProportional">samplesizedev</Emphasis> (simulation-based) across a wider range of realistic scenarios, evaluating recommended sample sizes, computational time, and achieved model performance on independent validation datasets of 30,000 observations. Results The GP-based search engine consistently yielded the most stable sample size estimates (lowest CV) across all outcome types, ranking highest in 9/12 outcome-aggregation metric configurations. Its advantage was most pronounced in low-signal, high-dimensional settings, and was accentuated with <InlineEquation ID="IEq3"><EquationSource Format="TEX">\(\kappa = 20\)</EquationSource></InlineEquation> replications per evaluation and a budget of <InlineEquation ID="IEq4"><EquationSource Format="TEX">\(B \approx 1000\)</EquationSource></InlineEquation>, after which gains were minimal. In benchmark comparisons, <Emphasis FontCategory="NonProportional">pmsims</Emphasis> (mean) achieved performance deviations within <InlineEquation ID="IEq5"><EquationSource Format="TEX">\(\pm 1\%\)</EquationSource></InlineEquation> of the prespecified target across binary, continuous, and survival outcomes, comparable to <Emphasis FontCategory="NonProportional">samplesizedev</Emphasis> and substantially outperforming <Emphasis FontCategory="NonProportional">pmsampsize</Emphasis> in high-discrimination settings (deviations up to <InlineEquation ID="IEq6"><EquationSource Format="TEX">\(-9.84\%\)</EquationSource></InlineEquation>). Conclusions The <Emphasis FontCategory="NonProportional">pmsims</Emphasis> package, using the GP-based search engine with <InlineEquation ID="IEq7"><EquationSource Format="TEX">\(\kappa \ge 20\)</EquationSource></InlineEquation> replications per evaluation and a budget of <InlineEquation ID="IEq8"><EquationSource Format="TEX">\(B \approx 1000\)</EquationSource></InlineEquation>, provides a computationally efficient and flexible framework for principled sample size planning in clinical prediction modelling. It reliably achieves performance targets across the range of standard-model scenarios evaluated while requiring fewer model evaluations than non-adaptive simulation approaches, offering a compelling alternative to both analytical formulae and exhaustive simulation-based search.

Adaptive Gaussian process search for simulation-based sample size estimation in clinical prediction models: validation of the pmsims R package

期刊论文
出版模式：

开放获取
发表日期：

2026年7月13日

查看全文

Oyebayo Ridwan Olaniran,
Diana Shamsutdinova,
Sarah Markham,
Felix Zimmer,
Daniel Stahl,
Gordon Forbes,
Ewan Carr

摘要

Background

Determining an adequate sample size is essential for developing reliable and generalisable clinical prediction models, yet practical guidance on selecting appropriate sample size calculation methods remains limited. Existing analytical and simulation-based tools impose restrictive assumptions and focus on mean-based criteria. We present and validate pmsims, an R package that uses Gaussian process (GP) surrogate modelling to provide a flexible and efficient simulation-based framework for sample size determination, validated here across binary, continuous, and survival prediction modelling contexts.

Methods

We conducted a comprehensive simulation study with two aims. Aim 1 compared three search engines implemented in pmsims, a GP surrogate-based adaptive procedure (gp), a deterministic bisection method (bisection), and a hybrid GP-bisection approach (gp-bs), across binary, continuous, and survival outcomes. Scenarios varied outcome prevalence or event rate, predictor dimensionality (\(p = \{10, 100\}\)), target performance metric (discrimination and calibration slope), aggregation criterion (mean vs 80% assurance), and total simulation budget (\(B = 200-2000\)). Each scenario was replicated 100 times; estimator stability was assessed via the coefficient of variation (CV). Aim 2 benchmarked the best-performing pmsims engine against pmsampsize (analytical) and samplesizedev (simulation-based) across a wider range of realistic scenarios, evaluating recommended sample sizes, computational time, and achieved model performance on independent validation datasets of 30,000 observations.

Results

The GP-based search engine consistently yielded the most stable sample size estimates (lowest CV) across all outcome types, ranking highest in 9/12 outcome-aggregation metric configurations. Its advantage was most pronounced in low-signal, high-dimensional settings, and was accentuated with \(\kappa = 20\) replications per evaluation and a budget of \(B \approx 1000\), after which gains were minimal. In benchmark comparisons, pmsims (mean) achieved performance deviations within \(\pm 1\%\) of the prespecified target across binary, continuous, and survival outcomes, comparable to samplesizedev and substantially outperforming pmsampsize in high-discrimination settings (deviations up to \(-9.84\%\)).

Conclusions

The pmsims package, using the GP-based search engine with \(\kappa \ge 20\) replications per evaluation and a budget of \(B \approx 1000\), provides a computationally efficient and flexible framework for principled sample size planning in clinical prediction modelling. It reliably achieves performance targets across the range of standard-model scenarios evaluated while requiring fewer model evaluations than non-adaptive simulation approaches, offering a compelling alternative to both analytical formulae and exhaustive simulation-based search.