Mixed-integer programming can handle optimization problems with complex constraints, but its computational cost often suffers from the combinatorial complexity of the problem. Decomposition-based matheuristics address this issue by splitting large-scale mixed-integer programs (MIPs) into smaller subproblems. Matheuristics typically exhibit hyperparameters that may affect their performance. An analysis of related work reveals that the optimization potential of hyperparameters is often left unexploited, leading to both inferior MIP-solutions and unnecessarily high computational costs. This paper studies a novel algorithmic approach to tune hyperparameters of matheuristics by Bayesian optimization. Fundamental properties of the algorithmic approach are examined by computational experiments with small- and large-scale instances of the use case. The results exhibit two natural and competing objectives of the tuning problem: optimizing the MIP-objective and the computational cost. While the two objectives can be optimized separately for small-scale instances, they need to be handled jointly for large-scale instances. In future research the multi-objective aspect of the hyperparameter tuning problem will be examined more deeply, and the single-instance approach will be extended to multiple instances.