This study provides a comprehensive evaluation and comparison of different families of optimization methods, including probabilistic methods (e.g., Bayesian optimization), evolutionary, and random methods (e.g., Monte Carlo), in the context of hyperparameter tuning for a gradient boosting model applied to a multiclass classification problem. The study evaluates these methods based on their performance in finding optimal solutions, the computation time and computational resources required, and their behavior in terms of the search space exploration and exploitation trade-off. The results indicate that probabilistic methods show superior performance in exploiting optimal solutions, but at the cost of increased computational time. Monte Carlo methods, on the other hand, are based on sampling the search space, potentially leading to diverse solutions. Evolutionary algorithms demonstrate a balance between exploration and exploitation, whereas random methods show mixed behavior. The study also highlights the importance of specific hyperparameters and the complex interplay between them.