This study aims to examine the efficacy of transformer models with bayesian optimization (BO) hyperparameters for short-term load forecasting (STLF) in the Australian electricity market. The proposed approach is compared with the convolutional neural network (CNN)-BO and long short-term memory (LSTM)-BO networks, utilizing historical load data and relevant exogenous variables. The transformer-BO enables efficient exploration of the parameter space. The results indicate that the proposed model significantly improves over CNN-BO and LSTM-BO models in forecast accuracy, particularly in capturing complex temporal dependencies and adapting to abrupt load variations. To validate the proposed model, an electricity dataset has been used for experiments. The transformer-BO model shows improvements ranging from about 2% to 33% across different metrics and datasets, with the most significant improvements in MSE for both datasets. This study contributes to load forecasting (LF) by elucidating the potential of transformer-BO in Australia, providing valuable insight to grid operators and energy market participants.