The dark web is often perceived by many as a concealed portion of the Internet, which accommodates, promotes, and shields criminal activities, making it one of the critical areas in cyber security. However, being designed to be both anonymous and encrypted, the dark web becomes highly challenging to monitor, control, or detect in case of threats. Threat intelligence feeds and information sharing initiatives which aim at combating these challenges through traditional means of scanning the dark web have disappointing results. The scope of this paper is a machine learning study aimed at evaluating the performance of traditional and transformer-based models on the DNRTI and Agora datasets. Performance of every model is evaluated in terms of accuracy, precision, recall, and F1 score. The findings indicate that BERT performs best on cyber threats in the DNRTI dataset, while DarkBERT excels at general threats in the Agora dataset. This reinforces the previously validated inference that transformer-based models outperform traditional machine learning models for multi-class classification of threat sentences for threat detection. This paper provides insights into the use of transformer-based models for threat detection on the dark web, with results that are promising for improving automated threat detection systems.