Darknet refers to an overlay network that requires the use of special software to access. Tor (The Onion Router) is one of the software used to access the darknet. Tor uses unique routing methods and encryption algorithms to hide users’ IP addresses and encrypt traffic to protect users’ privacy. However, some criminals use Tor's anonymity to conduct criminal activities. According to a survey report by network security service provider CloudFlare, 94% of network traffic from Tor is classified as malicious traffic. Therefore, if darknet traffic can be identified and further analyzed, it will help to combat these illegal activities. This study uses the Random Forest algorithm to analyze darknet traffic's communication content to identify services embedded within encrypted traffic. In addition to classifying known darknet communications, it also employs the Open-Set Recognition method proposed in this study to identify service types not included in the model, thus achieving detection of unknown darknet communications. Experimental results demonstrate that the classification of known darknet services has achieved an F1-Score of 0.99. Furthermore, after incorporating samples of unknown darknet activities, an overall accuracy of 0.95 can also be attained. These findings validate the effectiveness of the study in analyzing darknet traffic, thereby assisting enterprises and organizations in implementing appropriate countermeasures.