In this paper we present InsBrainTra - a new objective method for insomnia detection based on brain state sleep trajectories computed from a full-night EEG recording, without the need for sleep stage scoring. InsBrainTra automatically converts the EEG recording into a brain state trajectory of physiological parameters, extracts 3D-volume occupancy features and uses them in conjunction with machine learning algorithms to distinguish between people with insomnia and good sleepers. We conducted a comprehensive evaluation of different feature representations, feature subset selection and machine learning algorithms. The best model employed Random Forest and achieved 90.35% accuracy and 94.64 AUC on the [16, 16, 16]-EW-C data representation, using only 15 3D-volume features. To gain a better understanding of the most important features distinguishing insomnia and good sleepers, we performed statistical and model interpretability analysis. Our work demonstrates the potential of using brain state trajectory data with volume occupancy features for objective and efficient insomnia detection.