Event Temporal Relation Extraction (ETRE) aims to identify the temporal relationship between two events. Most previous works follow a single-label classification paradigm, classifying an event pair into either a well-defined temporal relation (e.g., Before, After) when its temporal relation is clear and unique, or a special relation Vague when there are multiple possible temporal relations. This paradigm treats Vague independently, ignoring that the instances labeled as Vague also convey temporal information about well-defined relations. To better handle the special relation Vague, we propose Multi-label Event Temporal Relation Extraction (METRE), a new method treating the ETRE as a multi-label classification task. METRE predicts the probability of each well-defined relation separately and infers the relation Vague if there is more than one relation possible for an event pair, instead of directly predicting the probability of Vague. Considering the gold composition of Vague is not available, we design a speculation mechanism to explore the possible relations contributing to Vague in each case, which enables the latent information to be used efficiently. Experiments on TB-Dense, MATRES and UDS-T show that our method can make better use of the relation Vague and outperform state-of-the-art methods.