With the development of user-generated content, many real-world corpora involve various types of metadata, such as timestamp and manually defined labels, and yet most of the classical topic modeling methods ignore this information. Besides, many contemporary topic models continue to grapple with the issue of noise contamination, particularly when they are applied to small or social media datasets. Given a corpus where each document contains a timestamp, lifelong topic models are developed to enhance topic quality over such documents sorted by timestamp, but the order of topics extracted by existing methods may shift sequentially. To address this issue, we exploit nonnegative matrix tri-factorization, a classical method for co-clustering, as a framework of lifelong topic modeling with an aligned topic space. We also develop a data distribution scheme followed by a distributed parallel algorithm, which scales our model to large-scale text streams. In this vein, a lifelong hierarchical topic model based on nonparametric clustering is further proposed. Finally, we develop a label aligned neural topic model, an anchor guided hierarchical topic model, and a self-supervised hierarchical topic model, to alleviate the issue of noise contamination by exploiting labels or embeddings.