A random walk is a process in which a random walker takes consecutive steps in space at equal intervals of time, with the length and direction of each step determined independently. Models related to random walks have been applied in diverse areas such as graph analysis, where node embedding attracts significant attention for artificial intelligence (AI) applications. However, existing random walk-based node embedding models either consider only topological information for local node similarity or struggle with large execution times as graph size increases. In this study, we propose RWEM, a versatile in-memory random walk-based node embedding framework, which relies on stationary transition probabilities while incorporating various node label information. The essence of RWEM is to construct a multiplex User-Item (MUI) graph that includes various node label information and to perform random walk computations on these MUI graphs without increasing computational complexity as the number of layers increases. Evaluation results demonstrate the effectiveness of RWEM with varied parameter settings on various downstream applications such as node classification and link prediction.