Recent advancements have demonstrated the potential of reinforcement learning (RL) in enhancing cache management policies. However, existing approaches still face two critical limitations: (1) most methods treat prefetching and cache replacement as separate tasks, which restricts their adaptability to diverse and dynamic workloads; and (2) the high computational and storage overheads of typical RL techniques hinder their deployment in real-world hardware systems. To overcome these challenges, we propose LightCacheRL, a lightweight and practical RL-based cache management framework that unifies prefetching and replacement decisions within a unified formulation. Specifically, LightCacheRL models the cache management problem as a lightweight online Multi-Armed Bandit (MAB) process, enabling efficient and adaptive policy learning with minimal runtime overhead. The reward function in LightCacheRL integrates both instruction-per-cycle (IPC) and system-level bandwidth feedback, providing a hardware-aware optimization objective. We conduct comprehensive evaluations through simulation and hardware synthesis across single-core and multi-core configurations. Experimental results show that LightCacheRL achieves 11.5% average IPC improvement over LRU on a 16-core system, and outperforms state-of-the-art policies, including Hawkeye (4.35%), Glider (3.75%), and Mockingjay (2.1%), with negligible hardware cost.