Offline reinforcement learning (RL) aims to learn policies from pre-collected datasets, eliminating the need for expensive, slow, and risky interactions and thereby addressing significant challenges in practical applications. Count-based model-based methods excel in this domain by providing an uncertainty quantification based on the state-action pairs frequency information and incorporating it as the reward penalty. This design helps reduce estimated transition dynamics error. However, the absence of certain state-action pairs in the dataset—probably due to high-dimensional continuous spaces, data imbalance, or inherently natural exclusions—does not equate to their zero probability. For these state-action pairs, the previous design has raised an issue of overly severe reward penalties. To handle this situation more precisely, DuCoMo employs dual counters for states and state-action pairs to refine uncertainty estimation and risk management. This methodology helps to relax the overly restricted policy, enrich frequency information, rediscover the value of rare yet pivotal actions, and guide the agent back to the support regions. This method offers a tighter upper bound on estimated transition dynamics in theoretical aspects. Empirical results demonstrate that DuCoMo surpasses existing methods on the D4RL benchmark. By systematically integrating uncertainty quantification and addressing distributional shifts, DuCoMo provides a robust solution for offline RL, advancing the field and delivering reliable strategies for practical decision-making.