Sequential decision-making problems are a major area of research in AI due to their wide applicability. In these problems, an agent interacts with an environment over time to achieve a certain goal, with the key challenge that each decision will influence future options and outcomes. While techniques like bandit algorithms and reinforcement learning have shown strong performance in several domains, they often require a large number of interactions with the environment to achieve satisfactory performance. In many real-world settings, however, imprecise but cheaper data (e.g., interactions generated by using a low-fidelity model of the environment) can be exploited to make the training process more efficient. This has led to growing interest in multi-fidelity learning, which seeks to improve training efficiency by combining high- and low-fidelity data. This work presents theoretically grounded methods for exploiting multi-fidelity data in general learning scenarios.