We introduce Active Learning with Binary Feedback (ALBF), a novel paradigm that minimizes annotation costs by transforming labeling into a series of binary verification queries based on model predictions. Unlike traditional active learning approaches that assume uniform annotation costs, ALBF dynamically adapts to varying labeling difficulties across samples. We propose the Info-Cost-Influence Selector (ICIS) method, which jointly optimizes sample selection based on information gain, rank-aware cost, and future annotation value. Our approach addresses the limitations of myopic selection strategies by considering both immediate model improvement and long-term annotation efficiency. Extensive experiments across diverse datasets demonstrate that our method achieves state-of-the-art performance while significantly reducing total annotation effort, offering a more practical solution for real-world machine learning deployments.