<p>Humans can flexibly change rules to categorize sensory stimuli, but their performance degrades immediately after a task switch. This switch cost is believed to reflect a limitation in cognitive control, although the bottlenecks remain controversial. Here, we show that humans exhibit a brief reduction in the efficiency of using sensory inputs to form a decision after a rule change. Participants classified face stimuli based on one of two rules, switching every few trials. Psychophysical reverse correlation and computational modeling reveal a reduction in sensory weighting, which recovers within a few hundred milliseconds after stimulus presentation. This reduction depends on the sensory features being switched, suggesting a constraint in routing the sensory information flow. We propose that decision-making circuits cannot fully adjust their sensory readout based on a context cue alone, but require the presence of an actual stimulus to tune it, leading to a limitation in flexible perceptual decision making.</p>