<p>Accurately deducing the emotional tenor of our visual surroundings has important repercussions, such as whether to approach a friendly group or flee a threatening mob. Perceptual decisions regarding individual emotional stimuli are heavily influenced by both task-driven (top-down) and stimulus-driven (bottom-up) attention. However, the impact of these attentional factors on perceptual decisions regarding the overarching or “gist” emotion conveyed by an ensemble of emotional stimuli remains unclear. We manipulated top-down and bottom-up attention in a task in which participants (<i>N</i> = 95) judged the average emotion of ensembles and applied drift diffusion modeling to uncover the underlying computational mechanisms. Our results showed different effects of attention, driven by task-related top-down compared to stimulus-related bottom-up factors, on overall ensemble perception. The inclusion of extremely fearful stimuli (bottom-up effect) led to (1) the ensemble being perceived as fearful more frequently and faster and (2) more efficient fearful evidence accumulation, allowing one to reach a fearful decision faster. In contrast, making fear-related decisions (top-down effect) led to (1) the ensemble being perceived as fearful less frequently and slower with a (2) wider boundary separation, indicating more evidence required for making a fearful decision. These distinct effects are unique to fear. Our findings provide important knowledge in understanding the interplay of top-down and bottom-up attentional mechanisms when swiftly integrating multiple sensory emotional inputs into a coherent perceptual experience, which carries significant implications across social and clinical contexts.</p>