One of the main concerns when dealing with electroencephalographic signals (EEG) is assuring that clean data with a high signal-to-noise ratio is recorded. The relevant denoising methods tend to have a narrow scope of application as what is noise for one application might be useful signal for some other application and there no general-purpose approach (or even paradigm) that works best across domains and applications. Machine learning methods are often used for this task, by training Autoencoders and Transformers on reconstruction and prediction, and then assuming the reconstruction/prediction error as an indication of anomalies. These approaches only take into account the morphology of the stream, and are not aware of the different, often highly contextualized, aspects of artifacts. In this article we explore the novel idea that we can create application-specific artifact detectors by training an attention-based deep neural network and then extracting from the attention layer information about what is ignored. This removes the most pressing challenge of artifact detection, namely that artifacts are vaguely defined and thus difficult to directly supervise, and allows application-specific artifact patterns to be extracted from non artifact-related supervision. We evaluated our method using electroencephalogram (EEG) signals on a sleep-stage labeling task. The performance of the proposed approach was compared against reconstruction/prediction error and against EEG-specific noise detection methods. The results indicate that the proposed method is a promising task-agnostic tool for anomaly detection in streaming data.