Fairness is a key aspect that needs to be considered when humans are affected by machine learning algorithms. While much work has been done for fairness in the batch setup, work on fair machine learning involving non-stationary, potentially imbalanced data streams is still quite limited. Moreover, current fair stream learning algorithms are mostly evaluated only with respect to the fairness score which the model optimized for and the choice of evaluation data is not ideal due to a lack of suitable fair stream learning benchmarks. In this work, we address these issues by proposing a pipeline for building drifting data streams with inherent biases. Our data generation framework extracts the causal relations from batch fairness benchmarks, thus yielding realistic scenarios while providing the possibility to control drift and class imbalance and introducing different biases. Additionally, we structure and summarize current fairness-aware stream learning methods and evaluate those with respect to a wider range of fairness notions on a variety of biased data streams.