Most research on subgroup fairness has been done with models that predict outcomes, where there typically should be parity between subgroups and main groups. In contexts where meaningful differences between different populations are expected, differences in classification could be due to actual differences or to biases in the classification process, confounding studies of subgroup fairness in such cases. This produces a fundamental challenge: how do we study the causes of and potential solutions to unfairness in classification when differences between subgroups are to be expected? To address this challenge, we aligned text from policy documents officially published in two different languages to test the fairness of classifiers designed to identify the same constructs in multiple languages by testing the extent to which the classifiers made the same coding decisions on items equivalent in content but expressed in different languages. This study presents a systematic analysis of the frequency and types of errors that occur in the classifier training process and lead to biased coding decisions, and it shows how a novel technique, negative reversion, can significantly reduce such errors.