Upper tribunal decisions form a vital component of the immigration appeal system in the United Kingdom (UK), since these decisions often act as a ‘corrective’ to initial judicial decision-making by the first-tier tribunal. This paper describes the process of annotating a corpus drawn from the UK’s Upper Tribunal Immigration and Asylum Chamber (UTIAC) decisions openly published by the tribunal, covering the years 2000 to 2021. A label taxonomy is developed and applied by two annotators for annotation of several types of features, including decision outcomes and the presence of and type of legal errors identified by the tribunal. Annotations were implemented via a low-cost, high-accessibility tool and validated calculating inter-annotator agreement. We discuss the pros and cons of our annotation tool and also critically reflect on annotator background. We successfully used the produced ground truth for supervised machine learning classification tasks, adding further confidence to the suitability of annotations for said tasks. We are considering ethics and data protection compliant ways to safely make the corpus and annotations available for future work.