We present the first ever dataset of manually segmented and transcribed Ajami manuscripts written in Fulfulde and Hausa. The term Ajami refers to modified Arabic-script orthographies in Africa. Existing handwritten text recognition (HTR) and optical character recognition (OCR) models for Arabic-script languages perform poorly on West African manuscripts due to a lack of these manuscripts representation in the models’ pre-training. This leads to models struggling to adapt to Ajami style calligraphy, being unequipped to recognize Ajami specific characters, and being unable to extract certain Arabic-script diacritics which are present in Ajami manuscripts but lacking in many manuscripts for other Arabic-script languages like Arabic and Persian. The latter poses a significant challenge to Ajami HTR. We release the following as an open-source dataset: an ALTO formatting of high-quality images of Fulfulde and Hausa manuscripts, manual segmentation (region and line), and manual transcriptions. Our HTR dataset is also the first to diplomatically transcribe newly Unicode-encoded, special Quranic recitation characters. We evaluate a suite of Arabic-script recognition models specifically for historical manuscripts and find that they produce character error rates of 65–84% when attempting to automatically transcribe our curated manuscripts. Transcriptions produced by the evaluated models are released as well.