Recognition of early neoplasia in Barrett’s Esophagus (BE) is challenging, despite advances in endoscopic technology. Even with correct identification, the subtle nature of lesions leads to significant inter-observer variability in placing targeted biopsy markers and delineation of lesions. Computer-Aided Detection (CADe) systems may assist endoscopists, however, compliance of endoscopists with CADe is often suboptimal, reducing joint performance below CADe stand-alone performance. Improved localization performance of CADe could enhance compliance. These systems often use fused consensus ground-truths (GT), which may not capture subtle neoplasia gradations, affecting classification and localization. This study evaluates five consensus GT strategies from multi-expert segmentation labels and four loss functions for their impact on classification and localization performance. The dataset includes 7,995 non-dysplastic BE images (1,256 patients) and 2,947 neoplastic images (823 patients), with each neoplastic image annotated by two experts. Classification, localization for true positives, and combined detection performance are assessed and compared with 14 independent Barrett’s experts. Results show that using multiple consensus GT masks with a compound Binary Cross-Entropy and Dice loss achieves the best classification sensitivity and near-expert level localization, making it the most effective training strategy. The code is made publicly available at: https://github.com/BONS-AI-VCA-AMC/BE-CADe-GT .