The recognition of oracle bone characters, one of the earliest mature writing systems in the world, is crucial for studying Chinese history. Current methods achieve satisfactory performance in closed-set scenarios but struggle with newly discovered characters in open-set environments. Existing open-set recognition approaches either use single prototypes per class or employ uniform decision boundaries for multiple prototypes, which cannot handle the problem of high intra-class variance in oracle characters. This paper proposes an open-set recognition algorithm with adaptive decision boundaries. By estimating intra-class prototype distributions, our method dynamically adjusts decision thresholds for each prototype. Key innovations include: (i) a self-attention mechanism is employed to aggregate global representations from multiple prototypes of the same category, (ii) a relational network is used to determine prototype-specific thresholds based on spatial position of each prototype in the global feature space, and (iii) domain adversarial training is adopted to align features between the rubbing images from testing stage and the handprinted images from training stage. Experiments on the OBC306 and SOC5519 datasets demonstrate improved performance over state-of-the-art methods, with AUROC scores of 87.60% and significant improvements in both known class acceptance (83.62%) and unknown class rejection (77.80%).