Abstract <p>Precise measurement of rock fracture toughness and accurate observation of crack propagation paths are crucial for disaster prevention and mitigation in rock mass engineering. Due to the lack of standardized testing methods for mixed-mode I/II fracture in rocks, significant discrepancies exist in characterizing the fracture behavior of specimens with different sizes and geometric configurations. To address this issue, this study employs three-dimensional discrete element models with four different sizes, five different geometric configurations, and six different mixed-mode Ⅰ/Ⅱ loading conditions to investigate the size and geometric effects during the rock fracture process. Overall, specimen size has a considerable impact on the mechanical parameters such as peak load and fracture toughness, while its influence on fracture characteristics is negligible. Specimen geometry not only affects the fracture toughness but also influences the fracture mode and the predictive accuracy of theoretical criteria. The research findings have important scientific value for establishing mixed-mode I/II fracture testing standards and disaster prevention of cracked rock masses engineering.</p> <b>Highlights</b> <p><UnorderedList Mark="Bullet"> <ItemContent> <p>The size and geometry effects in rock mixed-mode I/II fracture have been thoroughly investigated.</p> </ItemContent> <ItemContent> <p>The size effect primarily influences the mechanical parameters, while the geometric configuration governs both mechanical responses and fracture paths.</p> </ItemContent> <ItemContent> <p>The GMTS criterion is most accurate for diametrically loaded specimens, while the GMTSN criterion performs best for three-point bending specimens.</p> </ItemContent> </UnorderedList></p>