<p>Accurately describing the variation in rock strength within brittle–ductile domain is of significant importance for deep underground engineering. In this study, a novel three-dimensional strength criterion based on parameterization method is proposed, encompassing the criteria within brittle and ductile domains. The criterion within brittle domain is expressed as the multiplication of Hoek–Brown criterion under triaxial compression and a generalized brittle deviatoric function, while the criterion within ductile domain is expressed as the multiplication of Drucker–Prager cap model under triaxial compression and a generalized ductile deviatoric function. Both generalized brittle and ductile deviatoric functions are based on Matsuoka–Nakai criterion, which is controlled by deviatoric parameters. Besides, deviatoric parameters are the function of hydrostatic pressure. The brittle failure envelope intersects with ductile failure envelope. Therefore, only the portion closer to the hydrostatic pressure axis is selected. The strength data of Bentheim sandstone, green sandstone, and Sorcy limestone are used to validate the proposed strength criterion. Regardless of the rock type, the hydrostatic pressure at brittle–ductile transition increases with increasing Lode angle, indicating a positive correlation between brittleness of rock and Lode angle. During brittle–ductile transition, the deviatoric plane is jointly controlled by brittle and ductile deviatoric functions. Under low Lode angle conditions, it is controlled by the ductile deviatoric function, while under high Lode angle conditions, it is controlled by the brittle deviatoric function. Finally, a smoothing method is applied at the interface of brittle and ductile failure envelopes to form a smooth transition zone, which is regarded as the critical state.</p>