Detection reliability test is the basis of detection reliability engineering analysis method. The probability of detection curves based on the test directly reflects the influence of structure, process, environment and personnel. The design of defect samples is the key element of reliability test. In this paper, the rivet structure most used in aircraft is taken as the object, the porous thin plate sample is designed, and the natural crack formed by the fatigue test is to simulate the crack at the rivet hole edge. The crack length is interpreted by the optical microscope, and the distribution characteristics and distribution rules of cracks are developed from aspects such as crack length distribution, the relationship between the longest length crack and the hole number, the difference in crack length between front and back cracks, and the difference in length between left and the right cracks. The results show that the crack length distribution of the detection reliability flaw sample meets the requirement of eddy current detection probability coverage and is independent, which is suitable for eddy current reliability test.