<p>The screw-type grain unloading mechanism differs significantly from conventional gravity-based methods, particularly for high-fiber crops such as flax, whose threshing materials exhibit distinct physical and mechanical properties compared to traditional grains like wheat and rice. However, studies focusing on key unloading parameters and particle flow behavior for such materials remain limited. This study investigates the high-position grain unloading device of a flax combine harvester, integrating discrete element method (DEM) simulation, response surface methodology (RSM), and field experiments to systematically explore the interaction mechanisms between unloading parameters and flax threshing materials. From the perspective of particle kinematics and dynamics, the physical characteristics during unloading were revealed. Simulation results indicate that a lower discharge pipe inclination improves unloading efficiency, and higher screw conveyor speeds lead to increased fluctuations in material flow velocity and a higher coefficient of variation in velocity. Particle motion is dominated by translational kinetic energy, with a ring-like distribution inside the pipe; the periodic propulsion of the screw results in a radial cross-sectional structure characterized by “accumulation on one side and sparsity on the other.” An increased impurity rate significantly reduces unloading velocity. Response surface analysis indicates that, in decreasing order of influence, screw conveyor speed, pipe inclination angle, and impurity rate are the primary factors influencing unloading performance. The optimal working parameters were determined as: engine speed of 1835 r/min, impurity rate of 1%, and pipe inclination angle of -15°. The simulated mass flow rate of flax material showed good agreement with field test results, verifying the reliability of the simulation and providing a theoretical and practical reference for designing and optimizing grain unloading systems for flax and other high-fiber crops.</p>