This study focuses on optimizing the environmental reaction conditions for converting waste cooking oil into biodiesel using methanol and sea sand as catalysts, aiming to minimize environmental impact. The process, conducted in a batch reactor, was optimized using response surface methodology (RSM) and central composite rotatable design (CCRD). Variables like calcinated temperature (800–1000 °C), MeOH/Oil molar ratio (13:1–17:1), catalyst amount (5–15 wt.%), and conversion (ξ), biodiesel yield (Y), and environmental factor (E-factor) were set as responses. Multiple regression analysis and verification experiments confirmed the validity of the predicted models. Results from the multi-objective optimization study showed that optimal calcination temperature, MeOH/Oil molar ratio, and catalyst amount were 945.44 °C, 14.76, and 8.69 wt.%, respectively, within a reaction time of 4 h. The maximum conversion, yield, and minimum environmental factor under optimal operating reaction conditions were 95.59%, 86.14%, and 1.11, respectively. Also, the responses ξ, Y, and E-factor were modeled by a third-order polynomial, second-order polynomial, and third-polynomial models, with correlation coefficient (R2) of 0.9851, 0.9418, and 0.8084 for ξ, Y, and E-factor, respectively. Furthermore, the adequate precision ratios for the three responses (ξ, Y, and E-factor) were 12.4894, 12.0084, and 10.9326, respectively. Thus, the polynomial models fitted are precise and reliable.