<p>A new optical single-channel security system based on Schur decomposition, biometric random phase encoding in fractional Fourier transform domains is proposed. In this technique, the grayscale iris image is decomposed by Schur decomposition. Unitary matrix<i> U</i> and upper triangular matrix <i>T</i> are encoded and modulated independently by a random phase mask to produce a biometric random phase mask (BRPM). An original color image is split into <i>R</i>-, <i>G-</i> and <i>B-</i> channels. Each channel is decomposed by Schur decomposition. The unitary matrix<i> U.</i> of each channel is combined to construct a single-channel image as an input image. The upper triangular matrix <i>T</i> of each channel is utilized as decryption key. A single-channel image is modulated with <i>U-</i> BRPM and then fractional Fourier transformed. The fractional Fourier spectrum is amplitude- and phase-truncated to obtain first encrypted image and first decryption key. The first encrypted image is modulated with <i>T-</i> BRPM and then fractional Fourier transformed. The fractional Fourier spectrum is amplitude- and phase-truncated to get final encrypted image and second decryption key. The proposed cryptosystem has three main advantages. First, the proposed system has four decryption keys to evade potential attacks. Second, <i>U-</i> BRPM and <i>T-</i> BRPM are utilized as unique encryption keys to resist unauthorized access. Finally, the fractional orders of the fractional Fourier transform provide sensitive encryption keys. The proposed method can be executed using a hybrid optoelectronic system. The viability, efficiency, and security of the proposed system can be shown by numerical simulation results.</p>