The virtual try-on system can become a popular smart tool to offer remote, customized and enjoyable dressing experiences such as convenient e-commerce shopping. However, the current methodologies have limitations in accommodating a wide range of human postures, body types, and background fusion. This study proposes the Torso-3D-Fitting Virtual Try-On System (TF-VTON), designed for image-based virtual try-on. It ensures seamless 3D integration of diverse backgrounds and human postures, addressing existing limitations in the field. First, the pre-trained U2-Net performs the separation of both target individual and background, preserving the segmented background for subsequent use. Then, leveraging the simultaneous acquisition of human-extracted images, the 2D skeleton and the 3D exterior decomposability of the subject are inferred using the OpenPose and the DensePose-RCNN models, respectively. This step enhances the precision of garment-to-torso synthesis, enabling the system to accommodate a various spectrum of natural human postures. To further refining the realism of the try-on outcomes, our framework integrates a Mask2Former model to semantically segment the human torso and the virtual garments into appropriately delineated sections. The segmentation and the pose estimation networks serve as tailor-made processors for the TF-VTON module, where the system effortlessly achieves casual garment-to-torso try-on with customizable photo-realistic backgrounds. In essence, this study not only greatly enhances the functionality of virtual try-on system but also positions itself as a crucial foundation for future research in the field.