In the current NLP research status, there are active studies that have attempted to improve performance through fine-tuning or scaling up by suggesting various PLMs. However, it is difficult to find research analyzing which architecture features of PLMs are more suitable for the purpose of specific NLU or NLG tasks. Our primary goal in this study was to see the relationship between the architecture features of PLM inference text on the performance of NLG to deploy experiments via Korean BART, Korean GPT, and ChatGPT. Results were confirmed in our experiments. First, BART, equipped with both an encoder and a decoder, outperformed GPT, which only features a decoder in all documents. This underscores the significance of NLG capabilities and the encoder’s NLU functionalities in the transformer architecture. Furthermore, it highlights the importance of selecting a transformer architecture tailored to the specific downstream task over merely focusing on the size of PLM parameters. Moreover, Korean BART’s performance was associated with document features rather than document length, unlike Korean GPT and ChatGPT. Therefore, languages with agglutinative characteristics, like Korean or Japanese, may pose challenges even when employing BART. Hence, careful consideration of these linguistic characteristics could substantially enhance natural language generation performance. Future research may propose enhancements to the architecture of the previously disclosed transformer PLM to better accommodate the specific language’s characteristics.