Large Language Models (LLMs) have evolved from transformer architecture, revolutionizing natural language processing and attracting interest from a wide range of domains, including finance. This survey outlines multiple strategies for optimizing these models to improve their adaptability in diverse scenarios. It provides a comprehensive overview of how general-domain LLMs are constructed and the methodologies for customizing these models for specific domains, emphasizing two primary approaches. Additionally, the research showcases examples of specialized LLMs that utilize these techniques to enhance understanding. Relevant evaluation metrics for various tasks are also examined. Moreover, we investigate real-world applications in finance implemented by prominent global organizations. Finally, we address the limitations of current LLMs in the finance sector and propose several strategies for overcoming these challenges, along with outlining our plans for future research.