Deep Learning and Transformers
摘要
This chapter provides a high-level overview of Deep Learning and Transformers. It offers a new perspective on the narratological and linguistic significance of specific techniques. In making these concepts accessible to students and readers from fields outside computer science, the goal is to keep things as simple and intuitive as possible. The chapter starts with Simple Linear Perceptrons. After pointing out their well-known limitations, it turns to Feed Forward Neural Nets, taking the reader on a journey to build a simple net called Tiny Tot. Following a discussion of how to train a net, it explains how the power of the Distributional Semantics introduced in Chap. 1 is brought to bear in numeric representations for word meanings; these are called Word Embeddings. Next, the chapter discusses how rich models of linguistic context are used by Attention mechanisms that allow input words to be influenced by words both nearby and further away. This leads to a discussion of Transformers, the neural net architecture used by state-of-the-art systems like GPT. After explaining how, given a prompt, words are predicted using the Transformer, the chapter describes some practical extensions involving Fine-Tuning and Retrieval-Augmented Generation (RAG), along with some observations on the art of Prompt Engineering.