This chapter provides a high-level overview of Deep Learning and Transformers. It offers a new perspective on the narratological and linguistic significance of specific techniques. In making these concepts accessible to students and readers from fields outside computer science, the goal is to keep things as simple and intuitive as possible. The chapter starts with Simple Linear Perceptrons. After pointing out their well-known limitations, it turns to Feed Forward Neural Nets, taking the reader on a journey to build a simple net called Tiny Tot. Following a discussion of how to train a net, it explains how the power of the Distributional Semantics introduced in Chap.  1 is brought to bear in numeric representations for word meanings; these are called Word Embeddings. Next, the chapter discusses how rich models of linguistic context are used by Attention mechanisms that allow input words to be influenced by words both nearby and further away. This leads to a discussion of Transformers, the neural net architecture used by state-of-the-art systems like GPT. After explaining how, given a prompt, words are predicted using the Transformer, the chapter describes some practical extensions involving Fine-Tuning and Retrieval-Augmented Generation (RAG), along with some observations on the art of Prompt Engineering.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Deep Learning and Transformers

  • Inderjeet Mani

摘要

This chapter provides a high-level overview of Deep Learning and Transformers. It offers a new perspective on the narratological and linguistic significance of specific techniques. In making these concepts accessible to students and readers from fields outside computer science, the goal is to keep things as simple and intuitive as possible. The chapter starts with Simple Linear Perceptrons. After pointing out their well-known limitations, it turns to Feed Forward Neural Nets, taking the reader on a journey to build a simple net called Tiny Tot. Following a discussion of how to train a net, it explains how the power of the Distributional Semantics introduced in Chap.  1 is brought to bear in numeric representations for word meanings; these are called Word Embeddings. Next, the chapter discusses how rich models of linguistic context are used by Attention mechanisms that allow input words to be influenced by words both nearby and further away. This leads to a discussion of Transformers, the neural net architecture used by state-of-the-art systems like GPT. After explaining how, given a prompt, words are predicted using the Transformer, the chapter describes some practical extensions involving Fine-Tuning and Retrieval-Augmented Generation (RAG), along with some observations on the art of Prompt Engineering.