Obtaining Textual Data for Marketing Analytics: Web Scraping, APIs, and Structured Sources
摘要
This chapter reviews modalities for gathering text in R. It opens with approaches for dealing with electronic text, and it emphasizes that collection and preparation often take a significant amount of time. It also discusses digitization and optical character recognition (OCR) workflows (and their pitfalls) for printed and handwritten sources, with consideration of the legal and ethical aspects of collecting textual data. The chapter reviews the typical pathways for acquisition, which include self-generated content, copy-and-paste content, prepackaged downloads, web scraping, and curated datasets. It also discusses the use of several packages for reading in a variety of file formats while noting issues around encoding and security. It provides a worked example of downloading and reshaping PDFs and CSVs into tidy data frames. It also compares working with API access to web scraping via the rvest package, along with comparing the use of browser plug-ins, programmatic scraping schema, discovering text selectors via developer tools, and cleaning up scraped text fields. The chapter also covers dynamic web pages and using RSelenium to automate following links and pagination. It ends with structured sources, emerging tools for AI-assisted scraping, and repositories, as well as a set of exercises.