A central task in online learning is to design and test learning experiences that blend good pedagogy with elegant user interaction. In this work, we show that LLM-based web agents provide an efficient and helpful way to get immediate feedback on a learning design. We create a web agent that engages with an online learning experience much like a student would – autonomously navigating the interface and analyzing the website content – and generates a comprehensive description of the student experience. We validate the agent’s ability to understand student behavior by showing that it is useful in predicting student outcomes in a massive, open-access, online CS1 course. Specifically, agent-generated descriptions significantly improve our ability to predict student dropout in a lesson, outperforming all other measured approaches. We also demonstrate how designers could use the agent to compare potential lesson designs through case studies. Our qualitative analysis confirms that the agent can be used to identify confusing lesson content and provide actionable feedback for the designer. Such AI web agents have the potential to accelerate meaningful advancements in online learning by giving fast, low-cost, and helpful insights for new learning experiences.