There exist various approaches for the 3D reconstruction of dynamic scenes. In medicine, particularly in endoscopy, single-shot structured light systems are frequently explored, as they allow for the reconstruction of dynamic, feature-less surfaces. Design and manufacturing of structured light endoscopes, however, implies high initial costs that significantly hinder the availability and development of these systems. To streamline this process, simulation systems are necessary that allow researchers to not only model the intricacies of medical domains, but also of structured light systems themselves. To address this, we propose Fireflies, a differentiable framework for the physically-based simulation and domain randomization of structured light endoscopy. Based on the differentiable Mitsuba renderer, Fireflies facilitates and simplifies the development of domain-specific algorithms for endoscopic procedures. In this paper, we demonstrate the effectiveness of our framework by jointly optimizing domain-specific laser-based projection pattern for Structured Light Endoscopy, and generating large-scale synthetic training data for efficient supervised learning without manual labeling. We show that a) an optimized projection pattern can increase the reconstructability of a target domain and b) the synthetic data generated by Fireflies lowers the labeling effort required for endoscopic machine learning tasks. The source code is available at: https://github.com/Henningson/Fireflies