The research field of software protections—including code obfuscation and software tamperproofing—has introduced many new concepts and methodologies over the past three decades. However, while in other research fields (such as cryptography) the strength of new approaches can be accurately measured, evaluations in the field of software protections are often less methodologically sound due to several challenges. One major problem is the poor availability of samples. The majority of evaluations in the field of software protections is based on small, single-function toy programs or benchmarks borrowed from other domains and do not represent very well the variety of programs that are to be protected in practice. Furthermore, most measurements performed on these samples focus on evaluating the costs (i. e. negative effects on runtime performance) and not the strength of a protection. In this paper, we present OSAGE, a novel, open framework for creating and evaluating appropriate samples for software protections research. It combines a hand-crafted set of 141 programs including individual test cases as a foundation for sample generation, several different compilers and obfuscators, as well as a set of 16 static code analysis methods in a fully automated sample generation and analysis framework. OSAGE enables software protection researchers to easily conduct sound evaluations of their methodologies in regard to costs, protection strength, and stealth and helps compare the results with other protections. In a large-scale evaluation of the framework we demonstrate its practicality in software protections research as well as the reproducibility and comparability of research utilizing OSAGE.