Developing and executing field tests for autonomous vehicles is a time-consuming and expensive process, where only a limited number of tests can be performed. Therefore, it is important that each test provides the maximum amount of information to the system evaluator. We propose creating these testing suites by using a simulation-based framework to generate adversarial scenarios which are tailored to the autonomy under test. The challenge in the past is that test case generationtest case generation methods have been limited to a low number of dimensions and simple scenarios. Additionally, they are typically only verified in a simulation environment, and the difficulties of transfer from simulation to reality have not been explored. We address these issues by applying adaptive sampling adaptive sampling to efficiently explore a large scenario parameter space with multiple competing scoring criteria that represent a realistic unmanned underwater vehicle mission. The result is a diverse set of test cases that represent the performance boundaries of the autonomy software. Finally, we present the results of executing simulation generated tests in a live test range, which at the time of execution was the first field test of its kind.