We study finite groups G having a subgroup H and \(D \subset G {{\setminus }} H\) such that (i) the multiset \(\{ xy^{-1}:x,y \in D\}\) has every element that is not in H occur the same number of times (such a D is called a relative difference set); (ii) \(G=D\cup D^{(-1)} \cup H \text{(disjoint } \text{ union) }\) ; (iii) \(D \cap D^{(-1)} =\emptyset \) . We show that adding an additional symmetry condition allows a classification of such difference sets when G is a generalized quaternion group; for the other dicyclic groups we provide a classification where there are certain exceptions. These results have number-theoretic consequences.