We study special subvarieties, i.e., subvarieties containing a dense subset of CM points, of the moduli space \(A_5\) of principally polarized abelian varieties of dimension five, generically contained in the locus of intermediate Jacobians of cubic threefolds. The analogous question for Jacobians of curves is related to a conjecture of Coleman-Oort and has been studied by Shimura, Mostow, De Jong-Noot, Rohde, Moonen, Oort, Frediani, Ghigi and others. Adapting methods of Frediani, Ghigi and Penegini, we give a sufficient condition ensuring that the closure of the image of a family of smooth cubic threefolds with prescribed automorphisms via the period map is a special subvariety of \(A_5\) and classify the positive-dimensional families of cubic threefolds satisfying our condition. In particular, we discover two examples of positive-dimensional special subvarieties in the intermediate Jacobian locus that contain the intermediate Jacobian of the Klein cubic threefold.
