<p>Given a linear map on the vector space of symmetric matrices, every fiber intersected with the set of positive semidefinite matrices is a spectrahedron. Using the notion of the fiber body we can build the average over all such fibers and thereby construct a compact, convex set, the fiber body. We show how to determine the dimensions of faces and normal cones to study the boundary structure of the fiber body. We use this to study the fiber body of Gram spectrahedra in the case of binary sextics and ternary quartics and find a large amount of structure on the fiber body. We prove that the fiber body in the case of binary sextics has exactly one face with a full-dimensional normal cone, whereas the Gram spectrahedron of a generic positive binary sextic has four such points. The fiber body in the case of ternary quartics has a drastically different boundary structure compared to Gram spectrahedra of ternary quartics.</p>