Rubble stone monuments, common in Mediterranean countries, are particularly vulnerable to seismic activity due to their construction techniques and materials. These structures typically consist of irregular, uncut stones loosely bound with weak mortar, resulting in low tensile strength and poor cohesion. In addition, these historical structures are usually built as irregular aggregates, which contribute to their earthquake vulnerability. Due to their cultural importance, in addition to safety and economic reasons, it is of utmost importance to perform accurate seismic assessments of historical masonry structures to preserve them. However, in most cases, it is challenging for engineers to consider the interactions between different structural units in the aggregate, especially when it is difficult to define these units themselves. This paper focuses on the seismic behavior of aggregates, using the National Palace of Sintra—an emblematic example of an irregular, large-scale rubble stone monument built without prior planning—as a case study. It provides an overview of the research conducted on the Palace, including historical analysis and an experimental campaign developed to assess its seismic performance. The numerical modeling of this complex structure is also addressed, with particular attention to the interaction between structural units and the challenges of defining an equivalent-frame model. Given the limited number of input parameters required, this modeling approach was selected for its simplicity and widespread use among practitioners.