We consider the A series and exceptional E6 Restricted Solid-On-Solid lattice models as prototypical examples of the critical Yang-Baxter integrable two-dimensional A-D-E lattice models. We focus on type I theories which are characterized by the existence of an extended chiral symmetry in the continuum scaling limit. Starting with the commuting family of column transfer matrices on the torus, we build matrix representations of the Ocneanu graph fusion algebra as integrable seams for arbitrary finite-size lattices. In the A cases, the Ocneanu algebra coincides with the Verlinde algebra. In the other cases, the seam algebra contains the fused adjacency and graph fusion algebras as subalgebras. Our matrix representation of the Ocneanu algebra encapsulates the quantum symmetry of the commuting family of transfermatrices. In the continuumscaling limit, the integrable seams realize the topological defects of the associated conformal field theory and the known toric matrices encode the twisted conformal partition functions of Petkova and Zuber.