<p>Visual attention plays a critical role when our visual system executes active visual tasks by interacting with the physical scene. However, how to encode visual object relationships in the psychological world of the brain deserves exploration. Predicting visual fixations or scanpaths is a usual way to explore the visual attention and behaviors of human observers when viewing a scene. Most existing methods encode visual attention using individual fixations or scanpaths derived from raw gaze-shift data collected from human observers. This may not capture the common attention pattern well, because without considering the semantic information of the viewed scene, raw gaze shift data alone contain high inter- and intra-observer variability. To address this issue, we propose a new attention representation, called <i>visual attention graph (VAG)</i>, to simultaneously code the visual saliency and scanpath in a graph-based representation and better reveal the common attention behavior of human observers. In the visual attention graph, the semantic-based scanpath is defined by the path on the graph, while the saliency of objects can be obtained by computing fixation density on each node. Systemic experiments demonstrate that the proposed attention graph combined with our new evaluation metrics provides a better benchmark for evaluating attention prediction methods. Meanwhile, extra experiments demonstrate the promising potential of the proposed attention graph in assessing human cognitive states, such as autism spectrum disorder screening and age classification.</p>