Due to the unstructured nature of images, representing salient information present in images is challenging. Knowledge graphs are promising solutions for representing unstructured information. However, constructing a knowledge graph requires an ontology, which is a complex task due to the semantic gap (between object and their context-dependent relationship). Moreover, several visual datasets, such as MS-COCO, have vast domain coverage; however, accessing the attributes and information present in such datasets is limited to a single image only. To address this challenge, we have developed a domain-independent ontology (CaKnow) based on the MS-COCO dataset capable of representing information in visual datasets. The proposed ontology helps understand the dataset and connect the information present in images with each other using semantic links. In addition, CaKnow is also helpful in building the knowledge graph where the common objects and context can be linked to each other and semantics present in the image can be queried. The ontology presented in the paper defines a vocabulary for representing the various elements in the MS-COCO dataset along with other visual datasets, including the image attributes, object classes, relationships, and metadata. CaKnow ontology can be used to construct the knowledge graph of images present in the visual datasets where each image is added as an individual, providing a structured and comprehensive representation of the salient information present in an image and linking it with other images, resulting in a knowledge graph. Furthermore, the entities and relations of the knowledge can be connected to the Linked Open Data Cloud, becoming the Web of Data due to common and domain-independent context.