The classic feature-based visual-inertial odometry system estimates camera poses by tracking features between two images. However, indoor robots generally encounter situations wherein two images have a small overlapping area and sparse features, such as corridors, white walls, and corners. In these cases, the feature tracking is more likely to fail. This reduces the accuracy of the algorithm. To address these drawbacks, this paper presents a visual-inertial odometry system based on the fisheye camera, which ensures a larger overlapping area between two images and improves the performance of feature tracking. To track features robustly in a distorted image collected by the fisheye camera, a feature tracking and outlier elimination strategy is introduced based on the consistency constraints. Meanwhile, an EKF-based on-line calibration method of camera internal and external parameters is proposed to improve the accuracy of the camera projection model. Experiments were conducted with the TUM-VI dataset and a mobile robot platform equipped with a fisheye camera. The results show that the algorithm can accurately estimate the internal and external parameters of the camera online and robustly estimate camera pose in indoor environments.