This study proposes learning methods that work with limited resources as a model of the learning behavior of the fly brain, namely, the mushroom body. Recent research on the fly’s mushroom body has shown that some of its output neurons (MBONs) are activated by unknown odors. However, these effects were quickly suppressed by repeated exposure to the same odor. It appears that such MBON behaviors reflect learning of odors. We were interested in how flies could continue to learn about odors throughout their lives with their small brains. It has been suggested that learning about new odors can help the fly to forget its existing memories. Considering this, we hypothesized that the main reason for continual learning was that it serves as a strategy to forget. To test the validity of this hypothesis, we created three models using kernel perceptron. This is suitable for estimating the ongoing learning capacity within a budget. Through computer simulation and theoretical analysis, the model demonstrated the importance of having a forgetting mechanism for two reasons. One is prepare for the next new learning, and the other, is to reduce the negative effects of deleting memories.