<p>Smart agriculture brings massive amounts of real-time images generated via modern information and communication technology. Promptly providing accurate estimates of fruit/vegetable information, such as location, quantity, and size, is worth studying. Therefore, we focus on exploring a deep learning-based backbone model for heatmap regression to capture the yield information. This singular and lightweight architecture effectively addresses the unified challenge of object counting, location detection, and size estimation for fruits/vegetables. However, when dealing with real-world applications, the data distribution shift would happen in response to the collection of new data. Moreover, some unseen fruits/vegetables often appear during the training process. All of these give rise to the open set recognition (OSR) problem. In such an OSR environment, a test-time domain adaptation approach based on deep learning is proposed for multi-class object localization and size estimation. This is the first attempt at unsupervised domain adaptation for heatmap regression tasks. Furthermore, to overcome the drawback of lacking a public dataset, a new benchmark dataset (including synthetic and real image data) has been created and collected to train, test, and evaluate our approach. Extensive experimental evaluations prove that our approach can achieve accurate predictions in the OSR setting within a single epoch of test-time optimization without altering the training process.</p>