Image-based object detectors are increasingly being used in surveillance and autonomous driving systems in real-time. However, those systems are threatened by latency attacks which inflate the elapsed time of each query, such that the system cannot respond properly within a reasonable time interval. In this paper, we find the root cause of the vulnerability of latency attacks on object detectors is caused by the occurrences of minor page faults. We propose a decision algorithm to mitigate this problem. The decision algorithm can automatically decide the optimal implementation to be executed based on the current status of the target system. To the best of our knowledge, this is the first paper to solve the latency issue from the point of view of computing architecture. Our studies provide a useful guideline for designing real-time applications on edge devices with more efficient and resilient responses.