This paper investigates the role of an intelligent adversary in derailing the advantages offered by machine and deep learning algorithms. The first quantization matrix estimation (FQME) forensic research problem in Double JPEG (DJPEG) compressed images is chosen as an example case study to demonstrate the vulnerabilities of the machine and deep learning algorithms that an intelligent adversary can exploit. DJPEG compression involves two compression cycles: the first compression when the image is initially saved as JPEG and the second compression when a forger manipulates the image and again saves it in the JPEG format. In such cases, the information regarding the first compression is lost in the presence of the second JPEG compression. Specifically, the quantization coefficients are of interest since estimating the quantization coefficients for the first compression, often referred to as the primary quantization estimation, can give information about the history of the image and the possibility of forgery/tampering. Various methods exist for estimating the first quantization coefficients, both statistical and deep learning-based. However, existing works do not evaluate the robustness of these estimation models against adversarial attacks, which is an essential criterion from a security point of view. In this work, a comprehensive adversarial analysis is carried out to show the vulnerabilities of machine and deep learning models for the FQME forensic research problem. Such a detailed security analysis is the need of the hour, and this paper throws light on it from a forensic perspective.