In the modern security environment, detecting document forgery is an essential part of any strong identity verification system, and in our case, we are dealing with Indian PAN (Permanent Account Number) cards. The traditional method of forgery detection relies on deep learning models built on the strength of centralized data collection and in doing so puts user privacy at risk. To prevent this trade off, we propose a system that is federated and is competent of detecting PAN card forgery without breaching privacy. We suggest a federated hybrid system that includes Convolutional Neural Network (CNN) based on MobileNetV2, plus Tesseract Optical Character Recognition (OCR) pipeline. Each client with its own data trains a local model. In each round, only the new local model weights of clients are sent over a secure channel to a server in the middle, where they are combined using the Flower (FLWR) framework. A prediction only client/end user can then get updated global weights from the server to check for forgery in real time and then with Gradient weighted Class Activation Mapping (Grad-CAM) the localized tampered/forged areas are shown.We had three clients in our experiment, and each trained on their own private dataset. We then tested the final aggregated model on a test dataset. In this research, we propose a scalable technique that efficiently enables real time verification of PAN cards while respecting the privacy.