Federated Deep Neural Network (FedDNN) enables collaborative DNN model training without revealing data directly. As a high-performance FedDNN model is costly in training and profitable in marketing, its ownership may be abused during distribution and deployment. Most ownership verification methods for FedDNN models are inspired by watermarking, however, to verify ownership via the central server with ambiguity and vulnerability to network modifications. Therefore, these methods fail to reliably identify client contribution, which is crucial to source authorization and fair distribution in commercial practice. In this paper, we propose the FedTP, a novel Traceable Passport-based ownership verification scheme for FedDNN models, where each client independently embeds its own passport, instead of relying on the server for watermark assignment. The key insight of FedTP is to train a FedDNN model with passport-aware branches, derived from client identity, such that the performance will significantly decline if using forged passports. This enables fine-grained, client-level ownership verification with strong traceability and robustness. Extensive experimental results justify the effectiveness of our approach for FedDNN model ownership protection.