Interpretability is crucial in natural language processing to enhance transparency and trust. Rationalization models achieve this by extracting key input fragments, i.e., rationales, to explain decisions while preserving predictive performance. On the other side, Federated Learning (FL) is recently emerging as a key paradigm for training machine learning models because it can leverage training data from multiple clients without the requirement of uploading their original data. Considering this, we firstly propose training Rationalization models in a FL manner. However, we find that simply combining them suffers from serious performance degradation due to the data heterogeneity among clients, where there exists inconsistent rationale generation. To solve this issue, we propose FedRNL which introduces a soft-sharing mechanism to align generator and predictor encoders, ensuring shallow-consistency and deep-generalization. An encoder loss minimizes feature discrepancies, and a layer-wise aggregation strategy separately updates the generator and predictor at the server, enhancing model stability. Extensive experiments show that FedRNL significantly improves the performance as compared to existing general heterogeneity mitigation methods.