NormSoftmax Attention: Improving Transformer Model Performance
摘要
This paper addresses the indoor positioning problem with the objective of enhancing localization accuracy using data from WiFi signals. The performance of the positioning system is highly sensitive to fluctuations in WiFi signal strength, which can negatively impact accuracy. To address this challenge, we propose a transformer-based algorithm, leveraging its ability to effectively model complex dependencies in data and mitigate the impact of signal fluctuations. Attention weights play a crucial role in enabling the transformer model to focus on relevant features; however, traditional softmax-based attention mechanisms often lack the flexibility needed to adapt to variable signal environments. By exploiting the unique characteristics of WiFi fingerprint data, we investigate Dirichlet distribution-based self-attention to replace the original mechanism for improved performance. Incorporating the Dirichlet distribution allows for greater control over attention scores, enabling the model to assign more robust, context-dependent weights to each token. This customization dynamically adjusts the attention focus on various signal features, even amidst fluctuating WiFi signals, thereby enhancing the model’s adaptability and overall positioning accuracy. Experimental results illustrate the efficiency of the proposed methodology, which outperforms existing approaches and achieves an accuracy improvement of up to 64%.