This paper investigates modifications to traditional embedding representations in large language models (LLMs) through two approaches: the hashing trick and one-hot encoding. We conducted experiments on three families of models GPT-2, facebook/opt-1.3b and Microsoft/phi-4-mini evaluating three configurations: original embeddings, hash-based embeddings, and one-hot embeddings. Our evaluation considers metrics such as evaluation loss, perplexity, and training time, using the Wikitext/wikitext-2-raw-v1 dataset. The results indicate that, while the original embeddings yield the best performance in terms of predictive accuracy and training efficiency, the modified embeddings offer potential scalability benefits. Notably, the hash-based approach outperforms one-hot encoding by a small margin, albeit at a higher computational cost. We conclude by discussing the trade-offs inherent in these methods and propose directions for future work to optimize the balance between efficiency and accuracy.