The exponential growth of digital content necessitates the development of efficient and effective semantic search engines, especially for regional languages like Marathi. This paper addresses the challenge of retrieving semantically similar news articles in Marathi by developing a robust semantic search engine using a fine-tuned Marathi SBERT model. It captured the subtleties of Marathi, in which the positive pairs and MultipleNegativesRankingLoss optimize semantic similarity between sentences. Techniques like UMAP for dimensionality reduction and a contrastive loss function for optimizing similarity improve the model’s performance, achieving a Silhouette Score of 0.6726 compared to 0.1211 without UMAP. The search engine has practical applications for regional e-newspapers, helping readers quickly find contextually relevant articles in Marathi. It also supports linguistic research by facilitating the analysis of language patterns in news content. The model integrated with a Gradio-based interface and deployed on Hugging Face Spaces, will ensure better access to information in the digital age by providing a solution for users seeking contextually relevant Marathi news articles. Challenges remain in scaling to larger datasets and diverse dialects. Future work could enhance the synonym dataset and refine query expansion techniques.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Fine-Tuning SBERT for Semantic Search in Marathi News with Synonym-Based Positive Pairs

  • Srushti Pednekar,
  • Deepak C. Vegda

摘要

The exponential growth of digital content necessitates the development of efficient and effective semantic search engines, especially for regional languages like Marathi. This paper addresses the challenge of retrieving semantically similar news articles in Marathi by developing a robust semantic search engine using a fine-tuned Marathi SBERT model. It captured the subtleties of Marathi, in which the positive pairs and MultipleNegativesRankingLoss optimize semantic similarity between sentences. Techniques like UMAP for dimensionality reduction and a contrastive loss function for optimizing similarity improve the model’s performance, achieving a Silhouette Score of 0.6726 compared to 0.1211 without UMAP. The search engine has practical applications for regional e-newspapers, helping readers quickly find contextually relevant articles in Marathi. It also supports linguistic research by facilitating the analysis of language patterns in news content. The model integrated with a Gradio-based interface and deployed on Hugging Face Spaces, will ensure better access to information in the digital age by providing a solution for users seeking contextually relevant Marathi news articles. Challenges remain in scaling to larger datasets and diverse dialects. Future work could enhance the synonym dataset and refine query expansion techniques.