Automatic humor detection enables the computers to understand the jokes cracked by humans and thereby enables the computers to give appropriate responses through means like chatbots. Understanding the jokes in low-resource languages can make the environment more user-friendly since people can interact comfortably in their native languages. This paper proposes a methodology for humor detection in Telugu texts posted on the X platform. Various machine learning algorithms, namely logistic regression, decision tree, random forests, Naive Bayes, and SVM were used to detect humor after generating the tweet embeddings using Indic BERT, LaBSE, and MuRIL. In our study, we employed cost-sensitive learning to address the problem of class imbalance in the dataset. Out of all the models, the Random forest classifier trained on MuRIL's embeddings performed the best, with an accuracy of 81.9% and an F1 score of 0.80.