Rice is a popular staple diet in India, and its demand has recently increased. Thanjavur, located in the Cauvery Delta region, is known as the rice granary of South India. Due to recent technological advancements, digital farming and globalization have significantly impacted the agricultural industry. It is crucial to differentiate between types of rice grains to prevent fraudulent labeling during import and export. To achieve this, a dataset, namely “TaPaSe Dataset”, comprising five varieties of rice, including MTU 1010, MTU 1290, Narmadha, Pacha Ponni, and Sonna Masur, which are mainly cultivated in Thanjavur, has been collected. We designed an image acquisition system to capture the aforementioned varieties in real time. The captured paddy rice images are highly challenging in the sense that all the images are captured under illumination and scale variations. We evaluated existing deep learning models to understand their ability to classify paddy seed varieties. The existing pre-trained models attain remarkable recognition rates on the proposed paddy seed varieties dataset.