Personal Digital Assistants have gained a great momentum and many innovations are happening in this field in recent years. They are now being widely used in the form of home-based smart speakers such as Amazon’s Alexa, Google’s Home and Apple’s Homepod Mini etc. As voice assistants are becoming popular in India, people are discovering different ways in which they can use them. For more and more people to be able to use them, it not only relies on people to learn and adapt to how they work but also the speakers to adapt to the multilingual Indian audience. More than half of the population of India is predominantly Hindi speaking, therefore it is necessary for the voice assistants to not only function in the language but also be able to understand what the user is expecting, feel natural while responding and adapt to the language. This paper presents the results of an evaluation of the two smart personal voice assistants that support Hindi i.e. Amazon’s Alexa and Google’s Assistant in the dimensions of response rate, success rate and how helpful or correct and natural the responses feel to the users. The paper also tries to explore what participants considered as a good answer when given by a machine assistant and the rationale behind modifying their command to the assistant when they could not get the desired response. Eight people participated in the experiment and the results show that Amazon’s Alexa had a better success and response rate than Google’s Assistant and the participants also rated Amazon Alexa’s responses slightly good in terms of it’s way of responding and more helpful in terms of precision of response. But, Amazon’s Alexa had a significantly high interruption rate as the participants lost patience due to lengthy responses when compared to Google Assistant whose responses were short and crisp. The overall impact of this research is to serve a basis for finding gaps in the existing popular voice assistant devices and hence can be used as a guide towards a better experience of usage. Further studies could also help serve as a basis for understanding what native hindi speaking audience expects as a response and how differently they can frame the question even when the task given at hand is the same based on how they rated responses to be successful, failed or invalid.