Assistive sign language translation systems have advanced with the emergence of artificial intelligence. However, many of these systems do not adequately consider the interaction needs and user experience of deaf individuals during their development. This paper describes an interdisciplinary project focused on the healthcare setting, which includes the development of a bidirectional translator between Brazilian Sign Language (Libras) and Portuguese, using a photorealistic avatar capable of performing manual signs and facial expressions. In this context, it explores the HCI challenges faced in developing such system to aid deaf patients during pre-medical care, as well as the strategies adopted to deal with them. The paper contributes to researchers and developers interested in automatic translation between sign and spoken languages.