Emojis are increasingly designed into surveys to intuitively capture users’ emotions and opinions on products and services. Survey design and layout also significantly shape the user experience (UX) of survey takers. However, significant concerns exist regarding emoji misinterpretation, influenced by factors such as age, gender, culture, and operating systems. Despite the growing use of emojis, limited research addresses their effectiveness within bipolar differential rating scales, a common tool in survey design. This paper addresses these concerns by using a within-subjects design to compare three versions of an emoji-based bipolar differential rating scale. Data is collected through surveys to assess preferences, eye tracking to measure attention, and interviews for detailed feedback. The analysis of both subjective and objective data resulted in nine actionable design recommendations for survey designers, aimed at optimizing the UX of emoji-based surveys and reducing potential miscommunication between designers and respondents.