Questions in conversations can be highly informal, with the user intent becoming clear only with prior context. This is challenging when the conversation involves long-tail entities. Prior works addressed these issues with computationally expensive techniques; this work aims to improve efficiency while being competitive in effectiveness. To this end, we devise techniques to efficiently compute relatedness scores between questions and entity mentions in previous turns, and select a few mentions for question expansion. Answers are computed by a ranking pipeline, with candidate subsets of decreasing size as later stages incur higher cost. Experiments with two benchmarks show that our method outperforms all baselines on both efficiency and effectiveness.