We introduce a method for recognizing conversation activity in a group of people walking outdoors using a color video sequence acquired from a camera. Many methods have been developed to recognize whether people are walking together or talking together in a color video sequence. However, a method has yet to be proposed to recognize conversation activity in a pedestrian group walking outdoors. In this paper, we design a feature extraction approach for conversation activity recognition using physical body interactions caused by pedestrians’ conversations. Our method generates an interaction video sequence in a virtual space using a temporal posture signal and a temporal walking position signal that represent pedestrians’ body interactions. Our method uses the interaction video sequence as an informative and visible feature to determine a conversation activity label. The experimental results showed that our interaction video sequence recognized conversation activity more accurately than alternative techniques that use the appearance of the body regions of a pedestrian group or time-series changes of the posture and walking position among pedestrians.