Nowadays many mobile users still prefer to download apps through third-party app stores instead of official stores due to regional restrictions, usage habits and other specific reasons, even though third-party stores are not subject to the same level of scrutiny, vetting, and oversight as the official stores. Although ranking algorithms of search engines can help users choose third-party app stores, most of them primarily consider the semantics of search keywords and external links, while overlooking the differences of store performance. In order to help users make better choices based on personal preferences, we present a comparative study of third-party mobile app store, to understand the differences of these stores in terms of data, functionality, and services. First, we propose an evaluation index system that describes the quality of third-party app stores from 5 main categories and 26 single indices. Then, we curate a large-scale dataset from 17 popular third-party app stores, and compare these stores from the perspective of 5 index categories. Our evaluation result shows that there exist significant differences in the quality of third-party app stores, and almost all stores have a series of issues. At last, we propose a quality evaluation model and develop a lightweight search engine AppSearch, which could recommend app stores for users based on their personalized concerns. The study also provide valuable insights and guidance for users, developers, and app store maintainers, to build a much better app store ecosystem.