Nowadays, the problem of natural language understanding models evaluation is an essential part of theoretical and practical concern. In this paper, we build ( https://github.com/yaroslav-i-am/paramsum ) the annotated dataset for quality of in-domain aspect extraction assessment. We propose to design this task in domain of movie reviews. We used a crowdsourcing method to build a marked-up dataset. We use fine-tuned large language models to build the ultimate set of annotated reviews. We ended with MoRAE dataset of 986 reviews with 8413 extracted aspects.