Visual Question Answering is one of the essential parts of machine reasoning. Datasets are created to train a model to perform this task. However, there are only a few datasets for the Russian language. Moreover, existing sets may have strong biases, allowing models to score high without reasoning. In this paper, we adapt the idea of the English diagnostic dataset for compositional language and elementary visual reasoning, CLEVR, to Russian. We also evaluate multiple baselines and models on this dataset to see how well they perform. The results may be used to improve the performance of Russian multimodal LLMs.