Chart Question Answering (CQA) is a complex task that requires fine-grained, cross-modal, multi-step reasoning. This requirement leads to an extensive and intricate reasoning space. However, traditional models, relying solely on the final answer as supervision, can struggle with such complexity. Recent advancements have highlighted that using intermediate reasoning steps as supervision can simplify the reasoning process and reduce the reasoning space. Alignment, identifying question-relevant data points such as bars, lines, and dots in the chart, is a critical intermediate step that builds direct connections between the question and the chart. In light of this, we propose a simple yet effective weakly supervised model that integrates the alignment into the learning process. To facilitate this process, we introduce the Alignment Targets Identification (ATI) algorithm, designed to effectively align the question with the chart. Furthermore, we present a model that employs alignment supervision via ATI in a weakly supervised manner. In experiments, we verify the effectiveness of our model using the PlotQA-V1, PlotQA-V2, and ChartQA datasets. Compared with the non-alignment baseline models, our model outperforms the state-of-the-art CRCT by 5.59% on PlotQA-V1 and 29.53% on PlotQA-V2. Additionally, our model achieves competitive performance on ChartQA. Moreover, compared with some pre-trained CQA models with over one billion parameters, our model, with just a few million parameters, achieves significant advantages on the PlotQA-V1 and PlotQA-V2 datasets.