Multimodal Aspect-based Sentiment Analysis (MABSA) identifies aspect terms and their sentiment polarities by fusing textual and visual signals. MABSA hinges on cross-aspect interaction for proper sentiment reasoning and fine-grained integration of textual aspects with visual attributes. However, two key challenges remain unsolved: complications in modeling cross-aspect interactions caused by heterogeneous cross-modal dependencies, and the difficulty of integrating aspect-related visual semantics with textual aspects due to ambiguous aspect-visual associations. To address these challenges, we propose a framework, Aspect-Oriented Prompt with Adaptive Cross-Modal Fusion (APAC), which includes: (1) The Aspect-Oriented Compositional Prompting (ACP) module, extracting visual components, like attributes, sentiment cues, and relationships, from the visual and textual modalities to synthesize them into structured descriptions to capture cross-aspect interactions; (2) The Adaptive Cross-Modal Fusion (ACF) module, leveraging Cross-Attention and multimodal factorized bilinear pooling (MFB) for adaptive and efficient integration of aspect-related cross-modal semantics. Experiments on Twitter2015 and Twitter2017 show APAC achieves competitive performance, demonstrating its effectiveness in resolving cross-modal ambiguities and dependencies.