Artificial intelligence (AI) is increasingly being integrated into cancer care, prompting concerns about transparency, trust, equity, and clinical usability. Explainable AI (XAI) offers strategies to address these concerns, but its role in oncology remains underexamined from the perspective of those involved in delivering and receiving cancer care. In this study, we conducted a structured workshop with patients, family members, clinicians, researchers, and health system leaders to explore perceived benefits, concerns, and priorities related to AI in cancer care. Through analysis of 255 participant comments, we identified concerns around bias, accountability, clinical reasoning, and communication—many of which aligned with established characteristics of effective XAI, including iterability, faithfulness, completeness, and plausibility. Our findings suggest that even when not prompted about XAI, participants surfaced values and needs that explainability could help address. These results support the integration of explainability principles early in AI development to ensure tools are responsive to the complexity of oncology and the needs of both patients and clinicians.