Preliminary Results of LLM Vulnerability Testing in Less Common Languages
摘要
This study probes vulnerabilities of leading foundation models using a mixed manual/semi-automated evaluation via a custom agentic app. Two OpenAI-API agents run in tandem: a task-oriented responder, whose goal is to maximize query completion, and a safety sentinel focused on detecting jailbreaking and filtering out disallowed content. The authors deployed a 20-step “bank-heist” jailbreak script previously published in English and ran the suite in French, Japanese, Hebrew, Arabic, and Haitian Creole to assess the cross-lingual robustness of guardrails. Metrics include response latency and an experimental Adversarial Response Scoring System (ARSS). For analysis, all prompts and responses were visualized in a shared vector space to trace safe requests, guardrail rejections, and jailbreaking causes. Experiments show high variation between languages and models, and that no model was really jailbreak-proof. This work shows the need for more robust guardrails and shutdown to prevent attackers from exploiting AI models.