<p>Transformer architectures dominate contemporary machine learning, yet face critical limitations in security applications: vulnerability to adversarial attacks, lack of calibrated uncertainty estimates, and difficulty distinguishing confident predictions from uncertain cases requiring human review. We introduce stochastic probably approximately correct (PAC) Bayesian transformers that convert deterministic attention into probabilistic variants via variational inference, unifying uncertainty quantification, and adversarial robustness within a single framework. Our approach replaces fixed attention weights with learned variational distributions and propagates uncertainty through Monte Carlo (MC) sampling, creating moving targets that degrade adversarial effectiveness. We derive joint PAC-Bayesian bounds showing that parameter stochasticity improves both calibration and robustness, with complexity scaling as <InlineEquation ID="IEq1"> <EquationSource Format="TEX">$O\left( {\sqrt {KL\left( {\rho ||\pi } \right)/n} } \right)$</EquationSource> <EquationSource Format="MATHML"><math> <mrow> <mi>O</mi> <mfenced close=")" open="("> <msqrt> <mrow> <mi>K</mi> <mi>L</mi> <mfenced close=")" open="("> <mrow> <mi>ρ</mi> <mo stretchy="false">|</mo> <mo stretchy="false">|</mo> <mi>π</mi> </mrow> </mfenced> <mo stretchy="false">/</mo> <mi>n</mi> </mrow> </msqrt> </mfenced> </mrow> </math></EquationSource> </InlineEquation>, where KL denotes the Kullback–Leibler divergence between the learned posterior <InlineEquation ID="IEq2"> <EquationSource Format="TEX">$\rho$</EquationSource> <EquationSource Format="MATHML"><math> <mi>ρ</mi> </math></EquationSource> </InlineEquation> and prior <InlineEquation ID="IEq3"> <EquationSource Format="TEX">$\pi$</EquationSource> <EquationSource Format="MATHML"><math> <mi>π</mi> </math></EquationSource> </InlineEquation>, and <InlineEquation ID="IEq4"> <EquationSource Format="TEX">$n$</EquationSource> <EquationSource Format="MATHML"><math> <mi>n</mi> </math></EquationSource> </InlineEquation> is the sample size. Across network intrusion detection, toxic content detection, and fake news identification, we achieve <InlineEquation ID="IEq5"> <EquationSource Format="TEX">$96.8 \pm 0.8\%$</EquationSource> <EquationSource Format="MATHML"><math> <mrow> <mn>96.8</mn> <mo>±</mo> <mn>0.8</mn> <mo>%</mo> </mrow> </math></EquationSource> </InlineEquation> accuracy with the expected calibration error (ECE) of <InlineEquation ID="IEq6"> <EquationSource Format="TEX">$0.043 \pm 0.006$</EquationSource> <EquationSource Format="MATHML"><math> <mrow> <mn>0.043</mn> <mo>±</mo> <mn>0.006</mn> </mrow> </math></EquationSource> </InlineEquation>, and maintain <InlineEquation ID="IEq7"> <EquationSource Format="TEX">$88.3 \pm 1.5\%$</EquationSource> <EquationSource Format="MATHML"><math> <mrow> <mn>88.3</mn> <mo>±</mo> <mn>1.5</mn> <mo>%</mo> </mrow> </math></EquationSource> </InlineEquation> robust accuracy under multiple attack strategies. Active learning guided by uncertainty reduces labeling requirements by <InlineEquation ID="IEq8"> <EquationSource Format="TEX">$68{\text{\% }}$</EquationSource> <EquationSource Format="MATHML"><math> <mrow> <mn>68</mn> <mrow> <mtext>\%</mtext> <mspace width="0.333333em" /> </mrow> </mrow> </math></EquationSource> </InlineEquation>, reaching <InlineEquation ID="IEq9"> <EquationSource Format="TEX">$95{\text{\% }}$</EquationSource> <EquationSource Format="MATHML"><math> <mrow> <mn>95</mn> <mrow> <mtext>\%</mtext> <mspace width="0.333333em" /> </mrow> </mrow> </math></EquationSource> </InlineEquation> of full-data performance with only <InlineEquation ID="IEq10"> <EquationSource Format="TEX">$35{\text{\% }}$</EquationSource> <EquationSource Format="MATHML"><math> <mrow> <mn>35</mn> <mrow> <mtext>\%</mtext> <mspace width="0.333333em" /> </mrow> </mrow> </math></EquationSource> </InlineEquation> of labels.</p>

Stochastic PAC-Bayesian transformers for network intrusion detection and natural language processing applications

期刊论文
发表日期：

2026年8月3日

查看全文

Roger Nick Anaedevha,
Alexander Gennadievich Trofimov,
Yuri Vladimirovich Borodachev

摘要

Transformer architectures dominate contemporary machine learning, yet face critical limitations in security applications: vulnerability to adversarial attacks, lack of calibrated uncertainty estimates, and difficulty distinguishing confident predictions from uncertain cases requiring human review. We introduce stochastic probably approximately correct (PAC) Bayesian transformers that convert deterministic attention into probabilistic variants via variational inference, unifying uncertainty quantification, and adversarial robustness within a single framework. Our approach replaces fixed attention weights with learned variational distributions and propagates uncertainty through Monte Carlo (MC) sampling, creating moving targets that degrade adversarial effectiveness. We derive joint PAC-Bayesian bounds showing that parameter stochasticity improves both calibration and robustness, with complexity scaling as $O\left( {\sqrt {KL\left( {\rho ||\pi } \right)/n} } \right)$ $O (\sqrt{K L (ρ | | π) / n})$ , where KL denotes the Kullback–Leibler divergence between the learned posterior $\rho$ $ρ$ and prior $\pi$ $π$ , and $n$ $n$ is the sample size. Across network intrusion detection, toxic content detection, and fake news identification, we achieve $96.8 \pm 0.8\%$ $96.8 \pm 0.8 %$ accuracy with the expected calibration error (ECE) of $0.043 \pm 0.006$ $0.043 \pm 0.006$ , and maintain $88.3 \pm 1.5\%$ $88.3 \pm 1.5 %$ robust accuracy under multiple attack strategies. Active learning guided by uncertainty reduces labeling requirements by $68{\text{\% }}$ $68 \%$ , reaching $95{\text{\% }}$ $95 \%$ of full-data performance with only $35{\text{\% }}$ $35 \%$ of labels.