Scholarlink

<p>Privacy inference (PI) based on cryptographic protocols can effectively protect the privacy of models and user data. However, the nonlinear activation functions in neural networks, such as ReLU, contribute to most of the computational and communication overhead in PI, limiting its widespread adoption. Although existing research has reduced ReLU operations by introducing hyperparameters or redesigning network architectures, these methods typically require retraining the network model, which brings potential retraining cost issues. Especially in the era of large models, the high training costs are even more significant. To address the high latency of ReLU in privacy inference, this paper introduces a low-cost ReLU search space compression, replacement, and restoration framework (LC-SCRR), which aims to improve inference efficiency with minimal retraining overhead. Firstly, we analyze the correlation between the imbalance in the distribution of ReLU importance and the reduction in overhead costs. Based on this analysis, we employ a structured pruning method to prune ReLUs with high proportions but low importance, thereby significantly compressing the search space for subsequent unstructured selection and replacement. Secondly, within the compressed space, we adopt a gradient-based selection and replacement strategy, significantly reducing the overhead associated with retraining. Lastly, we restore some ReLUs from the compressed space to further enhance model accuracy at a reduced cost. Extensive experiments conducted on various models and datasets have demonstrated that LC-SCRR reduces retraining costs by approximately 25% and outperforms DReP (JSA 24) and AutoReP (ICCV 23) on the CIFAR-100 dataset.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Low-cost ReLU search space compression, replacement, and recovery for efficient private network inference

  • 期刊论文

  • 发表日期：

    2025年5月28日

查看全文
  • Jingwen Wang,
  • Peng Hu,
  • Lei Sun,
  • Cuiyun Hu,
  • Xiuqing Mao,
  • Zhihong Wang

摘要

Privacy inference (PI) based on cryptographic protocols can effectively protect the privacy of models and user data. However, the nonlinear activation functions in neural networks, such as ReLU, contribute to most of the computational and communication overhead in PI, limiting its widespread adoption. Although existing research has reduced ReLU operations by introducing hyperparameters or redesigning network architectures, these methods typically require retraining the network model, which brings potential retraining cost issues. Especially in the era of large models, the high training costs are even more significant. To address the high latency of ReLU in privacy inference, this paper introduces a low-cost ReLU search space compression, replacement, and restoration framework (LC-SCRR), which aims to improve inference efficiency with minimal retraining overhead. Firstly, we analyze the correlation between the imbalance in the distribution of ReLU importance and the reduction in overhead costs. Based on this analysis, we employ a structured pruning method to prune ReLUs with high proportions but low importance, thereby significantly compressing the search space for subsequent unstructured selection and replacement. Secondly, within the compressed space, we adopt a gradient-based selection and replacement strategy, significantly reducing the overhead associated with retraining. Lastly, we restore some ReLUs from the compressed space to further enhance model accuracy at a reduced cost. Extensive experiments conducted on various models and datasets have demonstrated that LC-SCRR reduces retraining costs by approximately 25% and outperforms DReP (JSA 24) and AutoReP (ICCV 23) on the CIFAR-100 dataset.