Vision Transformers (ViTs) are the current state-of-the-art in deep learning for computer vision tasks. They are trained on vast datasets and are capable of useful downstream tasks through clever use of the attention mechanism. The biggest limiting factor for ViTs is the number of pixels and tokens that can be processed in a given pass. Memory constraints on both patch size and the number of patches mean that ViTs are most effective at processing relatively low-resolution images. Whilst ViTs can attend very flexibly across an image, attending across images in a naive fashion requires memory proportional to the square of the number of images. This is a further limiting factor. Given the task of automated assessment of psoriasis severity, a chronic skin condition that can affect large portions of a person’s skin, it is necessary to look across multiple images and at fine detail in large images. We present a method that adapts ViTs to a two-stage design that allows for the regression of a patient’s psoriasis score across multiple images and resolutions and shows its effectiveness relative to a baseline ViT. The implementation of our method is available at https://github.com/KCL-BMEIS/multivit.git .

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

A Vision Transformer with Adaptive Cross-Image and Cross-Resolution Attention

  • Benjamin A. K. Murray,
  • Wei R. Tan,
  • Liane S. Canas,
  • Catherine H. Smith,
  • Satveer K. Mahil,
  • Sebastien Ourselin,
  • Marc Modat

摘要

Vision Transformers (ViTs) are the current state-of-the-art in deep learning for computer vision tasks. They are trained on vast datasets and are capable of useful downstream tasks through clever use of the attention mechanism. The biggest limiting factor for ViTs is the number of pixels and tokens that can be processed in a given pass. Memory constraints on both patch size and the number of patches mean that ViTs are most effective at processing relatively low-resolution images. Whilst ViTs can attend very flexibly across an image, attending across images in a naive fashion requires memory proportional to the square of the number of images. This is a further limiting factor. Given the task of automated assessment of psoriasis severity, a chronic skin condition that can affect large portions of a person’s skin, it is necessary to look across multiple images and at fine detail in large images. We present a method that adapts ViTs to a two-stage design that allows for the regression of a patient’s psoriasis score across multiple images and resolutions and shows its effectiveness relative to a baseline ViT. The implementation of our method is available at https://github.com/KCL-BMEIS/multivit.git .