Improvements in self- and semi-supervised machine learning (ML) afford an opportunity to train AI foundation models (FMs) for a variety of specialized sensor modalities. Because a foundation model’s ability to generalize across varied remote-sensing (RS) tasks depends on the data distribution it was trained on, different downstream applications require different levels of feature specificity in the model’s backbone. Therefore, users need a clear way to assess a model’s descriptive capacity so they can prioritize development and choose the right model for each task. Although the end goal of operationalizing an FM is likely to involve task-specific fine-tuning, as the corpus of available FMs for various sensors expands, it is not tractable to label fine-tuning sets for every possible combination of tasks and modalities; in fact, having sufficient volumes of labeled data to fine-tune every combination eliminates the benefit of self-/semi-supervision. To address this issue, we have invented a means of probing AI models’ embedding spaces themselves to gauge the specificity of their feature encodings. Our method characterizes feature spaces by comparing task performance via linear mappings of encoded features across multiple models’ embedding spaces, allowing for benchmarking of novel models against existing counterparts with fewer labeled examples. By constructing linear mappings between embedding spaces, we show that it may be possible to predict a model’s performance at a given task without labeling data in that modality or fine-tuning the model for that task. We discuss the use cases of our technique for research programs and for AI capability acquisitions.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Task Prioritization for Remote Sensing AI Models

  • Ben Thompson,
  • Eliza Mace

摘要

Improvements in self- and semi-supervised machine learning (ML) afford an opportunity to train AI foundation models (FMs) for a variety of specialized sensor modalities. Because a foundation model’s ability to generalize across varied remote-sensing (RS) tasks depends on the data distribution it was trained on, different downstream applications require different levels of feature specificity in the model’s backbone. Therefore, users need a clear way to assess a model’s descriptive capacity so they can prioritize development and choose the right model for each task. Although the end goal of operationalizing an FM is likely to involve task-specific fine-tuning, as the corpus of available FMs for various sensors expands, it is not tractable to label fine-tuning sets for every possible combination of tasks and modalities; in fact, having sufficient volumes of labeled data to fine-tune every combination eliminates the benefit of self-/semi-supervision. To address this issue, we have invented a means of probing AI models’ embedding spaces themselves to gauge the specificity of their feature encodings. Our method characterizes feature spaces by comparing task performance via linear mappings of encoded features across multiple models’ embedding spaces, allowing for benchmarking of novel models against existing counterparts with fewer labeled examples. By constructing linear mappings between embedding spaces, we show that it may be possible to predict a model’s performance at a given task without labeling data in that modality or fine-tuning the model for that task. We discuss the use cases of our technique for research programs and for AI capability acquisitions.