As the volume of data in the medical field grows exponentially, the issue of data redundancy has become increasingly prominent. For instance, in the case of the large public medical dataset MIMIC-III, the redundancy rate in hospitalization records reaches as high as 97%–98%. This redundancy not only significantly increases storage costs but also reduces data processing efficiency. To date, there has been no proposed method for table-level redundant data pruning specifically tailored to the unique data types found in medical datasets. To address this issue, this paper introduces a multi-level pruning method based on a data lake, which efficiently eliminates redundancy through a hierarchical processing strategy. The method progressively narrows the search space by constructing an inclusion graph (Containment Graph Construction) combined with the multi-level pruning technique, enabling the efficient identification and removal of redundant data. Additionally, to ensure the efficiency, accuracy, and clinical practicality of the pruning results, this paper proposes a “Clinical Decision Making” mechanism. Experimental results demonstrate that this approach can efficiently process large scale medical data and accurately eliminate approximately 23% of redundant data.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

A Multi-level Pruning Method for Medical Redundant Data Based on Data Lake

  • Tianyi Liu,
  • Jiaxu Guo,
  • Chao Xia,
  • Peng Ren,
  • Chunxiao Xing

摘要

As the volume of data in the medical field grows exponentially, the issue of data redundancy has become increasingly prominent. For instance, in the case of the large public medical dataset MIMIC-III, the redundancy rate in hospitalization records reaches as high as 97%–98%. This redundancy not only significantly increases storage costs but also reduces data processing efficiency. To date, there has been no proposed method for table-level redundant data pruning specifically tailored to the unique data types found in medical datasets. To address this issue, this paper introduces a multi-level pruning method based on a data lake, which efficiently eliminates redundancy through a hierarchical processing strategy. The method progressively narrows the search space by constructing an inclusion graph (Containment Graph Construction) combined with the multi-level pruning technique, enabling the efficient identification and removal of redundant data. Additionally, to ensure the efficiency, accuracy, and clinical practicality of the pruning results, this paper proposes a “Clinical Decision Making” mechanism. Experimental results demonstrate that this approach can efficiently process large scale medical data and accurately eliminate approximately 23% of redundant data.