A Radiology Report Generation Model Based on Boosting Cross-Modal Information Joint Memory Alignment

Abstract

Radiology Report Generation (RRG) is designed to improve the quality of radiology reports, reduce the workload and subjective errors of radiologists, and facilitate clinical automation. Previous approaches have typically used encoder-decoder architectures, mostly by injecting additional auxiliary information (e.g., knowledge graphs etc.) into the model to improve model performance; few studies have explored cross-modal information interaction and memory utilization. Here, we present a boosted cross-modal information joint memory alignment (BcI-mA) Transformer to efficiently generate radiology report tasks. First, we design a cross-modal information boost module (BcI), which can effectively enhance the interaction between intra- and inter-frame vector features and can be dynamically and iteratively updated with the training process as auxiliary information. Second, a plug-and-play module, which consists of BcI and the memory-aligned (mA) attention mechanism, is designed and embedded into BcI-mA. The method flexibly injects auxiliary information extracted from BcI into mA and utilizes it for information alignment with historical memory. The experimental results show that BcI-mA achieves substantial improvements and outperforms the latest state-of-the-art methods in benchmarking two popular radiology reporting datasets. Further analysis demonstrated that our method is capable of generating radiology reports that contain the necessary medical anomaly information terms.

Ähnliche Arbeiten