Surgical Action Triplet Recognition by Using A Multi-Task Prior-Reinforced and Cross-Sample Network

Abstract

Surgical action recognition is an important branch of computer-assisted surgery, which aims to provide real-time monitoring and decision support for clinical surgery. However, most of the existing algorithms only focus on a single aspect of the surgical process and cannot fully describe the surgical process. Following previous work, we modeled surgical action as a triplet <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$&lt;$</tex> instrument, verb, target <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$&gt;$</tex> to represent fine-grained information in surgical scenes. In this paper, we propose a Multi-task Prior-reinforced and Cross-sample Network (MPCNet) to model triplet recognition. We use a multi-task learning paradigm to allow component and triplet recognition to benefit from each other. Moreover, we design a Prior Reinforcement Module (PRM) that enhances the raw triplet features with the component features and learns the associations between components. Finally, we propose an external attention-based Transformer that learns cross-sample semantic relationships through shared keys and values. Our extensive experiments on the CholecT50 dataset demonstrate state-of-the-art results.

Ähnliche Arbeiten