CN-122024990-A - Radiology report generation method based on focus attention and semantic knowledge fusion

Abstract

The application belongs to the technical field of medical artificial intelligence, and particularly discloses a radiology report generation method based on focus attention and semantic knowledge fusion. The method comprises the steps of carrying out multi-range label prediction on the radioactive light images based on a pre-trained medical image encoder, carrying out coarse-fine granularity retrieval on similar images and corresponding similar reports, mapping identified medical entities to a unified medical semantic system, and generating a radiology report corresponding to the radioactive light images according to a screened candidate report based on a report integration generation strategy of chain reasoning. Through the mode, a multi-stage search enhancement mechanism formed by label matching, coarse and fine granularity search and knowledge screening is adopted to screen high-precision candidate reports, search errors are remarkably reduced, a report integration generation strategy based on chain reasoning controls a large language model to generate texts only in a reference basis range, medical illusions are avoided, and therefore accuracy and interpretability of generating radiological reports can be effectively improved.

Inventors

• XU TONG
• ZHUANG YIJUN
• XIE XIA

Assignees

• 海南大学

Dates

Publication Date

20260512

Application Date

20260130

Claims (10)

1. 1. A radiology report generation method based on focus attention and semantic knowledge fusion, comprising: acquiring a radioactive light image of a current part of a user, and performing multi-focus label prediction on the radioactive light image based on a pre-trained medical image encoder to obtain a current focus label; Screening similar images matched with the current focus label and corresponding similar reports from a focus label database, respectively carrying out coarse and fine granularity retrieval on the similar images and the corresponding similar reports, and determining candidate reports according to retrieval results; Identifying the medical named entity of the candidate report, mapping the identified medical entity to a unified medical semantic system, and screening the candidate report according to the mapping result; and generating a radiology report corresponding to the radiology image according to the screened candidate report based on a report integration generation strategy of chain reasoning.
2. 2. The method of claim 1, wherein the step of screening a lesion label database for similar images and corresponding similar reports matching the current lesion label, performing coarse-fine granularity retrieval on the similar images and the corresponding similar reports, and determining candidate reports according to retrieval results, comprises: matching the current focus label with each label in a focus label database respectively, and determining a focus label matched with the current focus label according to a matching result; Obtaining a similar image matched with the current focus label and a corresponding similar report according to the matched focus label; Extracting visual embedded vectors of the similar images; And respectively carrying out coarse-fine granularity retrieval on the similar images and the corresponding similar reports according to the visual embedding vectors, and determining candidate reports according to retrieval results.
3. 3. The method of claim 2, wherein the step of performing coarse-fine granularity search on the similar image and the corresponding similar report according to the visual embedding vector, and determining candidate reports according to search results, respectively, comprises: Respectively calculating cosine similarity between the visual embedded vector and each query image embedded vector; Sequencing the cosine similarity according to a preset sequence, and respectively carrying out coarse-granularity retrieval on the similar images and the corresponding similar reports according to a first proportion and a cosine similarity sequencing result; And based on the attention-guided class activation mapping strategy, respectively carrying out fine-granularity retrieval on the similar images after coarse-granularity retrieval and the corresponding similar reports, and determining candidate reports according to retrieval results.
4. 4. The method of claim 3, wherein the step of performing fine-grained retrieval of the coarse-grained retrieved similar images and the corresponding similar reports, respectively, and determining candidate reports according to the retrieval result, based on the attention-directed class activation mapping strategy, comprises: generating respective focus region attention diagrams for the similar images and the radial light images after coarse granularity retrieval based on an attention-guided class activation mapping strategy; Respectively comparing the focus area attention map of the similar image after coarse-grain retrieval with the focus area attention map of the radiation light image in the same area, and determining pixel level divergence difference according to a comparison result; and sorting the pixel level divergence differences according to the preset sequence, respectively carrying out fine-granularity retrieval on the similar images after coarse-granularity retrieval and the corresponding similar reports according to a second proportion and a difference sorting result, and determining candidate reports according to a retrieval result.
5. 5. The method of claim 1, wherein the step of medical named entity recognition of the candidate report, mapping the recognized medical entities to a unified medical semantic system, and screening the candidate report according to the mapping result comprises: Carrying out medical named entity identification on the candidate report, and mapping the identified medical entity to a unified medical semantic system based on a target knowledge alignment strategy; Calculating target semantic similarity among the candidate reports according to the mapping result, and sequencing the target semantic similarity according to a preset sequence; and screening the candidate reports according to the semantic similarity sequencing result.
6. 6. The method of any one of claims 1 to 5, wherein the step of generating a radiology report corresponding to the radiology image from the screened candidate report based on the chain reasoning-based report integration generation strategy comprises: When a report generation instruction is detected, controlling the access of the large language model; inputting the screened candidate report into an accessed large language model; and guiding the accessed large language model to generate a radiology report corresponding to the radiology image based on the report integration generation strategy of chain reasoning.
7. 7. A radiological report generating apparatus based on focus attention and semantic knowledge fusion, comprising: The prediction module is used for acquiring the radioactive light image of the current part of the user, and performing multi-focus label prediction on the radioactive light image based on a pre-trained medical image encoder to obtain a current focus label; the searching module is used for screening similar images matched with the current focus label and corresponding similar reports from a focus label database, respectively carrying out coarse and fine granularity searching on the similar images and the corresponding similar reports, and determining candidate reports according to searching results; The screening module is used for carrying out medical named entity identification on the candidate report, mapping the identified medical entity to a unified medical semantic system and screening the candidate report according to a mapping result; and the generation module is used for integrating the generation strategy based on the report of chain reasoning and generating a radiology report corresponding to the radiology image according to the screened candidate report.
8. 8. An electronic device, comprising: At least one memory for storing a computer program; At least one processor for executing the memory-stored program, which processor is adapted to perform the method according to any of claims 1-6 when the memory-stored program is executed.
9. 9. A computer readable storage medium storing a computer program, characterized in that the computer program, when run on a processor, causes the processor to perform the method according to any one of claims 1-6.
10. 10. A computer program product, characterized in that the computer program product, when run on a processor, causes the processor to perform the method according to any of claims 1-6.

Description

Radiology report generation method based on focus attention and semantic knowledge fusion Technical Field The application belongs to the technical field of medical artificial intelligence, and particularly relates to a radiology report generation method based on focus attention and semantic knowledge fusion. Background Medical image report generation refers to a process of automatically or semi-automatically converting visual information in radiological images into structured, specialized textual descriptions using medical artificial intelligence techniques. At present, the common methods for generating the radiology report are an end-to-end training method and a search method, wherein the end-to-end training method needs to rely on a large amount of labeling data and high calculation power, the interpretation is lacking in an reasoning process, the search method is easily affected by the whole similarity of images in a search stage, the capability of distinguishing tiny or complex focuses is insufficient, and the reference text often contains redundant or conflict information, so that the accuracy of generating the radiology report is low. Thus, the above-described manner of generating radiological reports is less accurate and lacks interpretability. Disclosure of Invention Aiming at the defects of the prior art, the application aims to provide a radiology report generation method based on focus attention and semantic knowledge fusion, and aims to solve the problems that the accuracy of generating radiology reports is low and the interpretability is lacking due to insufficient distinguishing capability of small or complex focuses because the prior art needs to rely on a large amount of labeling data and high calculation power in an end-to-end training mode and is easily influenced by the integral similarity of images in a retrieval stage. To achieve the above object, in a first aspect, the present application provides a radiology report generating method based on focus attention and semantic knowledge fusion, including: acquiring a radioactive light image of a current part of a user, and performing multi-focus label prediction on the radioactive light image based on a pre-trained medical image encoder to obtain a current focus label; Screening similar images matched with the current focus label and corresponding similar reports from a focus label database, respectively carrying out coarse and fine granularity retrieval on the similar images and the corresponding similar reports, and determining candidate reports according to retrieval results; Identifying the medical named entity of the candidate report, mapping the identified medical entity to a unified medical semantic system, and screening the candidate report according to the mapping result; and generating a radiology report corresponding to the radiology image according to the screened candidate report based on a report integration generation strategy of chain reasoning. In an embodiment, the step of screening the similar images and the corresponding similar reports matched with the current focus label from the focus label database, respectively performing coarse-fine granularity search on the similar images and the corresponding similar reports, and determining candidate reports according to the search result includes: matching the current focus label with each label in a focus label database respectively, and determining a focus label matched with the current focus label according to a matching result; Obtaining a similar image matched with the current focus label and a corresponding similar report according to the matched focus label; Extracting visual embedded vectors of the similar images; And respectively carrying out coarse-fine granularity retrieval on the similar images and the corresponding similar reports according to the visual embedding vectors, and determining candidate reports according to retrieval results. In an embodiment, the step of performing coarse-fine granularity search on the similar images and the corresponding similar reports according to the visual embedding vectors, and determining candidate reports according to search results includes: Respectively calculating cosine similarity between the visual embedded vector and each query image embedded vector; Sequencing the cosine similarity according to a preset sequence, and respectively carrying out coarse-granularity retrieval on the similar images and the corresponding similar reports according to a first proportion and a cosine similarity sequencing result; And based on the attention-guided class activation mapping strategy, respectively carrying out fine-granularity retrieval on the similar images after coarse-granularity retrieval and the corresponding similar reports, and determining candidate reports according to retrieval results. In an embodiment, the step of performing fine-granularity retrieval on the similar images after coarse-granularity retrieval and the correspon