Medical Report Generation
29 papers with code • 2 benchmarks • 3 datasets
Medical report generation (MRG) is a task which focus on training AI to automatically generate professional report according the input image data. This can help clinicians make faster and more accurate decision since the task itself is both time consuming and error prone even for experienced doctors.
Deep neural network and transformer based architecture are currently the most popular methods for this certain task, however, when we try to transfer out pre-trained model into this certain domain, their performance always degrade.
The following are some of the reasons why RSG is hard for pre-trained models:
- Language datasets in a particular domain can sometimes be quite different from the large number of datasets available on the Internet
- During the fine-tuning phase, datasets in the medical field are often unevenly distributed
More recently, multi-modal learning and contrastive learning have shown some inspiring results in this field, but it's still challenging and requires further attention.
Here are some additional readings to go deeper on the task:
- On the Automatic Generation of Medical Imaging Reports
https://doi.org/10.48550/arXiv.1711.08195
- A scoping review of transfer learning research on medical image analysis using ImageNet
https://arxiv.org/abs/2004.13175
- A Survey on Incorporating Domain Knowledge into Deep Learning for Medical Image Analysis
https://arxiv.org/abs/2004.12150
(Image credit : Transformers in Medical Imaging: A Survey)
Libraries
Use these libraries to find Medical Report Generation models and implementationsLatest papers
Multi-modal Pre-training for Medical Vision-language Understanding and Generation: An Empirical Study with A New Benchmark
With the availability of large-scale, comprehensive, and general-purpose vision-language (VL) datasets such as MSCOCO, vision-language pre-training (VLP) has become an active area of research and proven to be effective for various VL tasks such as visual-question answering.
Automatic Radiology Report Generation by Learning with Increasingly Hard Negatives
At each iteration, conditioned on a given set of hard negative reports, image and report features are learned as usual by minimising the loss functions related to report generation.
Interactive and Explainable Region-guided Radiology Report Generation
While previous methods generate reports without the possibility of human intervention and with limited explainability, our method opens up novel clinical use cases through additional interactive capabilities and introduces a high degree of transparency and explainability.
Dynamic Graph Enhanced Contrastive Learning for Chest X-ray Report Generation
To address the limitation, we propose a knowledge graph with Dynamic structure and nodes to facilitate medical report generation with Contrastive Learning, named DCL.
Cross-Modal Causal Intervention for Medical Report Generation
Medical report generation (MRG) is essential for computer-aided diagnosis and medication guidance, which can relieve the heavy burden of radiologists by automatically generating the corresponding medical reports according to the given radiology image.
Lesion Guided Explainable Few Weak-shot Medical Report Generation
To this end, we propose a lesion guided explainable few weak-shot medical report generation framework that learns correlation between seen and novel classes through visual and semantic feature alignment, aiming to generate medical reports for diseases not observed in training.
DeltaNet:Conditional Medical Report Generation for COVID-19 Diagnosis
To reduce the workload of radiologists, we propose DeltaNet to generate medical reports automatically.
M^4I: Multi-modal Models Membership Inference
To achieve this, we propose Multi-modal Models Membership Inference (M^4I) with two attack methods to infer the membership status, named metric-based (MB) M^4I and feature-based (FB) M^4I, respectively.
A Benchmark for Automatic Medical Consultation System: Frameworks, Tasks and Datasets
In recent years, interest has arisen in using machine learning to improve the efficiency of automatic medical consultation and enhance patient experience.
Transformers in Medical Imaging: A Survey
Following unprecedented success on the natural language tasks, Transformers have been successfully applied to several computer vision problems, achieving state-of-the-art results and prompting researchers to reconsider the supremacy of convolutional neural networks (CNNs) as {de facto} operators.