Glio-LLaMA-Vision: A Robust Vision-Language Model for Molecular Status Prediction and Radiology Report Generation in Adult-type Diffuse Gliomas

Abstract

BACKGROUND: To establish a robust vision-language model (“Glio-LLaMA-Vision”) for molecular status prediction and radiology report generation (RRG) in adult-type diffuse gliomas. METHODS: Multiparametric MRI data (T1, T2, FLAIR, and postcontrast T1-weighted images) and paired radiology reports (in English) from 1,001 patients with adult-type diffuse gliomas (144 oligodendrogliomas, 157 IDH-mutant astrocytomas, and 700 IDH-wildtype glioblastomas) diagnosed according to the 2021 WHO classification were included in the institutional training set. A vision-language model, Glio-LLaMA-Vision, was developed from LLaMA 3.1 pre-trained on 2.79 million biomedical image-text pairs from PubMed Central and further optimized via fine-tuning from the institutional training set. The performance was validated in 100 patients and 80 patients with paired MRI-radiology reports from an institutional validation set and another tertiary institution, and in 170 and 477 patients with MRI from TCGA and UCSF, respectively. RESULTS: In terms of IDH mutation status prediction, Glio-LLaMA-Vision showed an overall performance of area under the curve, accuracy, sensitivity, and specificity of 0.89 (95% confidence interval 0.81-0.95), 86.0%, 84.0%, and 88.0%, respectively. In terms of radiology report generation, the BLEU-1, ROUGE-L, and METEOR scores were 0.49, 0.42, and 0.24, respectively, while the majority (91.3%) of generated reports were considered clinically acceptable. CONCLUSION: Glio-LLaMA-Vision shows promising performance in molecular status prediction, and RRG in adult-type diffuse gliomas, and shows potential of clinical assistance.