Aims. To examine whether natural language processing (NLP) using a clinically based large language model (LLM) could be used to predict patient selection for total hip or total knee arthroplasty (THA/TKA) from routinely available free-text radiology reports. Methods. Data pre-processing and analyses were conducted according to the Artificial intelligence to Revolutionize the patient Care pathway in Hip and knEe aRthroplastY (ARCHERY) project protocol. This included use of de-identified Scottish regional clinical data of patients referred for consideration of THA/TKA, held in a secure data environment designed for artificial intelligence (AI) inference. Only preoperative radiology reports were included. NLP algorithms were based on the freely available GatorTron model, a LLM trained on over 82 billion words of de-identified clinical text. Two inference tasks were performed: assessment after model-fine tuning (50 Epochs and three cycles of k-fold cross validation), and external validation. Results. For THA, there were 5,558 patient radiology reports included, of which 4,137 were used for model training and testing, and 1,421 for external validation. Following training, model performance demonstrated average (mean across three folds) accuracy, F1 score, and area under the receiver operating curve (AUROC) values of 0.850 (95% confidence interval (CI) 0.833 to 0.867), 0.813 (95% CI 0.785 to 0.841), and 0.847 (95% CI 0.822 to 0.872), respectively. For TKA, 7,457 patient radiology reports were included, with 3,478 used for model training and testing, and 3,152 for external validation. Performance metrics included accuracy, F1 score, and AUROC values of 0.757 (95% CI 0.702 to 0.811), 0.543 (95% CI 0.479 to 0.607), and 0.717 (95% CI 0.657 to 0.778) respectively. There was a notable deterioration in performance on external validation in both cohorts. Conclusion. The use of routinely available preoperative radiology reports provides promising potential to help screen suitable candidates for THA, but not for TKA. The external validation results demonstrate the importance of further model testing and training when confronted with new clinical cohorts. Cite this article: Bone Joint J 2024;106-B(7):688–695

Aims

Total hip arthroplasty (THA) and total knee arthroplasty (TKA) are common orthopaedic procedures requiring postoperative radiographs to confirm implant positioning and identify complications. Artificial intelligence (AI)-based image analysis has the potential to automate this postoperative surveillance. The aim of this study was to prepare a scoping review to investigate how AI is being used in the analysis of radiographs following THA and TKA, and how accurate these tools are.

We obtained information from the Elective Orthopaedic Centre on 1523 patients with baseline and six-month Oxford hip scores (OHS) after undergoing primary hip replacement (THR) and 1784 patients with Oxford knee scores (OKS) for primary knee replacement (TKR) who completed a six-month satisfaction questionnaire. Receiver operating characteristic curves identified an absolute change in OHS of 14 points or more as the point that discriminates best between patients’ satisfaction levels and an 11-point change for the OKS. Satisfaction is highest (97.6%) in patients with an absolute change in OHS of 14 points or more, compared with lower levels of satisfaction (81.8%) below this threshold. Similarly, an 11-point absolute change in OKS was associated with 95.4% satisfaction compared with 76.5% below this threshold. For the six-month OHS a score of 35 points or more distinguished patients with the highest satisfaction level, and for the six-month OKS 30 points or more identified the highest level of satisfaction. The thresholds varied according to patients’ pre-operative score, where those with severe pre-operative pain/function required a lower six-month score to achieve the highest levels of satisfaction. Our data suggest that the choice of a six-month follow-up to assess patient-reported outcomes of THR/TKR is acceptable. The thresholds help to differentiate between patients with different levels of satisfaction, but external validation will be required prior to general implementation in clinical practice