AN AUTOMATED METHOD FOR CREATING UNDERSTANDABLE PATIENT CONTENT USING MEDICAL TEXTS WITH LLM

Abstract

The automatic simplifying of medical text into patient friendly language should be done in such a way that readability is not compromised and the clinically relevant meaning is not lost. In this work, therefore, we study such a trade-off using small encoder--decoder transformers and not very large language models as it is often impractical in clinical settings due to privacy and compute constraints. We compare supervised fine-tuning (SFT), retrieval-augmented generation (RAG) and lightweight reinforcement learning with (simulated) human feedback (RLHF) pipeline based on BART-base, T5-small and BioBART. Experiments are carried out on two parallel complementary corpora, which are SimpleDC (digestive-cancer education) and Med-EASi (clinically oriented) datasets, where the simplification outputs are controllable. We create a modular preprocessing pipeline to eliminate the HTML, standardize contractions, and expand common medical terminologies and filter stop words by domain to decrease the amount of noise in the supervision signal. Synthetic reward model is a combination of semantic similarity, readability, lexical simplicity and coherence and is trained using PPO with strong KL and entropy regularization. RAG applies dense retrieval with training pairs to prepend two in-domain simplification exemplars during the inference time. The excellent performance on SimpleDC is when the best small model set-up (BART with RAG+RLHF) achieves about a SARI of 61 around grade 9 readability as opposed to SFT-only baselines with a high F1 BERTScore of around 0.90. On Med-EASi, proactively (aggressive preprocessing) and weakly SFT alone already make SARI better than previous controlled baselines, but RLHF and RAG primarily are modifying the shape of the fidelity- vs. readability-borderline, rather than yielding large SARI increases. In either data set we find that (i) RAG will move the outputs into simpler register at some cost in lexical fidelity and (ii) RLHF will shift the Pareto frontier down the semantic preservation axis at the cost of increased reading level and (iii) their combination can be seen to approach but not match the performance of Llama-2 based systems reported in previous literature. We summarize by specifically addressing the clinical safety, constraint of synthetic feedback, and suggestions of implementing future humanin-the-loop assessment of medical text simplification systems.