Development and Validation of an Interactive, Bilingual Patient Decision Aid for Artificial Liver Support System Treatment in Acute-on-Chronic Liver Failure

Abstract

Background: Acute-on-chronic liver failure (ACLF) is a syndrome characterized by high short-term mortality. The decision to initiate an artificial liver support system (ALSS) is complex and preference-sensitive. Standardized tools to facilitate shared decision-making (SDM) in this critical context are lacking. Objective: This study aimed to systematically develop and validate an interactive, bilingual (Chinese/English) patient decision aid (PDA) for patients with ACLF considering ALSS, using a novel validation methodology employing large language models (LLMs). Methods: A three-phase, mixed-methods design was employed. Phase 1 (Scoping) involved a systematic literature review to identify core content. Phase 2 (Development) focused on content drafting and technical implementation of an interactive web-based prototype. Phase 3 (Validation) involved a two-pronged approach: 1) An innovative content validation process was conducted by systematically querying five distinct LLMs (GPT-4, Claude 3, Llama 3, Gemini Pro, and a domain-specific medical model) to simulate a multi-disciplinary expert review, assessing the PDA’s content for accuracy, comprehensiveness, clarity, and neutrality. 2) Usability and acceptability were evaluated by 10 representative users (patients and family members) through a think-aloud protocol, the System Usability Scale (SUS), and semi-structured interviews. Results: A web-based PDA titled “Artificial Liver Decision Aid for ACLF” was successfully developed. The LLM-driven validation process resulted in a high degree of consensus on core medical facts and alignment with major clinical guidelines. The iterative querying process generated 17 actionable refinements, primarily enhancing the clarity of technical descriptions and adding nuance to risk-benefit statements. In the user testing, the mean SUS score was 87.5 (SD 6.2; range 75 - 95), corresponding to an “excellent” usability rating. Qualitative user feedback was overwhelmingly positive, highlighting the tool’s clarity, ease of use, and the value of its interactive and bilingual features. Conclusions: We have developed and validated a high-quality PDA for the ALSS decision in ACLF, pioneering a novel and efficient LLM-based method for content validation. The tool demonstrates excellent usability and is a promising resource to support SDM. This study also presents a viable new paradigm for the rapid development of evidence-based patient-facing medical tools.