VeriLLMed: Visual Debugging of Medical Large Language Models with Knowledge Graphs

Authors

Yurui Xiang (Sun Yat-sen University), Xingyi Mao (Sun Yat-sen University), Rui Sheng (The Hong Kong University of Science and Technology), Zixin Chen (The Hong Kong University of Science and Technology), Zelin Zang (AI), Yuyang Wu (Carnegie Mellon University), Haipeng Zeng (Sun Yat-sen University), Huamin Qu (The Hong Kong University of Science and Technology), Yushi Sun (The Hong Kong University of Science and Technology), Yanna Lin (University of Waterloo)

Presentation

Session

Can we have a (real) doctor, please?

Time

Tuesday, Nov 10, 13:00 – 13:12 (US/Eastern) · session 13:00 – 14:30

Location

Hall America north

Keywords

Large language models, diagnosis and debugging, biomedicine, knowledge graph

Abstract

Large language models (LLMs) show promise in medical diagnosis, but real-world deployment remains challenging due to high-stakes clinical decisions and imperfect reasoning reliability. As a result, careful inspection of model behavior is essential for assessing whether diagnostic reasoning is reliable and clinically grounded. However, debugging medical LLMs remains difficult. First, developers often lack sufficient medical domain expertise to interpret model errors in clinically meaningful terms. Second, models can fail across a large and diverse set of instances involving different input types, tasks, and reasoning steps, making it challenging for developers to prioritize which errors deserve focused inspection. Third, developers struggle to identify recurring error patterns across cases, as existing debugging practices are largely instance-centric and rely on manual inspection of isolated failures. To address these challenges, we present VeriLLMed, a visual analytics system that integrates external biomedical knowledge to audit and debug medical LLM diagnostic reasoning. VeriLLMed transforms model outputs into comparable reasoning paths, constructs knowledge graph-grounded reference paths, and identifies three recurring classes of diagnosis errors: missing errors, relation errors, and branch errors. Case studies and expert evaluation demonstrate that VeriLLMed helps developers identify clinically implausible reasoning and generate actionable insights that can inform the improvement of medical LLMs.

For Practitioners

Medical AI developers, healthcare data scientists, and teams evaluating diagnosis-oriented LLMs would be interested in this work. They can use the proposed knowledge-grounded visual analytics workflow to identify recurrent reasoning errors, compare model reasoning with biomedical reference paths, prioritize problematic behaviors, and guide model refinement.