Enabling Scalable Evaluation of Bias Patterns in Medical LLMs

Abstract

Large language models (LLMs) have shown impressive potential in helping with numerous medical challenges. Deploying LLMs in high-stakes applications such as medicine, however, brings in many concerns. One major area of concern relates to biased behaviors of LLMs in medical applications, leading to unfair treatment of individuals. To pave the way for the responsible and impactful deployment of Med LLMs, rigorous evaluation is a key prerequisite. Due to the huge complexity and variability of different medical scenarios, existing work in this domain has primarily relied on using manually crafted datasets for bias evaluation. In this study, we present a new method to scale up such bias evaluations by automatically generating test cases based on rigorous medical evidence. We specifically target the challenges of domain-specificity of bias characterization, hallucinating while generating the test cases, and various dependencies between the health outcomes and sensitive attributes. To that end, we offer new methods to address these challenges integrated with our generative pipeline. Specifically, we use medical knowledge graphs and medical ontologies; and customize general LLM evaluation frameworks in our method. Through a series of extensive experiments, we show that the test cases generated by our proposed method are reliable and can effectively reveal bias patterns in LLMs. Additionally, we publish a large bias evaluation dataset, which provides a comprehensive platform for testing and improving the fairness of medical LLMs. A live demo of our application for vignette generation is available at \url{https://vignette.streamlit.app}.