Evidence of limited biases in probed language-based assessments

J Psychopathol Clin Sci. 2026 Nov 9. doi: 10.1037/abn0001183. Online ahead of print.

ABSTRACT

Large language models enable increasingly accurate language-based assessments (LBAs) of psychopathology. As these tools are more widely deployed in real-world settings, a critical question emerges: Do these tools work equally well for everyone? We examined whether probed LBAs, in which individuals respond to open-ended mental health questions in their own words, exhibit systematic biases that could undermine their clinical utility. We collected open-ended mental health descriptions from 1,260 participants alongside validated depression (Patient Health Questionnaire-9) and anxiety (Generalized Anxiety Disorder Scale-7) scores. We converted these descriptions to numeric representations using large-language-model-generated embeddings and applied ridge regression to predict symptom severity, achieving strong predictive accuracy (r > .60). We then examined whether assessment differences were systematically associated with sociodemographic factors (gender, age, education, employment, and income) or linguistic characteristics. Our analysis revealed limited evidence of demographic or linguistic bias: Most variables showed minimal association with assessment errors (i.e., the absolute differences between the LBAs and self-reports). However, higher depression and anxiety severity were associated with larger absolute assessment differences (standardized β = .193, p = .038; β = .252, p = .003). We explored mitigation strategies, finding that downsampling training data eliminated the bias for depression and anxiety. Our findings indicate that probed LBAs exhibit limited bias overall, particularly regarding sociodemographic equity. However, the observed severity-related pattern highlights a fundamental challenge. (PsycInfo Database Record (c) 2026 APA, all rights reserved).

PMID:43002526 | DOI:10.1037/abn0001183