PsyEval: a comprehensive large language model evaluation benchmark for mental health

AI Summary

PsyEval benchmark evaluates LLMs in mental health across knowledge, diagnosis and emotional support, capturing the field's complex, context dependent scenarios.
Eleven advanced LLMs were tested with varied prompting strategies to examine how prompts alter responses and performance.
Results reveal significant gaps in accurate reasoning and appropriate responses in mental health contexts while indicating promising directions for future model improvement.

Npj Ment Health Res. 2026 Jul 10. doi: 10.1038/s44184-026-00227-0. Online ahead of print.

ABSTRACT

Evaluating large language models (LLMs) in the mental health domain presents distinct challenges due to the subtle, context-dependent, and subjective nature of psychological symptoms. We introduce PsyEval, a benchmark specifically designed to evaluate LLMs in mental health-related tasks across three core dimensions: knowledge, diagnosis, and emotional support. PsyEval is constructed to reflect the complexity of mental health scenarios and provides a structured framework for assessing model performance within this sensitive domain. Using PsyEval, we evaluate eleven advanced LLMs with different prompting strategies to investigate how prompting affects their responses. The results reveal considerable gaps in LLMs’ current ability to reason accurately and respond appropriately in mental health contexts, while also indicating promising directions for future model enhancement.

PMID:42426307 | DOI:10.1038/s44184-026-00227-0

Document this CPD