- Interpretable random forest using nine predictors achieved strong discrimination for MCI in OSA with external ROC-AUC 0.957 and PR-AUC 0.809.
- Model showed systematic probability overestimation and miscalibration; local calibration assessment and recalibration are required before clinical application.
- Findings derive largely from male patients; outcome not reproducible from questionnaires alone and tool is for research screening prioritisation, not diagnosis.
BMC Psychiatry. 2026 Oct 11;26(1):833. doi: 10.1186/s12888-026-08712-8.
ABSTRACT
BACKGROUND: Mild cognitive impairment (MCI) is clinically relevant in obstructive sleep apnea (OSA). We developed and externally validated an interpretable machine-learning model for concurrent clinically defined MCI to support prioritization for formal cognitive assessment rather than diagnosis.
METHODS: This multicenter cross-sectional diagnostic-prediction study included 400 eligible patients with OSA from Changchun, divided into a training set (n = 280) and an internal algorithm-selection hold-out (n = 120), plus an independent Shenzhen external-validation sample (n = 192). MCI classification integrated reported cognitive decline, objective cognitive evidence, largely preserved everyday independence, dementia exclusion, and assessment-team consensus with physician participation. Of 27 candidate predictors, redundancy review left 24 and LASSO-Boruta consensus retained nine. Feature selection was performed once on the full training set before repeated cross-validation and was not nested within resampling folds. Eight algorithms were tuned using repeated 10-fold cross-validation with training-fold-only SMOTE and PR-AUC as the primary metric. The RF threshold (0.3228) was derived from training out-of-fold probabilities and locked before validation.
RESULTS: MCI was classified in 93/280 (33.2%), 38/120 (31.7%), and 57/192 (29.7%) participants. The nine RF predictors were coffee consumption, physical activity, AHI, MSaO2, PHQ-9, GAD-7, ESS, CPSS-14, and PSQI total scores. In the internal hold-out used for algorithm selection, RF achieved ROC-AUC 0.953, PR-AUC 0.936, and Brier score 0.060; these selection-conditioned estimates are not unbiased generalization estimates. In independent external validation, ROC-AUC was 0.957, PR-AUC 0.809, and Brier score 0.051. Calibration intercept – 0.616 and slope 0.701 indicated systematic probability overestimation despite strong discrimination.
CONCLUSIONS: The RF model discriminated strongly in Shenzhen, but external miscalibration and systematic probability overestimation mean that high AUC does not make raw probabilities transportable; local calibration assessment and, if needed, recalibration are required before clinical use. The MCI outcome cannot be reproduced from questionnaire/scale scores alone, and Supplementary Table S10 is a retrospective operational specification rather than the original CRF. Performance evidence comes almost entirely from male OSA patients; validity in women is unestablished, and the model should not be used for clinical assessment in females. It is a research screening-prioritization tool, not a diagnostic, causal, prognostic, or clinical-impact instrument.
PMID:42947071 | DOI:10.1186/s12888-026-08712-8
Share Evidence Blueprint
Save to Google Notes
Search Google Scholar
Save as PDF
⭐ My Revision List
