MMSTG-Net: Multi-scale Spatiotemporal Graph Network with Cross-modal Contrastive Alignment for Multimodal Emotion Recognition

IEEE J Biomed Health Inform. 2026 Feb 24;PP. doi: 10.1109/JBHI.2026.3667946. Online ahead of print.

ABSTRACT

Emotion recognition is best positioned as a research tool in psychiatry, with potential to assist future clinical decision-making pending rigorous validation, rather than providing immediate diagnostic or monitoring capability. However, current assessments largely rely on subjective and temporally coarse self-report scales and interviews. To address this gap, we propose MMSTG-Net, a multimodal spatiotemporal graph network for fine-grained seven-class emotion classification from EEG and eye-tracking, which jointly models temporal dynamics, cortical connectivity, and cross-modal alignment. We evaluate MMSTG-Net on two public datasets, SEED-VII and SEED-IV, achieving 82.74% (subject-dependent) and 63.06% (cross-subject) on SEED-VII, and 87.50% (subject-dependent) and 84.85% (cross-subject) on SEED-IV, attaining state-of-the-art performance on SEED-VII and competitive results on SEED-IV. Ablation studies across both datasets confirm the contribution of each module and the stability of the fusion design. These findings support MMSTG-Net as a research-use framework with promise as a future assistive component for clinical emotion assessment, contingent on multi-center, prospective validation.

PMID:41734107 | DOI:10.1109/JBHI.2026.3667946