Shelby Hall Graduate Research Forum Posters
Files
Download Full Text (1.2 MB)
Description
Edge AI models for real-time cognitive engagement detection will have major effects on clinical neuroscience, human-computer interface, and education. This work provides a lightweight deep learning model for frame-by- frame video classification and develops an edge device. The proposed edge device delivers low latency, energy-efficient processing, and high temporal resolution, achieving 29 frames per second (FPS) with each frame taking 30 milliseconds, all without reliance on cloud services. Our investigation also introduces a Multimodal Fusion Framework designed to integrate physiological signals and behavioral data into the engagement classification process, in contrast to conventional systems that rely solely on video-based categorization. Using synchronized EEG, ECG, and face video data from 17 student volunteers engaged in controlled cognitive activities, the study examined. While EEG alpha and beta frequency bands gave important new perspectives on cognitive workload, ECG-derived heart rate variability (HRV) acted as an indirect predictor of cognitive stress. Reaction times and task accuracy among behavioral measures provided outside confirmation of cognitive states. This work investigates the strengths and limits of each modality by means of independent and integrated data streams, therefore stressing the possibilities of improving classification accuracy and contextual knowledge of engagement levels. In contrast to conventional systems that mostly rely on video-based categorization, this study provides a Multimodal Fusion Framework that integrates behavioral and physiological information into the engagement classification process. Without requiring cloud infrastructure, the proposed edge device demonstrates low latency, energy-efficient computation, and excellent temporal resolution.
Publication Date
3-2025
Department
Electrical & Computer Engineering
City
Mobile
Disciplines
Bioelectrical and Neuroengineering | Biomedical | Digital Communications and Networking | Electrical and Electronics | Systems and Communications | Systems and Integrative Engineering | VLSI and Circuits, Embedded and Hardware Systems
Recommended Citation
Arnold, Isaac; Burch, Eva; Zha, Shenghua; and Shelley-Tremblay, John, "Edge AI-Based Cognitive Engagement Detection and Multimodal Exploration" (2025). Shelby Hall Graduate Research Forum Posters. 3.
https://jagworks.southalabama.edu/southalabama-shgrf-posters/3
Included in
Bioelectrical and Neuroengineering Commons, Biomedical Commons, Digital Communications and Networking Commons, Electrical and Electronics Commons, Systems and Communications Commons, Systems and Integrative Engineering Commons, VLSI and Circuits, Embedded and Hardware Systems Commons