Anne-Marie Brouwer

Professorship chair

Mental state monitoring (Radboud University Nijmegen).

Research area

Information about individuals’ mental state and upcoming performance extracted from physiological measures can be of interest in a range of application domains. The advantage of this type of information is that, unlike information as collected through verbal reports, it is in principle continuous, it does not interfere with individuals’ task or experience, and it can avoid response biases as caused by e.g. demand characteristics or cultural background. Examples of applications are repeating information on an operator’s display when it is likely that it was not attended to before;  using physiological responses to a controlled stressor to predict coping with other (real-life) stress; comparing the affective experience of a product across cultures; and adapting a supportive exoskeleton to better accommodate a likely upcoming body movement.

The research area of the chair can be described as ‘physiological signals for mental state monitoring’. Physiological signals encompass brain signals, peripheral physiological signals such as heart rate and skin conductance, but here, should be viewed broader to also encompass other implicit potential information ‘broadcasted’ by individuals, e.g. eye movements, facial expression or behavior such as mobile phone usage. These types of signals can inform us about individuals’ mental state, such as attention, emotion, mental effort or fatigue. Because of this, they can also inform us about individuals’ future actions or performance. In the proposed chair, special focus is on multimodal mental state monitoring – that is, combining different potential sources of information.

The chair falls under research fields that are referred to as Neuroergonomics, Brain-Computer Interfaces, Applied Neuroscience, Neuro-Adaptive Technology,  Physiological Computing and Affective Computing. These are highly interdisciplinary fields, involving neuroscience, psychology, sensors and signal processing engineering, human factors, machine learning and AI.

Key challenges

Most research in this domain is done in the laboratory, and the validity of existing commercial applications (neuromarketing, neurofeedback, smart coaches) is often unclear. Challenges to successfully make the step from the laboratory to validated application are dealing with noisy data, the lack of context information, generalization of models across people and situations, and usability. The general approach to diminish these problems that will be focused on in this chair, are multi-modal systems. Using information from multiple modalities have long been mentioned as an important way to improve mental state monitoring. However, studies that record and integrate multimodal information usually do this by combining the different variables in a machine learning model, without further taking advantage of what we know about these variables’ meaning and properties. In contrast to what one might expect, multimodal models from these studies do not convincingly improve with respect to using only the best performing feature.

A challenge of another order lies in the ethical domain. In mental state monitoring research, the aim is to extract information from individuals without requiring them to explicitly communicate this information. Even though this is normally done in order to improve their performance or well-being, the diminished or lack of control of what to communicate brings up ethical considerations concerning privacy, data protection, informing the public and the general desirability of different types of mental state monitoring applications.

Ambition and goals of the chair

In the research domain as sketched above, the chair is centered around two aspects.

1. Application: The chair aims to connect mental state monitoring research more strongly to the application. This can be 1) through working towards a concrete application, as desired by society or industry, where the potential added value of physiological measures is clear, given expected accuracies; 2) through working towards solving general challenges in the field such as dealing with noise, lack of context, generalization across people and situations.
2. Multimodal approach: The chair focusses around multimodal research, i.e. examining multiple, concurrently recording sources of information, and to realize its potential to enable and improve applications. 1) Multiple signals are expected to improve mental state estimations and performance predictions. This is because signals are expected to convey complementary information (e.g. emotional arousal will be reflected by skin conductance and focused attention by EEG); they can provide context information (signals can be viewed in the context of what an individual is looking at, or an individual’s current location); together they can be more robust against noise where different signals are robust to different types of noise. 2) Multimodal recordings, including simultaneous measurements of different types of lab grade and wearable systems enables identification of methods that have a favorable cost-benefit in terms of accuracy, ease of use, comfort and financial costs.

Top publications

• Stuldreher, I. V., Thammasan, N., van Erp, J.B.F., & Brouwer, A.-M. (2020). Physiological synchrony in EEG, electrodermal activity and heart rate reflects shared selective auditory attention. Journal of Neural Engineering, 17(4).
• Brouwer, A.-M., Hogervorst, M.A., Oudejans, B., Ries, A.J. & Touryan, J. (2017). EEG and Eye Tracking Signatures of Target Encoding during Structured Visual Search. Frontiers in Human Neuroscience, 11, 264.
• Brouwer, A.-M., Zander TO, van Erp JBF, Korteling JE and Bronkhorst AW (2015) Using neurophysiological signals that reflect cognitive or affective state: six recommendations to avoid common pitfalls. Frontiers in Neuroscience, 9,136.
• Brouwer, A.-M, Hogervorst, M. A., van Erp, J. B. F., Heffelaar, T., Zimmerman, P. H., & Oostenveld, R. (2012). Estimating workload using EEG spectral power and ERPs in the n-back task. Journal of Neural Engineering, 9(4), 045008.
• Brouwer, A.-M., & van Erp, J.B.F. (2010). A tactile P300 Brain-Computer Interface. Frontiers in Neuroscience, 4:19.