Biofeedback games

The objective of this work is to develop techniques that teach stress-management skills while patients play video games. As a step towards our goal of relaxation training, we combine biofeedback and adaptive games. Our approach combines game adaptation with a proportional positive feedback controller that modulates to reward low states of arousal/slow breathing rate. When tested against a traditional relaxation method (deep breathing), game biofeedback is more effective in transferring deep breathing skills to a subsequent stress inducing task. Further, it also leads to significantly lower arousal as measured by electrodermal activity and heart rate variability. The advantage of our method over traditional techniques is that  it teaches and allows practice of relaxation techniques while performing a task (i.e. playing a game) that is designed to increase user’s arousal level. And herein lies the main difference with traditional relaxation training, which encourage practice in quiet and peaceful settings that do not reflect the environments we encounter in daily life.

Prior to this, in another study we demonstrated an adaptive biofeedback game that aims to maintain the player’s arousal level by monitoring physiological signals. Here we model the player as a dynamical system whose output (varying arousal levels) must follow an external set-point (constant arousal). Specifically, the control law  manipulates the game’s difficulty level  so as  to maintain  a sustained  arousal  level, as  measured by electrodermal activity (EDA). We also proposed quantitative measures to characterize the effectiveness of the game adaptation mechanics in maintaining the arousal level of the user. Our  approach  has several advantages. First, it allows us to simulate the behavior of the system under different parameter settings, which can help guide the game development process. Second, the model provides a compact parameterization of the system, which facilitates the evaluation of different game mechanics. Finally, it provides objective measures of system performance (error, oscillation) that are complementary to subjective and observation-oriented measures often used in game evaluations.

2023

da C. Silva, D. R.; Wang, Z.; Gutierrez-Osuna, R.

Towards Participant-Independent Stress Detection Using Instrumented Peripherals Journal Article

In: IEEE Transactions on Affective Computing, 2023.

Links | BibTeX

2021

Chakravarthy, N. V.; Silva, D. R. Da Cunha; Gutierrez-Osuna, R.

Evaluating the role of breathing guidance on game-based interventions for relaxation training Journal Article

In: Frontiers in Digital Health, 2021.

Links | BibTeX

Hair, A.; Ballard, K. J.; Markoulli, C.; Monroe, P.; McKechnie, J.; Ahmed, B.; Gutierrez-Osuna, R.

A Longitudinal Evaluation of Tablet-Based Child Speech Therapy with Apraxia World Journal Article

In: ACM Transactions On Accessible Computing, vol. 14, no. 1, 2021.

Abstract | Links | BibTeX

Parnandi, A.; Gutierrez-Osuna, R.

Partial reinforcement in game biofeedback for relaxation training Journal Article

In: IEEE Transactions on Affective Computing, vol. 12, no. 1, pp. 141-153, 2021.

Links | BibTeX

2020

Ahmed, B; Zafar, M; Rihawi, R; Gutierrez-Osuna, R

Gaming away stress: Using biofeedback games to learn paced breathing Journal Article

In: IEEE Transactions on Affective Computing, vol. 11, no. 3, pp. 519-531, 2020.

Links | BibTeX

Zaman, S.; Wesley, A.; Silva, D. Rodrigo Da Cunha; Buddharaju, P.; Akbar, F.; Gao, G.; Mark, G.; Gutierrez-Osuna, R.; Pavlidis, I.

Stress and productivity patterns of interrupted, synergistic, and antagonistic office activities Journal Article

In: Scientific Data, vol. 6, no. 1, 2020.

Links | BibTeX

Blank, C.; Zaman, S.; Wesley, A.; Tsiamyrtzis, P.; Silva, D. R. Da Cunha; Gutierrez-Osuna, R.; Mark, G.; Pavlidis, I.

Emotional Footprints of Email Interruptions Proceedings Article

In: Proc. CHI, 2020.

Links | BibTeX

2019

Akbar, F.; Mark, G.; Pavlidis, I.; Gutierrez-Osuna, R.

An Empirical Study Comparing Unobtrusive Physiological Sensors for Stress Detection in Computer Work Journal Article

In: Sensors, vol. 19, 2019.

Links | BibTeX

Parnandi, A; Gutierrez-Osuna, R

Visual Biofeedback and Game Adaptation in Relaxation Skill Transfer Journal Article

In: IEEE Transactions on Affective Computing, vol. 10, no. 2, pp. 276 - 289, 2019.

Links | BibTeX

Akbar, F.; Bayraktaroglu, A. E.; Gao, G.; Grover, T.; Mark, G.; Storer, K.; Silva, D. R. Da Cunha; Gutierrez-Osuna, R.; Wang, Z.; Buddharaju, P.; Jones, N. Cooper; Pavlidis, I.; Wesley, A.; S. Zaman,

Email Makes You Sweat: Examining Email Interruptions and Stress with Thermal Imaging Proceedings Article

In: Proc. CHI, 2019.

Links | BibTeX

2018

Ahmed, B; Monroe, P; Hair, A; Tan, C-T; Gutierrez-Osuna, R; Ballard, K J

Speech-driven mobile games for speech therapy: User experiences and feasibility Journal Article

In: International Journal of Speech-Language Pathology , vol. 20, no. 6, pp. 644-658, 2018.

Links | BibTeX

Wang, Z; Parnandi, A; Gutierrez-Osuna, R

BioPad: Leveraging off-the-Shelf Video Games for Stress Self-Regulation Journal Article

In: IEEE Journal of Biomedical and Health Informatics, in press, 2018.

Links | BibTeX

2017

Hair, A; Gutierrez-Osuna, R

Deep Breaths: An Internally- and Externally-Paced Deep Breathing Guide Workshop

Proc. 7th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW), 2017.

Abstract | Links | BibTeX

Parnandi, A; Gutierrez-Osuna, R

Physiological modalities for relaxation skill transfer in biofeedback games Journal Article

In: Journal of Biomedical and Health Informatics, vol. in press, 2017.

Links | BibTeX

2016

McKechnie, J; Ballard, K J; McCabe, P; Murray, E; Lan, T; Gutierrez-Osuna, R; Ahmed, B

Influence of type of feedback on effect of tablet-based delivery of intensive speech therapy in children with Childhood Apraxia of Speech Proceedings Article

In: Proceedings of the Motor Speech Conference, 2016.

BibTeX

2015

Parnandi, A; Karappa, V; Lan, T; Shahin, M; McKechnie, J; Ballard, K; Ahmed, B; Gutierrez-Osuna, R

Development of a remote therapy tool for childhood apraxia of speech Journal Article

In: ACM Transactions on Accessible Computing, vol. 7, no. 3, pp. 10:1-10:23, 2015.

Links | BibTeX

Ahmed, B; Ali, H; Choi, J; Gutierrez-Osuna, R

ReBreathe: A calibration protocol that improves stress/relax classification by relabeling deep breathing relaxation exercises Journal Article

In: IEEE Transactions on Affective Computing, vol. in press, 2015.

Links | BibTeX

Bhandari, R; Parnandi, A; Shipp, E; Ahmed, B; Gutierrez-Osuna, R

Music-based respiratory biofeedback in visually-demanding tasks Proceedings Article

In: 15th International Conference on New Interfaces for Musical Expression (NIME), 2015.

Links | BibTeX

Shahin, M; Ahmed, B; Parnandi, A; Karappa, V; McKechnie, J; Ballard, K; Gutierrez-Osuna, R

Tabby Talks: an automated tool for the assessment of childhood apraxia of speech Journal Article

In: Speech Communication, vol. in press, 2015.

Links | BibTeX

2014

Al-Rihawi, R; Ahmed, B; Gutierrez-Osuna, R

Dodging Stress With A Personalized Biofeedback Game Proceedings Article

In: Proc. CHI-PLAY, 2014.

Links | BibTeX

Lan, T; Aryal, S; Ahmed, B; Ballard, K; Gutierrez-Osuna, R

Flappy Voice: An Interactive Game for Childhood Apraxia of Speech Therapy Proceedings Article

In: Proc. CHI-PLAY, 2014.

Links | BibTeX

McKechnie, J; Ballard, K; McCabe, P; Gutierrez-Osuna, R; Karappa, V; Parnandi, A; Shahin, M; Murray, E; Ahmed, B

Tablet-based delivery of intensive speech therapy in children with Childhood Apraxia of Speech - Pilot Phase Proceedings Article

In: Speech Pathology Australia National Conference, 2014.

BibTeX

Parnandi, A; Gutierrez-Osuna, R

A Comparative Study of Game Mechanics and Control Laws for an Adaptive Physiological Game Journal Article

In: Journal on Multimodal User Interfaces, 2014.

Links | BibTeX

Harris, J; Vance, S; Fernandes, O; Parnandi, A; Gutierrez-Osuna, R

Sonic Respiration: Controlling Respiration Rate Through Auditory Biofeedback Proceedings Article

In: Proc. ACM CHI Conference on Human Factors in Computing Systems (CHI 2014) Works-in-Progress, 2014.

Links | BibTeX

2013

Parnandi, A; Ahmed, B; Shipp, E; Gutierrez-Osuna, R

Chill-Out: Relaxation training through respiratory biofeedback in a mobile casual game Conference

Fifth International Conference on Mobile Computing, Applications and Services (MobiCASE 2013), 2013.

Links | BibTeX

Parnandi, A; Karappa, V; Son, Y; Shahin, M; McKechnie, J; Ballard, K; Ahmed, B; Gutierrez-Osuna, R

Architecture of an automated therapy tool for childhood apraxia of speech Conference

The 15th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS), 2013.

Links | BibTeX

Parnandi, A; Gutierrez-Osuna, R

Contactless Measurement of Heart Rate Variability from Fluctuations in Pupillary Dilation Conference

Humaine Association Conference on Affective Computing and Intelligent Interaction (ACII), 2013.

Links | BibTeX

Parnandi, A; Son, Y; Gutierrez-Osuna, R

A Control-Theoretic Approach to Adaptive Physiological Games Conference

Humaine Association Conference on Affective Computing and Intelligent Interaction (ACII), 2013.

Links | BibTeX

Khan, H; Ahmed, B; Choi, J; Gutierrez-Osuna, R

Using an Ambulatory Stress Monitoring Device to Identify Relaxation Due to Untrained Deep Breathing Conference

35th Annual International Conference of the IEEE Engineering in Medicine & Biology Society, 2013.

Abstract | Links | BibTeX

2012

Parnandi, A; Gutierrez-Osuna, R

Contactless Measurement of Heart Rate Variability from Pupillary Fluctuations Technical Report

2012.

Abstract | Links | BibTeX

Son, Y; Parnandi, A; Gutierrez-Osuna, R

A Control-Theoretic Approach to Adaptive Physiological Games Technical Report

2012.

Abstract | Links | BibTeX

Masood, K; Choi, J; Ahmed, B; Gutierrez-Osuna, R

Consistency and Validity of Self-reporting Scores in Stress Measurement Surveys Conference

2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBS), 2012.

Abstract | Links | BibTeX

Parnandi, A; Son, Y; Shahin, M; Ahmed, B; Gutierrez-Osuna, R

Architecture of an Automated Therapy Tool for Childhood Apraxia of Speech Technical Report

2012.

Abstract | Links | BibTeX

Alamudun, F; Choi, J; Gutierrez-Osuna, R; Khan, H; Ahmed, B

Removal of Subject-Dependent and Activity-Dependent Variation in Physiological Measures of Stress Conference

Proceedings of Pervasive Computing Technologies for Healthcare, 2012, ISBN: 978-1-936968-43-5.

Abstract | Links | BibTeX

Choi, J; Ahmed, B; Gutierrez-Osuna, R

Development and Evaluation of an Ambulatory Stress Monitor Based on Wearable Sensors Journal Article

In: IEEE Transactions on Information Technology in Biomedicine, no. 99, pp. 279 - 286, 2012.

Abstract | Links | BibTeX

2011

Choi, J; Gutierrez-Osuna, R

Removal of respiratory influences from heart rate variability in stress monitoring Journal Article

In: IEEE Sensors Journal, vol. 11, no. 11, pp. 2649-2656, 2011.

Abstract | Links | BibTeX

2010

Choi, J; Ahmed, B; Gutierrez-Osuna, R

Ambulatory Stress Monitoring with Minimally-Invasive Wearable Sensors Technical Report

2010.

Abstract | Links | BibTeX

Choi, J; Gutierrez-Osuna, R

Estimating mental stress using a wearable cardio-respiratory sensor Conference

Proceedings of IEEE Sensors, IEEE 2010.

Abstract | Links | BibTeX

2009

Choi, J; Gutierrez-Osuna, R

Using heart rate monitors to detect mental stress Conference

Sixth International Workshop on Wearable and Implantable Body Sensor Networks, IEEE 2009.

Abstract | Links | BibTeX