Can you hear the Colour? Towards a Synaesthetic and Multimodal Design Approach in Virtual Worlds

scite shows how a scientific paper has been cited by providing the context of the citation, a classification describing whether it supports, mentions, or contrasts the cited claim, and a label indicating in which section the citation was made.

Header Abstract Content Author and article information 7. REFERENCES Comments

Record: found
Abstract: found
Conference Proceedings: found

Is Open Access

Can you hear the Colour? Towards a Synaesthetic and Multimodal Design Approach in Virtual Worlds

Published

proceedings-article

Author(s): Victoria Wright , Genovefa Kefalidou

Publication date (Print): July 2021

Conference name: 34th British HCI Conference (HCI2021)

Conference theme: Post-pandemic HCI – Living Digitally

Conference date: 20th - 21st July 2021

Keywords: Multisensory, Virtual Environment, Synaesthetic-oriented approach, Change blindness, Time perception

Bookmark

Abstract

Content

Author and article information

Contributors

Victoria Wright

Genovefa Kefalidou

Conference

Publication date: July 2021

Publication date (Print): July 2021

Pages: 48-59

Affiliations

[0001]School of Informatics

University of Leicester, University Road

Leicester, LE1 7RH, UK

Article

DOI: 10.14236/ewic/HCI2021.4

SO-VID: 1419ecaa-cc5b-40f5-92e0-f859880db055

License:

This work is licensed under a Creative Commons Attribution 4.0 Unported License. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/

Conference name: 34th British HCI Conference

Conference acronym: HCI2021

Conference number: 34

Conference location: London, UK

Conference date: 20th - 21st July 2021

Conference sponsor: Electronic Workshops in Computing (eWiC)

Conference theme: Post-pandemic HCI – Living Digitally

History

Product

1477-9358 BCS Learning & Development

Self URI (article page): https://www.scienceopen.com/hosted-document?doi=10.14236/ewic/HCI2021.4

Self URI (journal page): https://ewic.bcs.org/

7. REFERENCES

AWE - Augmented World Expo (2016) Curtis Hickman THE VOID: Creating The Illusion of Reality. Available from: https://www.youtube.com/watch?v=Ebwtq1HZJ2A (6 May 2021).
G. Bargary, K.J. Barnett, K.J. Mitchell and F.N. Newell (2009) Colored-speech synaesthesia is triggered by multisensory, not unisensory, perception. Psychological Science, 20(5), pp.529-533.
S. Baron-Cohen, M.A. Wyke and C. Binnie (1987) Hearing words and seeing colours: an experimental investigation of a case of synaesthesia. Perception, 16(6), pp.761-767.
M.I. Berkman and E. Akan (2018) Presence and Immersion in Virtual Reality. Encyclopedia of Computer Graphics and Games, pp. 1-10.
Blender (2020) blender.org - Home of the Blender project – Free and Open 3D Creation Software. Available from: https://www.blender.org/ (6 May 2021).
R.A. Block and D. Zakay (1997) Prospective and retrospective duration judgments: A meta-analytic review. Psychonomic bulletin & review, 4(2), pp.184-197.
V. Braun and V. Clarke (2006) Using thematic analysis in psychology. Qualitative research in psychology, 3(2), pp.77-101.
A.L. Brooks (2021) Interactive Multisensory VibroAcoustic Therapeutic Intervention (iMVATi). Recent Advances in Technologies for Inclusive Well-Being, 196, p.325.
S. Casini (2017) Synesthesia, transformation and synthesis: toward a multi-sensory pedagogy of the image. The Senses and Society, 12(1), pp.1-17.
C. Cheng, G.B. Baker and S.M. Dursun (2019) Use of multisensory stimulation interventions in the treatment of major neurocognitive disorders. Psychiatry and Clinical Psychopharmacology, 29(4), pp.916-921.
L. Diesendruck, L. Gertner, L. Botzer, L. Goldfarb, A. Karniel and A. Henik (2010) Months in space: Synaesthesia modulates attention and action. Cognitive neuropsychology, 27(8), pp.665-679.
K. Hinckley and H. Song (2011, May) Sensor synaesthesia: touch in motion, and motion in touch. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 801-810).
Hyper Reality Experience (2017) Hyper Reality Experience | Tick Tock Unlock. Available from: http://hyperrealityexperience.com/ (6 May 2021).
A. Jaimes and N. Sebe (2005, October) Multimodal human computer interaction: A survey. In: International workshop on human-computer interaction (pp. 1-15). Springer, Berlin, Heidelberg.
Jisc Online surveys (2020) Online surveys. Available from: https://www.onlinesurveys.ac.uk/ (Accessed 6 May 2021).
C.H. Lee, D. Lockton, J. Stevens, S.J. Wang and S. Ahn (2019, May) Synaesthetic-Translation Tool: Synaesthesia as an Interactive Material for Ideation. In: Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (pp. 1-6).
L. Maister, M.J. Banissy and M. Tsakiris (2013) Mirror-touch synaesthesia changes representations of self-identity. Neuropsychologia, 51(5), pp.802-808.
R. McLay, V. Ram, J. Murphy, J. Spira, D.P. Wood, M.D. Wiederhold, B.K. Wiederhold, S. Johnston and D. Reeves (2014) Effect of Virtual Reality PTSD Treatment on Mood and Neurocognitive Outcomes. CyberPsychology, Behavior & Social Networking, 17(7), pp. 439-446.
S. Merter (2017) Synesthetic approach in the design process for enhanced creativity and multisensory experiences. The Design Journal, 20(sup1), pp.S4519-S4528.
Niantic (2016) Pokémon GO. Available from: https://pokemongolive.com/en/ (6 May 2021).
S. Nicholson (2016) The State of Escape: Escape Room Design and Facilities. Meaningful Play Available from: http://scottnicholson.com/pubs/stateofescape.pdf (6 May 2021)
Oculus (2019) Oculus Rift S: VR Headset for VR-ready PCs | Oculus. Available from: https://www.oculus.com/rift-s/ (6 May 2021).
J.H. Patrick, R.A. Pruchno and M.S. Rose, 1998. Recruiting research participants: a comparison of the costs and effectiveness of five recruitment strategies. The Gerontologist, 38(3), pp.295-302.
D. Putrino, J. Ripp, J.E. Herrera, M. Cortes, C. Kellner, D. Rizk and K. Dams-O’Connor (2020) Multisensory, Nature-Inspired Recharge Rooms Yield Short-Term Reductions in Perceived Stress Among Frontline Healthcare Workers. Frontiers in Psychology, 11, p.3213.
S. Rajasekaran, C. Luteran, H. Qu and C. Riley-Doucet, 2011, January. A portable autonomous multisensory intervention device (pamid) for early detection of anxiety and agitation in patients with cognitive impairments. In: 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society (pp. 4733-4736). IEEE.
G. Riva, F. Mantovani, C.S. Capideville, A. Preziosa, F. Morganti, D. Villani, A. Gaggioli, C. Botella and M. Alcañiz (2007) Affective interactions using virtual reality: the link between presence and emotions. CyberPsychology & Behavior, 10(1), pp.45-56.
A. Sánchez, J.C. Millán-Calenti, L. Lorenzo-López and A. Maseda (2013) Multisensory stimulation for people with dementia: a review of the literature. American Journal of Alzheimer's Disease & Other Dementias®, 28(1), pp.7-14.
T. Sanders and P. Cairns (2010) Time perception, immersion and music in videogames. Proceedings of HCI 2010 24, pp.160-167.
H.N. Schifferstein (2011, October) Multi sensory design. In: Proceedings of the Second Conference on Creativity and Innovation in Design (pp. 361-362).
E. Shaw, T. Roper, T. Nilsson, G. Lawson, S.V. Cobb and D. Miller (2019, April) The Heat is On: Exploring User Behaviour in a Multisensory Virtual Environment for Fire Evacuation. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (pp. 1-13). ACM.
S.P. Smith and D. Trenholme (2009) Rapid prototyping a virtual fire drill environment using computer game technology. Fire safety journal, 44(4), pp.559-569.
F. Spector and D. Maurer (2013) Synesthesia: A new approach to understanding the development of perception. Psychology of Consciousness: Theory, Research, and Practice, 1(S), pp. 108-129.
C. Spence and J. Youssef (2019) Synaesthesia: The multisensory dining experience. International Journal of Gastronomy and Food Science, 18, p.100179.
E. Striem-Amit, M. Guendelman and A. Amedi (2012) ‘Visual’ acuity of the congenitally blind using visual-to-auditory sensory substitution. PloS one, 7(3), p.e33136.
E.A. Suma, S. Clark, D. Krum, S. Finkelstein, M. Bolas and Z. Warte (2011, March). Leveraging change blindness for redirection in virtual environments. In: 2011 IEEE Virtual Reality Conference (pp. 159-166). IEEE.
The VOID (2019) The VOID | A Virtual Reality Experience. Available from: http://web.archive.org/web/20200304070516if_/https://www.thevoid.com/ (8 May 2021).
H. Van Kerrebroeck, M. Brengman and K. Willems (2017) Escaping the crowd: An experimental study on the impact of a Virtual Reality experience in a shopping mall. Computers in Human Behavior, 77(1), pp. 437-450.
M.S. Vitevitch (2003) Change deafness: The inability to detect changes between two voices. Journal of Experimental Psychology: Human Perception and Performance, 29(2), p.333.
M. Voong and R. Beale (2007, April) Music organisation using colour synaesthesia. In CHI'07 extended abstracts on Human Factors in Computing Systems (pp. 1869-1874).
J. Ward, A. P. Field and T. Chin (2019). A meta-analysis of memory ability in synaesthesia. Memory, 27(9), pp. 1299-1312.
J. Wareing, G. Lawson, C. Abdullah and T. Roper (2018, September) User Perception of Heat Source Location for a Multisensory Fire Training Simulation. In: 2018 10th Computer Science and Electronic Engineering (CEEC) pp. 214-218. IEEE.
D. Watson, L.A. Clark and A. Tellegen (1988) Development and validation of brief measures of positive and negative affect: the PANAS scales. Journal of personality and social psychology, 54(6), p.1063.
B.G. Witmer and M.J. Singer (1998) Measuring presence in virtual environments: A presence questionnaire. Presence, 7(3), pp.225-240.

Comments

Comment on this article

[1] AWE - Augmented World Expo (2016) Curtis Hickman THE VOID: Creating The Illusion of Reality. Available from: https://www.youtube.com/watch?v=Ebwtq1HZJ2A (6 May 2021).

[2] G. Bargary, K.J. Barnett, K.J. Mitchell and F.N. Newell (2009) Colored-speech synaesthesia is triggered by multisensory, not unisensory, perception. Psychological Science, 20(5), pp.529-533.

[3] S. Baron-Cohen, M.A. Wyke and C. Binnie (1987) Hearing words and seeing colours: an experimental investigation of a case of synaesthesia. Perception, 16(6), pp.761-767.

[4] M.I. Berkman and E. Akan (2018) Presence and Immersion in Virtual Reality. Encyclopedia of Computer Graphics and Games, pp. 1-10.

[5] Blender (2020) blender.org - Home of the Blender project – Free and Open 3D Creation Software. Available from: https://www.blender.org/ (6 May 2021).

[6] R.A. Block and D. Zakay (1997) Prospective and retrospective duration judgments: A meta-analytic review. Psychonomic bulletin & review, 4(2), pp.184-197.

[7] V. Braun and V. Clarke (2006) Using thematic analysis in psychology. Qualitative research in psychology, 3(2), pp.77-101.

[8] A.L. Brooks (2021) Interactive Multisensory VibroAcoustic Therapeutic Intervention (iMVATi). Recent Advances in Technologies for Inclusive Well-Being, 196, p.325.

[9] S. Casini (2017) Synesthesia, transformation and synthesis: toward a multi-sensory pedagogy of the image. The Senses and Society, 12(1), pp.1-17.

[10] C. Cheng, G.B. Baker and S.M. Dursun (2019) Use of multisensory stimulation interventions in the treatment of major neurocognitive disorders. Psychiatry and Clinical Psychopharmacology, 29(4), pp.916-921.

[11] L. Diesendruck, L. Gertner, L. Botzer, L. Goldfarb, A. Karniel and A. Henik (2010) Months in space: Synaesthesia modulates attention and action. Cognitive neuropsychology, 27(8), pp.665-679.

[12] K. Hinckley and H. Song (2011, May) Sensor synaesthesia: touch in motion, and motion in touch. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 801-810).

[13] Hyper Reality Experience (2017) Hyper Reality Experience | Tick Tock Unlock. Available from: http://hyperrealityexperience.com/ (6 May 2021).

[14] A. Jaimes and N. Sebe (2005, October) Multimodal human computer interaction: A survey. In: International workshop on human-computer interaction (pp. 1-15). Springer, Berlin, Heidelberg.

[15] Jisc Online surveys (2020) Online surveys. Available from: https://www.onlinesurveys.ac.uk/ (Accessed 6 May 2021).

[16] C.H. Lee, D. Lockton, J. Stevens, S.J. Wang and S. Ahn (2019, May) Synaesthetic-Translation Tool: Synaesthesia as an Interactive Material for Ideation. In: Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (pp. 1-6).

[17] L. Maister, M.J. Banissy and M. Tsakiris (2013) Mirror-touch synaesthesia changes representations of self-identity. Neuropsychologia, 51(5), pp.802-808.

[18] R. McLay, V. Ram, J. Murphy, J. Spira, D.P. Wood, M.D. Wiederhold, B.K. Wiederhold, S. Johnston and D. Reeves (2014) Effect of Virtual Reality PTSD Treatment on Mood and Neurocognitive Outcomes. CyberPsychology, Behavior & Social Networking, 17(7), pp. 439-446.

[19] S. Merter (2017) Synesthetic approach in the design process for enhanced creativity and multisensory experiences. The Design Journal, 20(sup1), pp.S4519-S4528.

[20] Niantic (2016) Pokémon GO. Available from: https://pokemongolive.com/en/ (6 May 2021).

[21] S. Nicholson (2016) The State of Escape: Escape Room Design and Facilities. Meaningful Play Available from: http://scottnicholson.com/pubs/stateofescape.pdf (6 May 2021)

[22] Oculus (2019) Oculus Rift S: VR Headset for VR-ready PCs | Oculus. Available from: https://www.oculus.com/rift-s/ (6 May 2021).

[23] J.H. Patrick, R.A. Pruchno and M.S. Rose, 1998. Recruiting research participants: a comparison of the costs and effectiveness of five recruitment strategies. The Gerontologist, 38(3), pp.295-302.

[24] D. Putrino, J. Ripp, J.E. Herrera, M. Cortes, C. Kellner, D. Rizk and K. Dams-O’Connor (2020) Multisensory, Nature-Inspired Recharge Rooms Yield Short-Term Reductions in Perceived Stress Among Frontline Healthcare Workers. Frontiers in Psychology, 11, p.3213.

[25] S. Rajasekaran, C. Luteran, H. Qu and C. Riley-Doucet, 2011, January. A portable autonomous multisensory intervention device (pamid) for early detection of anxiety and agitation in patients with cognitive impairments. In: 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society (pp. 4733-4736). IEEE.

[26] G. Riva, F. Mantovani, C.S. Capideville, A. Preziosa, F. Morganti, D. Villani, A. Gaggioli, C. Botella and M. Alcañiz (2007) Affective interactions using virtual reality: the link between presence and emotions. CyberPsychology & Behavior, 10(1), pp.45-56.

[27] A. Sánchez, J.C. Millán-Calenti, L. Lorenzo-López and A. Maseda (2013) Multisensory stimulation for people with dementia: a review of the literature. American Journal of Alzheimer's Disease & Other Dementias®, 28(1), pp.7-14.

[28] T. Sanders and P. Cairns (2010) Time perception, immersion and music in videogames. Proceedings of HCI 2010 24, pp.160-167.

[29] H.N. Schifferstein (2011, October) Multi sensory design. In: Proceedings of the Second Conference on Creativity and Innovation in Design (pp. 361-362).

[30] E. Shaw, T. Roper, T. Nilsson, G. Lawson, S.V. Cobb and D. Miller (2019, April) The Heat is On: Exploring User Behaviour in a Multisensory Virtual Environment for Fire Evacuation. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (pp. 1-13). ACM.

[31] S.P. Smith and D. Trenholme (2009) Rapid prototyping a virtual fire drill environment using computer game technology. Fire safety journal, 44(4), pp.559-569.

[32] F. Spector and D. Maurer (2013) Synesthesia: A new approach to understanding the development of perception. Psychology of Consciousness: Theory, Research, and Practice, 1(S), pp. 108-129.

[33] C. Spence and J. Youssef (2019) Synaesthesia: The multisensory dining experience. International Journal of Gastronomy and Food Science, 18, p.100179.

[34] E. Striem-Amit, M. Guendelman and A. Amedi (2012) ‘Visual’ acuity of the congenitally blind using visual-to-auditory sensory substitution. PloS one, 7(3), p.e33136.

[35] E.A. Suma, S. Clark, D. Krum, S. Finkelstein, M. Bolas and Z. Warte (2011, March). Leveraging change blindness for redirection in virtual environments. In: 2011 IEEE Virtual Reality Conference (pp. 159-166). IEEE.

[36] The VOID (2019) The VOID | A Virtual Reality Experience. Available from: http://web.archive.org/web/20200304070516if_/https://www.thevoid.com/ (8 May 2021).

[37] H. Van Kerrebroeck, M. Brengman and K. Willems (2017) Escaping the crowd: An experimental study on the impact of a Virtual Reality experience in a shopping mall. Computers in Human Behavior, 77(1), pp. 437-450.

[38] M.S. Vitevitch (2003) Change deafness: The inability to detect changes between two voices. Journal of Experimental Psychology: Human Perception and Performance, 29(2), p.333.

[39] M. Voong and R. Beale (2007, April) Music organisation using colour synaesthesia. In CHI'07 extended abstracts on Human Factors in Computing Systems (pp. 1869-1874).

[40] J. Ward, A. P. Field and T. Chin (2019). A meta-analysis of memory ability in synaesthesia. Memory, 27(9), pp. 1299-1312.

[41] J. Wareing, G. Lawson, C. Abdullah and T. Roper (2018, September) User Perception of Heat Source Location for a Multisensory Fire Training Simulation. In: 2018 10th Computer Science and Electronic Engineering (CEEC) pp. 214-218. IEEE.

[42] D. Watson, L.A. Clark and A. Tellegen (1988) Development and validation of brief measures of positive and negative affect: the PANAS scales. Journal of personality and social psychology, 54(6), p.1063.

[43] B.G. Witmer and M.J. Singer (1998) Measuring presence in virtual environments: A presence questionnaire. Presence, 7(3), pp.225-240.

UK Computing Summit 2025: Navigating change (surviving and beyond) - 29-30 April @ Sheffield Hallam University - Register here.

Can you hear the Colour? Towards a Synaesthetic and Multimodal Design Approach in Virtual Worlds

Contributors

Conference

Affiliations

Article

History

Product

Categories

Comment on this article