Study 03
Evaluating HMD Interaction in Museums
Developing practical methods for observing and evaluating HMD-based interaction in real museum environments.
This study examines how HMD-based interaction can be evaluated in real museum environments, where visitors are often first-time users and interaction unfolds simultaneously inside the headset and within the surrounding physical exhibition space.
The evaluation approach developed progressively from practical field-observation methods toward a combination of structured observation and standardized self-report measures.
1. Qualitative evaluation of first-time HMD interaction in real museum environments
The initial evaluation phase focused primarily on qualitative methods. The aim was to understand how museum visitors react when encountering head-mounted AR for the first time, including their spontaneous interpretation of interface cues, uncertainty when facing unfamiliar interaction techniques, and the practical challenges that emerge during first-time use.
To capture these reactions in context, the studies combined think-aloud procedures with structured observation sheets. Visitors were encouraged to verbalize what they were noticing, expecting, or trying to do, while researchers recorded observable interaction problems, hesitation, missed cues, repeated actions, and other behaviors directly during the museum experience.
A particular challenge was that HMD interaction is largely invisible to an external observer. To address this, the headset view was streamed in real time to a portable monitor, allowing researchers to observe the in-headset interface and the visitor’s physical behavior simultaneously without changing the interaction setup itself. This made it possible to identify specific interface problems while preserving the visitor’s natural interaction with the HMD.
Example of the qualitative field-evaluation setup used in the early studies. Researchers observed the visitor’s behavior in the exhibition space while simultaneously monitoring the streamed headset view on a portable display, allowing in-situ documentation of first-time interaction challenges.
Immediately after the experience, short interviews were conducted while the interaction was still fresh in the visitor’s memory. These interviews were used to clarify observed behaviors, unexpected reactions, and the reasoning behind particular interaction difficulties.
Together, think-aloud data, structured observation, streamed headset views, and post-experience interviews provided a practical qualitative evaluation setup for studying first-time HMD interaction under realistic museum conditions.
Related research
2. Extending field evaluation with quantitative experience measures
Building on the qualitative field-evaluation approach, later studies introduced quantitative measures to support more systematic comparison across prototypes and interaction concepts.
In the study of playful locative interaction, two questionnaire instruments were used alongside field observation. The first was a playful-experience questionnaire adapted from the Playful Experiences Questionnaire (PLEXQ) developed by Boberg et al. [1]. The second was a usability questionnaire derived from Lewis’s IBM Computer Usability Satisfaction Questionnaire [2]. These measures made it possible to complement behavioral observations with structured self-reports of perceived playfulness and usability.
Example of the questionnaire-based evaluation approach used alongside field observation. The playful-experience items were adapted from PLEXQ [1], while the usability items were derived from the IBM Computer Usability Satisfaction Questionnaire [2].
In later evaluations, including the comparison of three game-mechanic-based interaction concepts described in Game-Mechanic-Based Interaction Design, the quantitative component was expanded further.
These studies incorporated the System Usability Scale (SUS) [3], the NASA Task Load Index (NASA-TLX) [4], the Immersive Experience Questionnaire (IEQ) [5], and the Game Engagement Questionnaire [6] to capture multiple dimensions of the visitor experience.
These instruments supported the evaluation of usability, perceived workload, immersion, and game-related engagement, while the qualitative field methods continued to provide contextual evidence about where and why interaction difficulties occurred.
The resulting approach therefore combined direct observation, think-aloud data, post-experience interviews, and standardized questionnaire measures. This allowed HMD-AR interaction to be examined not only in terms of whether visitors could complete an experience, but also how understandable, demanding, immersive, playful, and engaging that experience was under realistic museum conditions.
References
[1] Boberg, M., Karapanos, E., Holopainen, J., & Lucero, A. (2015). PLEXQ: Towards a Playful Experiences Questionnaire. Proceedings of the 2015 Annual Symposium on Computer-Human Interaction in Play (CHI PLAY ‘15). DOI: 10.1145/2793107.2793124.
[2] Lewis, J. R. (1995). IBM computer usability satisfaction questionnaires: Psychometric evaluation and instructions for use. International Journal of Human-Computer Interaction, 7(1), 57–78. DOI: 10.1080/10447319509526110.
[3] Brooke, J. (1996). SUS: A “quick and dirty” usability scale. In P. W. Jordan, B. Thomas, B. A. Weerdmeester, & I. L. McClelland (Eds.), Usability Evaluation in Industry (pp. 189–194). Taylor & Francis.
[4] Hart, S. G., & Staveland, L. E. (1988). Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. In P. A. Hancock & N. Meshkati (Eds.), Human Mental Workload, Advances in Psychology, 52, 139–183. DOI: 10.1016/S0166-4115(08)62386-9.
[5] Jennett, C., Cox, A. L., Cairns, P., Dhoparee, S., Epps, A., Tijs, T., & Walton, A. (2008). Measuring and defining the experience of immersion in games. International Journal of Human-Computer Studies, 66(9), 641–661. DOI: 10.1016/j.ijhcs.2008.04.004.
[6] Brockmyer, J. H., Fox, C. M., Curtiss, K. A., McBroom, E., Burkhart, K. M., & Pidruzny, J. N. (2009). The development of the Game Engagement Questionnaire: A measure of engagement in video game-playing. Journal of Experimental Social Psychology, 45(4), 624–634. DOI: 10.1016/j.jesp.2009.02.016.
