Reusable interaction design patterns for HMD-based AR in cultural heritage

🔍

Publications

Publications behind the HMD-AR pattern system.

Peer-reviewed publications documenting the empirical studies, interaction design explorations, evaluation methods, and reusable design knowledge developed through this research.

2026

Figure from A Pattern Language for Head-Mounted AR in Museums: Parameterized, Composable, and Reproducible

ACM International Conference on Interactive Media Experiences (IMX), 2026, pp. 46–64

A Pattern Language for Head-Mounted AR in Museums: Parameterized, Composable, and Reproducible

Yu Liu, Ulrike Spierling

Head-mounted AR (HMD-AR) is entering museums and exhibitions, yet creators such as AR developers and designers lack a reusable, parameterized, and verifiable interaction vocabulary for first-time visitors. We present a pattern language for cultural-heritage HMD-AR organized into three classes—PoI Guiding, Experience Indicating, and Experience Presenting—with nine sub-patterns that specify default safety bands and composition rules. Patterns map one-to-one to Unity prefabs on an instrumented runtime with unified logging. Using a Design Science Research (DSR) combined with Research-through-Design (RtD) approach, we instantiated two in-situ prototypes at European natural-history exhibits and evaluated them through creator panels (AR developers and cultural-heritage practitioners, N = 10) and visitor studies (N = 68) on HoloLens 2 and Quest 3. Experts judged the language clear and recommendable, and in-museum results met accepted thresholds for usability, workload, user experience, and presence, with stable behavioral logs under the studied operational conditions. Cross-site synthesis across two natural-history museum deployments showed directionally consistent outcomes and suggested that the interaction layer could be transferred with low effort within this scope: deployments required parameter localization, while core compositions and state machines remained unchanged. We contribute a parameterized, composable interaction language; aligned reference implementations and prototypes; creator- and user-side evidence; and a standardized mapping from constructs to metrics, measures, and decision rules for reproducible cross-study comparison. We release pattern cards, parameter documentation, and reusable Unity prefabs through a public GitHub repository to support transparent reuse, replication, and cross-study comparison.

HCI HMD-AR Design Patterns Museums
DOI ↗ Publication ↗

International Journal of Human-Computer Interaction, 2026

Toward Relational Interactivity: Investigating More-than-Human Engagement in Hybrid Museological Spaces

Pedro Galvão-Ferreira, Marta Ferreira, Yu Liu, Noura Kräuter, Teresa Paulino, Beste Soybilge, Nuno Nunes, Ulrike Spierling, Valentina Nisi

This paper investigates how interactive technologies can cultivate more-than-human (MtH) engagement within natural history museums when conceptualized as hybrid museological spaces. Responding to ecological and posthuman turns in HCI and museum studies, we present a comparative, in-the-wild study of five interactive prototypes – an AI conversational agent, an interactive visual narrative installation, a collaborative story-driven board game, and two augmented reality experiences (based on MtH interactions in the museum floor and outdoor garden) – designed to foster relational encounters between humans, technologies, and nonhumans. Drawing on qualitative data from seventy-one visitors, we identify six overarching themes that elucidate the experiential qualities through which MtH relations were negotiated, including nonhuman co-participation, situated ecologies of participation, affective-ethical entanglements, and narrative co-storying. Synthesizing these insights, we propose relational interactivity as a strong concept that shifts focus from individualized, system-centered interaction toward distributed agency, shared perceptual access, and ecological attunement, offering new directions for multispecies heritage technologies.

HCI More-than-Human Interaction Hybrid Museological Spaces Cultural Heritage
DOI ↗ Publication ↗
Figure from Semantic Anchoring in Head-Mounted Display-based Museum Augmented Reality: A Mixed-Methods Study of User Interface Placement and Experience

HCII 2026

Semantic Anchoring in Head-Mounted Display-based Museum Augmented Reality: A Mixed-Methods Study of User Interface Placement and Experience

Yu Liu, Anna Behrens, Annika Bender, Sarah-Sophie Kiesow, Lea Trageser, Ulrike Spierling

Head-mounted display (HMD) augmented reality (AR) is increasingly used in museums, yet in-situ evidence on where to place and anchor UI controls remains limited. We compare body-anchored controls (hand-attached) and exhibit-anchored controls (placed on/near the artefact) in an AR experience for a life-size orca exhibit. Following a Design Science Research process, we iterated the system into two design packages and evaluated them in two consecutive field studies with independent visitor cohorts (N=46; 23 per round) using questionnaires, structured observation, and interviews. Both versions were perceived as accessible and acceptable, but showed distinct experience profiles. The body-anchored package was associated with more user-initiated interventions and movement-based exploration and tended to show higher self-reported spatial presence and gameful experience. The exhibit-anchored package foregrounded exhibit-centric controls and clearer closure and was accompanied by more stationary, observational use and more neutral presence and gameful ratings. These findings indicate a design-package-level trade-off between semantic coherence and interaction initiative, suggesting that museum AR should preserve action readiness and mobility while keeping controls meaningfully tied to exhibits.

HCI HMD-AR Spatial UI UI Anchoring
Figure from Parameterized Head-Mounted Augmented Reality Design Patterns for Cultural Heritage: An Empirical Study

HCII 2026

Parameterized Head-Mounted Augmented Reality Design Patterns for Cultural Heritage: An Empirical Study

Yu Liu, Ulrike Spierling

Head-mounted display-based augmented reality systems enrich museum and cultural heritage visits, yet designing intuitive and reusable interactions in diverse exhibit settings remains challenging. This paper presents a set of domain-specific interaction design patterns that incorporate explicit design parameters, developed through a Design Science Research approach and iterative prototyping in a natural science museum. We identify three core AR interaction patterns, namely Point of Interest Guide, AR Experience Indicator, and AR Experience Presenter, which address common visitor needs such as exhibit navigation, content discovery, and information exploration. Each pattern is documented using a five-part template (Name, Problem, Context, Use When, Solution) and can be customized via parameters such as animation speed, activation mechanism, and interface positioning. Empirical validation using adapted SUS- and NASA-TLX-inspired 5-point questionnaires and qualitative feedback demonstrates the robustness of each pattern and highlights how certain parameter adjustments influence usability, perceived workload, and parameter-dependent preferences in museum-like settings, providing a flexible and empirically grounded roadmap for creating immersive, visitor-centric AR experiences in cultural heritage settings.

HCI HMD-AR Design Patterns Cultural Heritage
Figure from Comparing Flow, Narrative, and Spatial Immersion in an HMD-AR Serious Game for Museum Learning

European Conference on Games Based Learning (ECGBL), 2026

Comparing Flow, Narrative, and Spatial Immersion in an HMD-AR Serious Game for Museum Learning

Yu Liu, Michael Lurz, Ulrike Spierling

Head-mounted display augmented reality (HMD-AR) can enrich museum-based game-based learning by situating interactive digital content in relation to physical exhibits, but may also distract visitors or shift attention away from interpretation. This paper investigates how flow, narrative, and spatial immersion relate to appreciation-related outcomes in an HMD-AR serious game for museum learning. We designed a coral-reef protection game in three variants, each foregrounding one immersive quality. In a between-subjects field study with 144 valid museum visitors, participants experienced one variant and completed questionnaires and short interviews. Results showed a differentiated pattern: the flow-enhanced variant showed the clearest alignment between design emphasis and perceived immersion, while the spatial-enhanced variant highlighted a tension between environmental richness and awareness of the surrounding museum space. Narrative immersion was associated with perceived meaning and educational value, whereas post-experience engagement intentions were mainly shaped by prior engagement. We argue that HMD-AR museum games should align specific immersive qualities with interpretation, attention, and meaning-making rather than maximize immersion as a generic goal.

HMD-AR Serious Games Immersion Museum Learning

2025

Figure from Design Patterns for Playful Augmented Reality: Enhancing Cultural Heritage Engagement with Game Mechanics

ACM Designing Interactive Systems Conference (DIS), 2025

Design Patterns for Playful Augmented Reality: Enhancing Cultural Heritage Engagement with Game Mechanics

Yu Liu, Manuel Feller, Ulrike Spierling

Emerging advancements in augmented reality (AR) offer substantial potential to enrich museum exhibits with interactive and engaging content, yet head-mounted AR devices pose usability challenges for novice users. This study investigates how three distinct game mechanics—jigsaw puzzles, object catching, and spatial drawing—can address these challenges in a natural science museum setting. Implemented on Microsoft HoloLens 2, these mechanics were designed to enhance visitor engagement, learning, and immersion, while also contributing to the ongoing refinement of an interaction design pattern language tailored for AR applications. A structured evaluation framework combining usability, cognitive load, immersion, and engagement metrics suggests that each mechanic can improve user participation and comprehension of exhibit content, despite environmental distractions and gesture-recognition errors. Additionally, our findings illuminate how integrating game mechanics within a pattern language can guide developers in creating playful, instructive interactions for cultural heritage contexts. By situating the developed prototypes within a design pattern framework, this research demonstrates both theoretical and practical advances in gamified AR experiences, providing tangible guidelines for adapting and personalizing museum exhibits and informing similar applications in education and other cultural heritage contexts.

Interaction Design HMD-AR Game Mechanics Cultural Heritage
DOI ↗ Publication ↗
Figure from Playful Locative Interaction in Museums and Exhibitions with Immersive Augmented Reality

JCSG 2024, LNCS 15259, Springer, 2025, pp. 247–262

Playful Locative Interaction in Museums and Exhibitions with Immersive Augmented Reality

Yu Liu, Manuel Feller, Ulrike Spierling

Head-mounted augmented reality (AR) technology enhances the museum experience by immersing visitors in heritage content. Location-based applications attract online audiences to physical sites, which is also crucial for the dissemination of cultural heritage. This study develops and evaluates specific interaction design patterns to address typical challenges of physical navigation and interaction with AR in a museum, focusing on usability and the enhancement of visitor enjoyment through adding game elements. By prototyping and testing in both simulated and real-world museum environments, the research confirms that these patterns can enhance playfulness. The findings advocate for tailored interaction patterns that enhance usability and engagement, supporting the wider adoption of AR in cultural heritage domains.

HMD-AR Locative Interaction Playfulness Cultural Heritage
DOI ↗ Publication ↗

2024

Figure from Considerations for Cross-device Augmented Reality Interaction Patterns in the Field of Cultural Heritage Mediation

INFORMATIK 2024, Gesellschaft für Informatik (GI), pp. 871–882

Considerations for Cross-device Augmented Reality Interaction Patterns in the Field of Cultural Heritage Mediation

Yu Liu, Ulrike Spierling

In museums and exhibitions, augmented reality (AR) technology increases visitor engagement by providing immersive experiences. Using an example in a natural history museum, we discuss how interaction patterns specifically developed for a head-mounted display can also be applied to hand-held devices. The aim is to use the same content on different platforms. The interaction patterns discussed are spatial guidance with following a virtual avatar, the indication of AR content in space and its activation, the control of sequential explanations, and the interaction with annotations such as precisely applied labels. Cross-device solutions and patterns are compared.

Cross-device AR Interaction Patterns HMD-AR Cultural Heritage
DOI ↗ Publication ↗
Figure from Demonstration of Playful Interaction Patterns for Head-mounted Device-based AR Experiences in Cultural Heritage

Eurographics Workshop on Graphics and Cultural Heritage (GCH), 2024

Demonstration of Playful Interaction Patterns for Head-mounted Device-based AR Experiences in Cultural Heritage

Yu Liu

This proposal explores AR applications with engaging interaction patterns to provide immersive and entertaining museum experiences. Focusing on cultural heritage, this demonstration showcases innovative interaction patterns for navigation and exhibit activation. The navigation interaction pattern uses dynamic visual and auditory cues, while the exhibit activation interaction pattern incorporates playful orca-inspired elements. Implemented in Unity3D and deployed on HoloLens 2, the demonstration effectively balances usability and entertainment, enhancing visitor engagement and showcasing new AR possibilities for museums.

HMD-AR Interaction Patterns Playfulness Cultural Heritage
Figure from Creating Location-based Visual Presence Experiences for Museums with Immersive Technology

Eurographics Workshop on Graphics and Cultural Heritage (GCH), 2024

Creating Location-based Visual Presence Experiences for Museums with Immersive Technology

Ulrike Spierling, Jessica L. Bitter, Noura Kräuter, Yu Liu

This Poster (accompanying demos) presents an overview of creative approaches and considerations in the design research process of exploring different feelings of immersion and presence through Augmented Reality with headmounted devices in a museum. It shows ongoing work in the EU-funded project LoGaCulture (Locative Games in Cultural Heritage), in particular in the practical research that develops case studies for a Natural History museum. We summarise design criteria, concepts and prototypical implementations regarding the qualities of presence, playfulness and usability.

HMD-AR Presence Immersion Cultural Heritage
DOI ↗ Publication ↗

2023

Figure from Evaluating Interaction Challenges of Head-Mounted Device-based Augmented Reality Applications for First-time Users at Museums and Exhibitions

HCII 2023, Springer, pp. 150–163

Evaluating Interaction Challenges of Head-Mounted Device-based Augmented Reality Applications for First-time Users at Museums and Exhibitions

Yu Liu, Jessica L. Bitter, Ulrike Spierling

Many museums are seeking opportunities to apply head-mounted device (HMD)-based augmented reality (AR) technology to their exhibitions. However, due to the novel interaction method of HMDs, visitors are usually first-time users who find it difficult to gain enjoyment from using HMD-based AR applications directly. Thus, we argue that it is crucial to understand what has prevented visitors from experiencing the convenience of HMD-based AR applications. There are two main contributions to computer science in this paper: first, we introduced an HMD-based AR prototype with three interaction patterns that are made based on a natural science exhibition; Second, we proposed a usability test method to evaluate the HMD-based AR prototype to identify potential interaction problems in the exhibition sector. The results show that the suggested usability test method is effective in identifying interaction challenges and it helps media developers to improve AR experiences for first-time users at exhibitions and museums.

HMD-AR Usability First-Time Users Museums
DOI ↗ Publication ↗
Figure from Newbie Guides for Omnidirectional Guidance in Head-Mounted-Device-Based Museum Applications

HCII 2023, CCIS 1834, Springer, pp. 478–485

Newbie Guides for Omnidirectional Guidance in Head-Mounted-Device-Based Museum Applications

Yu Liu, Yan Huang, Ulrike Spierling

Novel digital museum guides using omnidirectional cues through head-mounted devices (HMDs) have been presented in previous research. However, due to the novel interaction technologies of HMDs, today's visitors are not yet familiar with these types of guides. We developed and tested instructions ("newbie guides") to help first-time users learn novel interaction patterns of HMDs, inspired by those used in website design or game tutorial design. First, we created three basic example patterns to direct attention using visual, auditory, and a combination of visual and auditory cues. Second, we created a newbie guide as an example showcase for each guidance pattern based on seven design criteria. The newbie guides were tested and evaluated in an exhibition situation, and the results indicate that the seven design criteria can assist HMD-based AR developers in creating effective newbie guides to support first-time users learn omnidirectional guidance patterns in the museum and exhibition context.

HMD-AR Omnidirectional Guidance First-Time Users Onboarding
DOI ↗ Publication ↗
Figure from Chances and Limitations of Immersive Augmented Reality for Game-based Learning in Museums

European Conference on Games Based Learning (ECGBL), 2023, pp. 643–650

Chances and Limitations of Immersive Augmented Reality for Game-based Learning in Museums

Ulrike Spierling, Jessica L. Bitter, Yu Liu, Thorolf Müller

This article discusses opportunities and limitations of using immersive augmented reality (AR) with head-mounted displays for gaming and education in museums. We first report on a prototype application for HoloLens 2 that we developed and tested with 109 museum visitors in a recent project for an exhibition with large animal skeletons. The results show that the feedback from visitors was overall positive, while more steps are needed before it can be implemented in a museum routine. Based on the findings, we discuss hypotheses for the planned further development of the approaches towards game-based learning in the real environment of a natural history museum.

HMD-AR Game-based Learning Museums Cultural Heritage
DOI ↗ Publication ↗

2022

Figure from Human-Computer Interaction Patterns for Head-Mounted-Device-based Augmented Reality in the Exhibition Domain

ACM International Conference on Interactive Media Experiences (IMX), Doctoral Consortium, 2022

Human-Computer Interaction Patterns for Head-Mounted-Device-based Augmented Reality in the Exhibition Domain

Yu Liu

Head-mounted-device (HMD) based augmented reality (AR) applications bring new opportunities to the exhibition domain. However, first-time users such as exhibition visitors are not familiar with the interaction method of the HMD, which also brings challenges to exhibition authors while implementing HMD-based AR technologies. This research project focuses on the accessibility of HMD-based AR applications in the exhibition domain. It explores potential interaction patterns based on the technical features of the HMD for exhibition-related use cases. Both information system research and design research methods are applied while exploring interaction solutions for human factor challenges. Several prototypes will be created and iteratively tested and evaluated at the exhibition for generalizing effective and accessible interaction patterns. As a result, we will achieve a pattern-based interaction system as an artifact with integrated design knowledge. This will support the future development of AR authoring tools.

HMD-AR Interaction Patterns Accessibility Authoring
Figure from Follow the Blue Butterfly – An Immersive Augmented Reality Museum Guide

HCII 2022 Posters, CCIS 1582, Springer, pp. 171–178

Follow the Blue Butterfly – An Immersive Augmented Reality Museum Guide

Jessica L. Bitter, Ralf Dörner, Yu Liu, Linda Rau, Ulrike Spierling

We present our concept and prototypical implementation of an avatar guide that helps museum visitors to navigate to chosen points of interest. The prototype shows an animated butterfly with interactive behavior that is controlled by a state machine. The avatar’s path can be pre-defined at an authoring stage, by walking the space with the head-mounted display and by placing path nodes interactively, or it can be generated at runtime, in which case the whole area needs to be scanned and processed beforehand. As the app is iteratively redesigned and tested with first-time users for accessibility, we report on first lessons learnt.

HMD-AR Museum Guidance Avatar Guide Navigation
DOI ↗ Publication ↗
Figure from Supporting the Creation of Non-linear Everyday AR Experiences in Exhibitions and Museums: An Authoring Process Based on Self-contained Building Blocks

Frontiers in Virtual Reality, Vol. 3, 2022

Supporting the Creation of Non-linear Everyday AR Experiences in Exhibitions and Museums: An Authoring Process Based on Self-contained Building Blocks

Linda Rau, Jessica L. Bitter, Yu Liu, Ulrike Spierling, Ralf Dörner

The use of Augmented Reality (AR) has the potential to make everyday experiences exciting and educational. For example, AR can augment exhibits in museums with animated and interactive content. The creation of this content, however, is still facing challenges. To meet these, we employ reusable, pattern-based building blocks called AR nuggets. An AR nugget implements one application pattern in a small and self-contained piece of software to provide a complete solution for recurrent AR-based experiences. For example, in the application context of museums and exhibitions, we identify superimposition or visualization of echolocation as general patterns for AR use cases. AR nuggets encapsulate AR-specific knowledge and sophisticated functionalities to support authors and reduce the authoring task to tweaking existing templates to individual exhibits. To connect AR nuggets used in different exhibition rooms, we present novel AR nuggets that encapsulate the functionalities needed for creating a path between the exhibits. Additionally, we provide examples of AR nuggets that implement a virtual character that guides visitors to exhibits of interest. With this new set of AR nuggets, spatial connections can be authored, e.g., in the form of a guided tour with interactive narration. For this authoring task, we introduce an AR nugget manager that supports authors in creating and adapting multiple non-linear AR experiences. We illustrate our approach with the creation of an everyday AR application for a museum of natural history, share our experiences and discuss to what extent our approach can mitigate authoring challenges for everyday AR applications from a museum’s point of view.

AR Authoring Design Patterns Non-linear Experiences Museums
DOI ↗ Publication ↗

2021

Figure from Handheld vs. Head-Mounted AR Interaction Patterns for Museums or Guided Tours

INTETAIN 2020, LNICST 377, Springer, 2021, pp. 229–242

Handheld vs. Head-Mounted AR Interaction Patterns for Museums or Guided Tours

Yu Liu, Ulrike Spierling, Linda Rau, Ralf Dörner

In recent years, Augmented Reality (AR) technology has been adopted in various fields. The development of handheld devices (HHD) such as smartphones and tablets gives people more chances to use AR technology in their daily lives. However, AR applications using head-mounted devices (HMD) such as Microsoft HoloLens or Magic Leap provide stronger presence experiences than HHD, so that users can immerse themselves better in AR scenarios. While currently there already exist prototypical examples of HMD in museum contexts, widely used interaction patterns are not yet well established, although they would play an important role for accessibility by large user groups. This paper explores existing and potential interaction patterns for guided tours in museums, led by the question how to reconcile AR interaction patterns on HHD and HMD. We use an existing museum showcase for handheld AR from the project “Spirit” to transfer its interaction patterns to an HMD, such as the MS HoloLens. Technical constraints and usability criteria regarding the potential overlaps and applicability have been analyzed in this paper.

Cross-device AR HHD vs. HMD Interaction Patterns Cultural Heritage
DOI ↗ Publication ↗
Figure from A Nugget-Based Concept for Creating Augmented Reality

IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), 2021, pp. 212–217

A Nugget-Based Concept for Creating Augmented Reality

Linda Rau, Robin Horst, Yu Liu, Ralf Dörner

Creating Augmented Reality applications often requires substantial technical expertise and repeated implementation work. This paper introduces the concept of AR nuggets: small, self-contained and reusable AR applications based on recurring application patterns. Each nugget encapsulates objects, parameters and a target for anchoring virtual content in the real world, allowing authors to adapt an existing solution instead of creating an AR application from scratch. The concept aims to reduce the complexity of AR authoring and provide reusable building blocks that can be customized for different application scenarios.

AR Authoring Design Patterns Reusable Components Augmented Reality
DOI ↗ Publication ↗

2020

Figure from A Tangible Object for General Purposes in Mobile Augmented Reality Applications

INFORMATIK 2020, Lecture Notes in Informatics (LNI), Gesellschaft für Informatik

A Tangible Object for General Purposes in Mobile Augmented Reality Applications

Linda Rau, Robin Horst, Yu Liu, Ralf Dörner, Ulrike Spierling

Smartphones and tablets are common technologies within today’s private living environments. They are well-suited to serve as a platform for mobile Augmented Reality (AR). Tangible AR is a subclass of AR which includes tangible objects and can make interactions intuitive. With this, new options for human-computer interaction become available at home. Based on literature research and design rationale, we identify requirements that help to develop a tangible object which can intuitively be used as tangible user interface (TUI) for mobile AR applications. Users should be able to handle the tangible object comfortably. Additionally, it needs to be reliably trackable with today’s tracking algorithms. The tangible object should also offer affordances to the users. We strive to develop a single, versatile object that is usable in different application scenarios at home. Our approach is to design a tangible object that combines different surfaces and shapes to offer various affordances and interaction possibilities. A physical instance of this object can be created with a 3D printer. We argue that this allows users to trigger actions intuitively in an AR environment or to manipulate virtual content.

Mobile AR Tangible Interaction Tangible User Interfaces Interaction Design
DOI ↗ Publication ↗
© 2025–2026 Yu Liu CC BY-NC 4.0 Creative Commons BY-NC 4.0 License License