Published: Mar 01, 2024

mixed reality heightens art appreciation among museum visitors

Traditionally, museums have focused on preserving history and tradition, not just in their artworks and artefacts, but also in their communication with visitors. Institutions now face the challenge of engaging a growing generation of digital natives, and are looking at ways to provide innovative experiences to draw a wider and more diverse audience set towards art appreciation. To achieve this, NCS in partnership with National Gallery Singapore (NGS), developed the proof of concept “Beyond the Canvas: Raden Saleh's Forest Fire Immersive Experience” by tapping on the hardware capabilities of Meta Quest Pro, to bring static art pieces to life in a physical museum space.

A key challenge identified was giving exhibits a broader appeal, especially to visitors who are beginners in art appreciation. While these visitors may be keen on learning more about the displays, written wall text alone may not be the most optimal in having the audience connect with the artwork. Some visitors may feel discouraged or excluded if they are not familiar with art, and assume that they must know art techniques and history before being able to understand the exhibits. There has to be a barrier-free way to experience, consume and appreciate the arts.

With a holistic approach encompassing Customer Experience Strategy, Experience Design Behavioural Science, and on-site User Testing, NCS developed the “Beyond the Canvas: Raden Saleh's Forest Fire Immersive Experience” proof of concept that allows paintings to come alive for visitors to interact with the exhibits in new and surprising ways. The team comprises of designers, developers, product managers and UX/UI experts from the NCS NEXT Digital Experience, NEXT Open Innovation and Xperience Works teams.

Visitors will be able to don a mixed reality headset and interact with static paintings through hand gestures in an intuitive way. Their experience comes by way of the following technologies and features (please check the video to witness the full experience):

• Artwork animation: Animation was overlaid on an actual artwork (Raden Saleh’s “Forest Fire”).  What were previously static figures of the tiger, ox and deer are now animated, and the effects were further heightened by a dynamic portrayal of a blazing fire accompanied with crackling sound effects and voice-over story narration. This converts the scene into a life-like spectacle.
• Spatial design: Using a spatial design platform for advanced mapping, we accurately measured and recreated the dimensions and features of the physical room within the virtual environment. This ensures a coherent experience where virtuality and reality coexist naturally.
• Narration delivered by the artist: Through an animated portrait, the artist unfolds his past experiences, leading visitors on a visual journey through his artistic process and sharing his influences. By providing narratives about the artists, their inspirations, and the meaning behind the artworks, visitors can forge a deeper emotional connection with both the art and the artist.
• Realistic 3D models: Key elements of the painting including the forest fire, tiger, and Gale (the AI docent ambassador guide) which were developed in detail and with realism. Lighting and shadows were finetuned to enhance authenticity, and photogrammetry techniques captured the textures and details from real-world objects.
• Interactive elements: The interactive design was built based on a blend of touch controls, eye tracking and gestures. We iteratively identified the most effective “tappable” buttons for the menu panel for realistic interactions with virtual objects, and the eye tracking technology follows the user’s gaze to facilitate interactions with Gale, the AI docent.

Snapshot of capabilities:

• Immersive and realistic: Visitors are transported into the heart of the artwork, with life-sized characters in close proximity, sound effects and other 3D elements
• Hyper-personalised: Gale, the guide powered by generative AI, is able to promptly answer questions that visitors may have when viewing the artworks.
• Interactive and informative: A magnifying glass feature allows visitors to explore varied artistic styles and highlights elements of an artwork for greater enrichment. Narratives about the artist and his or her inspirations and story behind the artwork help visitors forge a deeper connection with the art and its creator.

The successful testing of the “Beyond the Canvas: Raden Saleh's Forest Fire Immersive Experience” proof of concept paves a potential way forward for a more innovative and engaged museum visit among visitors of all backgrounds, from art connoisseurs to the general public.

The development team conducted user testing among eight participants within the exhibition spaces. After interacting with the prototype, users were highly positive about the immersive experience and cited such novelty within museums as a strong pull factor— and their willingness to pay an additional entry fee to experience it.

Overall, these users found that the experience increased their accessibility to the arts, felt an increased emotional connection to the exhibits and were appreciative of having a generative AI guide to answer their questions promptly.

“Modern art can be a challenging topic for most of our general visitors. The potential for mixed reality to help them see beyond the canvas, may begin to unlock their curiosity and appreciation for art, by understanding the intricate stories behind them.” – Kevin Lim, Director, Innovation & Technology, National Gallery Singapore