Author’s Note: Several months ago, putting on an Augmented Reality (AR) headset was an uncomfortable nuisance for museum visitors. Now, it’s a potential health risk. The COVID-19 pandemic has only amplified the need for museums to rapidly develop the kind of frictionless AR experiences discussed below. While I was originally skeptical of AR experiences that require visitors to bring their own smartphones, this is now a valid way to deliver “touchless” interactive content both at home and in museums. Recent technological breakthroughs finally make it possible to view AR content on a smartphone simply by clicking a picture; no app needed. Still, museums must prioritize inclusivity and provide enriching experiences for those without smartphones in their pockets.

---

Almost ten years ago, an article in Museum magazine asked whether augmented reality (AR) was “the ultimate museum app.” With AR’s ability to nondestructively layer context around existing objects and democratize the curatorial process by allowing visitors to display ​their own commentary​ on museum walls, the answer seemed like a resounding “yes!” But the article cautioned that we should probably wait until the technology of AR caught up with our sci-fi expectations of floating data holograms. Indeed, while there has been rapid technological improvement over the last decade, driven by the smartphone industry and the mainstreaming of AR with apps like Pokemon Go​, there is still a lot we need to learn about this technology before it is adapted wholeheartedly by the museum community.

For example, we still don’t even know if visitors find AR more interesting and educational than traditional museum labels. And if they do, are the gains in learning outcomes and engagement justified given the high staff costs for developing and deploying AR experiences? Here at the Natural History Museums of Los Angeles County ​(NHMLAC), which includes the famous ​La Brea Tar Pits​, we’re rigorously investigating how emerging technologies like AR and virtual reality (VR) can be effectively used in museum environments. NHMLAC is particularly well situated in this field given our proximity to leading AR/VR researchers at the University of Southern California (a major collaborator on several grants), innovative tech firms in Silicon Beach, and the immersive entertainment industry born out of Hollywood and theme parks like Disneyland.

Throughout our research, we’ve found that the primary reason AR fails in museums is that it is not presented as a frictionless experience to visitors. AR is used more often as a high-tech gimmick than something that blends seamlessly into the exhibition experience, allowing visitors to engage in content without any tutorial or downtime to adjust equipment. Buzzwords like “frictionless” are bandied about constantly, but how do we actually evaluate whether an experience meets this requirement? We developed the following simple rubric.

The frictionless AR rubric

Frictionless AR is:

Immediate

The experience does not require any downtime to wait in line, gear up, or download an app. The experience either begins as soon as visitors approach or is already running in the background when they do. They can engage or disengage from the experience quickly and at will.

Intuitive

The experience doesn’t require the user to read complicated instructions or learn complex new interfaces. Intuitive, naturalistic behavior should be all that is required to interact with the augmented world.

Inclusive

The experience should be both comprehensive and inviting. “Comprehensive” means it is self-contained and does not require the visitor to bring along her own external devices or software. “Inviting” means it can be enjoyed by everyone*, whether they are young or old, standing or in a wheelchair. It is not isolating and does not require families or friends to unduly split up their shared experience.

*Note that although AR is often presented as visual technology, augmentation can occur through any or all of the five senses and AR ​audio​, ​touch​, and ​smell experiences have already been produced.

Is current technology frictionless?

Perhaps counterintuitively, the bleeding edge of AR technology—headsets like the ​HoloLens​ or MagicLeap​—often fails this rubric. Although technologically impressive, these headsets are still cumbersome and complicated to use for the novice visitor.

For example, in ​the Petersen Automotive Museum‘s ​Hollywood Dream Machines​ exhibition, visitors can use HoloLens headsets to view sci-fi cars augmented with visually striking holographic tour guides and labels. The high-tech headsets are integrated thematically with the futuristic cars on display, and the interface for the experience is very intuitive. However, several logistical factors add friction. Despite a well-run virtual queuing system, there is often a backlog of headsets, which require multiple professionals to charge, put on, boot up, and constantly troubleshoot them. Traffic flow patterns in this popular gallery also dictate that visitors be throttled individually through tour stops, forcing members of the same party to finish up to ten minutes apart from each other. Previously, the Petersen had linked together HoloLenses so that visitors enjoyed the same visuals simultaneously, adding significantly to inclusiveness. Although staff at the Petersen admit headsets may not currently provide frictionless AR, the tens of thousands of visitors who have already enjoyed their innovative experiences prepare the museum for a near future when AR headset technology has matured.

Smartphone-based AR, while probably the most popular and familiar with consumers, also fails to meet our frictionless standards. Because no single AR company or app controls the market, visitors can rarely just hold up their phones and activate an AR experience. Instead, they must spend significant time and data (which often means money) to search for and download a third-party or museum-specific app to even see a preview of the experience. You might be asking, “How hard can it really be to download an app?” Given the abysmal ​6 percent average adoption rate​ of museum apps by visitors, the answer is clearly “too hard.” And the reasons can extend beyond mere inconvenience. At La Brea Tar Pits, for example, many of our visitors are international tourists, who need their phones to navigate around a large, unfamiliar city and are loath to waste battery power and expensive international data plans on trivial AR experiences that quickly burn through both. The uncertain payoff of a museum’s phone-based AR experience simply does not justify the certain costs for most visitors.

It’s possible to get around this reluctance with apps, as The Tasmanian Museum and Art Gallery​ did by commissioning ​Handbuilt Creative ​to develop a frictionless AR experience for its ​Dinosaur Evolution​ exhibition. In the exhibition, visitors could use tethered iPads already running the AR app in the gallery to see a stunningly photorealistic ​Psittacosaurus ​skeleton come to life. The experience was immediate, intuitive (only pointing was required), and inclusive (groups of visitors didn’t need anything to view the experience together). After experiencing the payoff themselves, visitors could easily see the value proposition for downloading the AR app to view on their own phones and share with friends. Thanks to a frictionless deployment, within weeks, this sticky AR experience had been seen by one out of eight Tasmanians. Millions more experienced the activation online.

Surprisingly, older technologies actually tend to perform better on the frictionless rubric. Magic mirror kiosks, often driven by the now-defunct Xbox Kinect, function like normal mirrors but layer on content to a visitor’s reflection, to varying degrees of augmentation. Many magic mirror kiosks in history museums allow visitors to virtually try on historic garments or armor. The ​Bakeru​ exhibition at ​Japan House, for instance, ​completely replaced visitors’ bodies with those of northern folk monsters, inviting participants to sway and jump to spread seeds and bring in good harvests. As long as the interface is simple and intuitive enough, magic mirror setups are close to frictionless: easy to approach, use, share, and leave. Few museum displays so consistently produce immediate me-to-we experiences.

Projection mapping is an even older technology where virtual augmentation is projected directly onto an object, allowing multiple visitors to simultaneously see the augmentation without ever raising their phones or strapping on headsets. In one award-winning example, ​Ideum​ encouraged visitors to cast different patterns onto a large clay water jug. Even those visitors not actively choosing patterns to display could walk completely around the pot to appreciate the artwork, which scored high marks on immediacy for the experience. In another example, ​The Power of Poison ​exhibition at the American Museum of Natural History projected virtual pages of an ancient tome onto the physical pages of a blank book. Flipping a page of the physical book also ​flipped the virtual page​, the epitome of an intuitive interface. Projection mapping need not even be interactive to be engaging and frictionless. One of our most popular ​exhibits​ at NHMLAC is an overhead projection of Interstate 5—the freeway which Los Angeles’ most famous mountain lion, P-22, had to cross to reach his territory. The virtual freeway is crammed daily with children trying to Frogger their way through oncoming traffic, reenacting P-22’s perilous journey. The simplicity of the design is key to its popularity; gamifying the experience with points, interactive wrecks, and immersive headsets would kill an otherwise frictionless experience.

Proponents of AR headsets stress that they provide a higher level of immersion than other technologies, but high immersion can still be achieved by combining different low-friction techniques together. For instance, The ​Moesgaard Museum​ combined a magic mirror, projection mapping onto a physical book, and good old-fashioned set design to give visitors the experience of running a medieval town counsel in a candlelit ale house. All the visitor had to do was sit down at the oak table and raise a hand to add their vote. It was immediate, intuitive, inclusive, and totally immersive—no headset required.

The promise of AR viewfinders

Admittedly, these older technologies do not work for every context. For instance, projection mapping and magic mirrors are horrible solutions for augmenting large outdoor features like sculptures or historic buildings during daylight hours. This is where the still-developing technology of AR viewfinders shows the most promise. AR viewfinders work like the coin-operated binoculars so ubiquitous at national parks and historic sites, except they can layer on virtual content to the real, physical landscape being viewed. Depending on how much of this real landscape is augmented, the viewfinders could shift from AR to full immersive VR. Self-contained and intuitive to use, viewfinders could provide a frictionless alternative to phone-based AR for large outdoor sites like battlefields or zoos.

We were particularly interested in using AR viewfinders at La Brea Tar Pits because, unlike most paleontology museums, all our fossils come from dig pits immediately surrounding the museum. The fossils visitors see come from Ice Age animals that lived and died right here in Los Angeles, under our feet. This sense of place is incredibly important to the Tar Pits experience and was something we wanted to emphasize through any digital experience we created. Less philosophically, AR viewfinders could help immensely with visitor wayfinding. The museum at La Brea Tar Pits sits in a large, confusing campus of winding paths that connect three separate museums and multiple outdoor and indoor exhibitions and active dig sites. Many visitors who think they have seen the “Tar Pits” unknowingly walk right past our museum.

Fortunately for us, local inventor Ben Sax had developed a prototype AR viewfinder called the Perceptoscope​ and needed an institutional partner on his NSF grant to help him develop the technology further. Ben’s open-source/maker mentality jibed well with our desire to rapidly prototype and test different forms of AR at our museums, and a collaboration was born.

The Perceptoscope operates more or less like a HoloLens on a stick, using pass-through optics to project virtual features onto a real view of the landscape, rather than a flat digital screen as with phone-based AR. Its advanced optics provide true binocular vision and a wide field of view so that augmented additions, say a saber-toothed cat, appear like they are actually walking far off in the distance—or rapidly bounding towards you. Also, unlike headsets and phones, the Perceptoscope doesn’t need to miniaturize its internal processors or waste computation cycles constantly figuring out where it is in physical space (it’s bolted into the ground). This potentially means higher fidelity AR graphics at a cheaper price. Most importantly, though, the Perceptoscope could be frictionless. How frictionless? We put it to the test.

Teaming up with Ben and AR developers ​AfterNow​, we created an astonishingly simple minimum viable product to test: a simulation of Ice Age animals like Harlan’s ground sloths and saber-toothed cats walking around the current-day La Brea Tar Pits. Through another NSF grant with the University of Southern California, we had already developed scientifically accurate animated models of many Tar Pits animals that we could use.

Ben’s team used drones to photogrammetry scan the entire Tar Pits campus, creating an accurate 3D model of every sidewalk and tree to ensure that our animals looked like they were walking on real ground instead of floating mysteriously in the air.

After several spot checks and prototyping sprints, we put the Perceptoscope out for four days in the park, unfacilitated and devoid of any instructions and signs. It turned out to be remarkably popular, and sensors in the unit reported that it was in use, on average, 79 percent of the time it was deployed. We randomly picked seventy-three people walking by the Perceptoscope to observe.

All of them stopped to examine the unit and look through it. A young child shouted to his dad, “Come and see this! It’s so cool.” A teenager, unsurprisingly, was harder to impress: “It’s stupid. Just animals.” Overall, though, the Perceptoscope was very “sticky.” Half of the people who tried it called someone else over to check it out. It was easy to use, too. Without any instructions, over 93 percent of people used it as intended. Even though it can freely rotate 360 degrees left or right, some people did spin it too far up or down. But this problem is easily corrected with a modified gasket.

Even this very simple experience showed the Perceptoscope approaches frictionless AR. It works immediately, without any gearing up or downloading. And the familiar form factor of a coin-operated binocular makes it very intuitive to use. The Perceptoscope is still far from inclusive, though. True, it is self-contained, but since we 3D-printed the components to make prototyping faster, it can’t remain outside unsupervised forever. Ben is currently working with a local foundry to bombproof the Perceptoscope so that it can withstand both the blistering SoCal sun and that most destructive of elements: five-year-olds. The Perceptoscope is also currently sized for the ergonomics of an average middle-schooler. We need to develop a way that anyone, including the very tall and wheelchair users, can comfortably use the unit. But these problems have been solved before with traditional viewfinders, so we don’t see any long-term obstacles in AR viewfinders like the Perceptoscope becoming a common part of any museum or cultural site’s engagement portfolio.

It will still be a while before AR is completely frictionless the way we imagine it in sci-fi movies, but that doesn’t mean we can’t provide visitors with engaging AR experiences right now. We recommend thinking “beyond the goggles” by using tried and true frictionless technology like magic mirrors and projection mapping. There is no reason not to consider even older analog tech, too. The centuries-old Pepper’s ghost illusion still does a good job overlaying holographic bodies onto the skeletons in our museum. And a sturdy glass matte painting can show tourists how ruined temples once looked without any need for Wi-Fi and batteries. If you are interested in chasing the bleeding edge, though, we recommend holding some pop-up experiments with AR viewfinders, a promising emerging technology that could provide high-fidelity, frictionless experiences both inside and outside the museum.

About the author:

Dr. Matt Davis is an Exhibition Developer for the ​Natural History Museums of Los Angeles County​, which includes ​La Brea Tar Pits​. As an expert on Ice Age fossils and science communication, he helps lead museum initiatives exploring the potential of emerging technologies like augmented and virtual reality.

 

Skip over related stories to continue reading article