“It turns out that the killer application for virtual reality is other human beings. Build a world that people want to inhabit, and the inhabitants will come.” —Charles Stross

Augmented reality (AR) and virtual reality (VR) are portals to other places and times (real or imagined). They can be enchanted mirrors, offering reflections of the world as it could be—more exciting, interesting and informative. They can be magic wands, revealing an invisible world hidden around us. These technologies may generate huge profits for some while impoverishing others; they can enhance empathy while creating new forms of discrimination. AR and VR hold promise and peril for museums as well. Why go to a museum when you can just don a headset to experience fabulous sights, sounds, touch—and hang out with friends—without leaving home? On the other hand, with such a direct, high-impact platform to reach people where they live, how many converts can museums court to visit IRL (in real life)? If VR and AR experiences become both affordable and widely accessible, museums will need to sharpen their positioning and value proposition with their communities.

First let’s map the relationship between these digital cousins. Virtual reality refers to media that transport a user to a wholly digital, simulated environment—an imaginary world, or a recreation of the real world, present or past. The most highly populated virtual realm is World of Warcraft, where about 7 million users roam the plains, mountains and oceans of Azeroth. The most widely known nongaming virtual world may be Second Life, launched in 2003, whose real estate includes a few score (lightly attended) virtual museums. The current focus in VR, gaming or nongaming, is on making these worlds immersive through new display technologies that create the illusion that a user is actually in the midst of a fully realized, 3D world. The latest buzz is about Oculus Rift, an immersive virtual reality headset for gamers that Facebook purchased in 2014 for over $2 billion, but Samsung, Google and Sony are racing to introduce sophisticated gear as well. As in previous decades, competing hardware platforms create challenges for the broad adoption of VR, as content is often locked into just one platform.

Augmented reality, by contrast, adds digital information to the real sensory input from the world around us—pasting content and information on top of what we see or hear, and at its most sophisticated, interacting with and adapting to the user. Early versions of museum AR included camera overlays using Layar and, during its brief “Explorer” period, Google Glass. The demo videos for Microsoft’s soon-to-be-released HoloLens show off its ability to create a shared holographic work environment—enabling multiple users to see and manipulate the same imaginary objects. Up-and-comers in this category also include the somewhat mysterious Magic Leap—a product so far only glimpsed in a tantalizing GIF—that promises to create superrealistic AR. (It’s rumored to work by projecting augmented illusions directly into the user’s cornea.)

Skip over related stories to continue reading article

VR and AR have made huge strides in the past couple decades. The first rig, created back in 1968, was dubbed “The Sword of Damocles” because it was so heavy it had to be suspended from the ceiling above the user. Now the software creating these experiences is quickly becoming more sophisticated, while the gear itself becomes smaller, lighter and less expensive. This rapid evolution is largely driven by the prospect of huge payoffs from the lucrative world of online gaming, but games have a venerable history of being hijacked for other purposes. In fact, many modern games are designed from the outset to allow and encourage such ”modding.” Minecraft, for example, a video game that challenges players to build structures, villages, countries, even whole worlds, out of cubes, has been used for real-world urban planning and community input. (Denmark generated a 1:1 replica of their whole country in Minecraft to facilitate public use of geodata.) Cited as a perfect application for VR/AR rigs, Minecraft was the subject of Microsoft’s first public display of HoloLens.

AR/VR has been dogged by a number of persistent problems, but these barriers are falling one by one. It’s hard to trick the human brain into playing nicely with digital data, but companies are slowly overcoming the tendency of their gear to induce “virtual reality sickness.” Early adopters of Google Glass had to flip back and forth between focusing on whatever was being projected on Glass and on the real world, which made the user look perpetually distracted (as well as inducing eyestrain). The iOptik prototype, by contrast, lets users focus on the data and the distance simultaneously. While early headgear of any sophistication costs several thousand dollars or more, costs are coming down, and gaming-quality headsets are coming onto the market for a few hundred dollars. Google Cardboard (which can literally be made out of cardboard from open-source specifications), lets anyone turn their smartphone into a simple VR rig for less than 10 US dollars. AR and VR aren’t limited to sight and sound. Devices like Nintendo Power Glove (1989) and Microsoft Kinect (2010) give the user broad gestural control, and Oculus Touch functions like a working pair of virtual hands. The Tactical Haptics Controller creates a sense of friction and weight in the user’s hand, and “ultrahaptics” can use ultrasonic waves to induce a sense of touch from a distance. Eventually (as more than one scifi author has forecast), we may interact with digital worlds through direct neural inputs. The basic technologies to monitor brain waves have already been miniaturized and embedded in toys such as Mindflex (2009). AR and VR aren’t limited to sight and sound. Now we are pioneering noninvasive devices that enable users to move cursors with their thoughts and even control the muscles of another user from a distance.

In a neat example of life imitating art, many of these AR/VR projects are being inspired, funded, and developed with the help of writers. Neal Stephenson, chief futurist at Magic Leap, often sets his novels in near-future worlds shaped by ubiquitous AR and VR. Neil Gaiman (author of Sandman and American Gods, among many works of fantasy and fable) is an investor in WoofbertVR. And Woofbert’s work with museums was inspired, in part, by the classic scifi novel ReadyPlayerOne, in which the protagonist attends school via VR rig. Those of us who grew up enthralled by one of the many incarnations of Gene Roddenberry’s Star Trek are tickled to learn that Microsoft is testing proof of concept for RoomAlive— the first working holodeck.

Museum Examples

There are too many museum experiments in AR/VR to cover comprehensively in this essay. Here are a few of the most recent, organized by the Mooshme Matrix categories.

Me/Here

The Art++ augmented reality app at the Cantor Arts Center, Stanford uses image recognition to create a “digital halo” of supplemental multimedia information around a photo, painting or sculpture. The project was made possible by support from the Brown Institute for Media Innovation, a collaboration between Stanford and Columbia University dedicated to innovative storytelling.

Me/There

The Courtauld Gallery in London is one of the earliest adopters of WoofbertVR, an app that will profile collections from around the world. The Courtauld tour gets an extra popularity boost from narrator Neil Gaiman, who is also an investor in Woofbert. Woofbert represents a collaboration between technologists

We/Here

While Hololens (a device that provides shared AR experiences) is not yet commercially available, in March 2015 the Mondrian 3D Museum provocatively tweeted “Meeting with @microsoftabout #hololens #museums #3D, looking forward to making next generation of #education and #art in #realmuseums.” Stay tuned?

We/There

The Dallas Museum of Art (DMA) organized a performance by artist Jon Rafman that took place in the online virtual world Second Life (SL), in conjunction with the exhibit “Mirror Stage: Visualizing the Self After the Internet.” DMA audiences were invited to log in and attend, via avatar, a tour of hidden corners of SL, guided by Rafman’s online persona, “Kool-Aid Man.”

What This Means for Society

The surge in new AR/VR technology will both boost traditional businesses (like gaming) and create new business opportunities. The market for virtual reality content is projected to reach $5.4 billion by 2025, (Samsung, Facebook) and museums, with each bringing their strengths to the venture. The project intends to eventually provide virtual docents to interact with users. Glass HoloLens Cardboard Oculus here me there we while the hardware generates another $63 billion. On the other hand, there may be losers in the marketplace as well. Online shopping has already damaged place-based retail. How much more attractive will virtual shopping be when you can “see” yourself in a dress and “feel” the fabric? Or audition digital couches in your virtual living room (an option Ikea premiered in 2014)?

VR may revitalize some flagging sectors by providing a compelling and accessible platform for rich content. In 2015 The New York Times sent every subscriber—all 1.1 million of them—a free Google Cardboard headset, paired with a free app that gives readers access to a series of short immersive documentaries. One of their first films—The Displaced—charts the plight of three refugee children out of the 30 million currently displaced by conflict and persecution. Can VR help traditional journalism win back readers? Like newspaper subscriptions, attendance at traditional music performances has been dropping for years. The Los Angeles Philharmonic recently outfitted a truck (dubbed Van Beethoven) as a mini concert hall that travels LA, giving residents an immersive VR experience of attending the symphony—in some ways better, as it provides close-ups and perspectives you wouldn’t get as an audience member at an actual performance.

Researchers are building an impressive body of evidence for the “prosocial” benefits of virtual experiences. VR can help people understand their impact on the environment, reduce conflict by letting people inhabit the lives, situations and identities of others, and connect people with their future selves (hopefully prompting them to make better decisions today). Preliminary research suggests that the effects of virtual reality may last longer than those of traditional media such as reading or TV. This being so, VR could be an empathy tool used to unite our increasingly fragmented world. The Robert Wood Johnson Foundation is funding a large-scale, long-term study on using virtual reality to teach empathy. What could this look like? (Gary Marcus has suggested that augmented reality apps could foster empathy by superimposing information about a stranger’s hobbies and family background to remind us of each other’s humanity.) We are waiting to see how well these experiments work. Will Americans who view The Displaced be more sympathetic to the plight of international refugees? Will people who previewed the future of Marin Valley through the sea-level virtual reality project be more likely to support efforts that combat climate change?

VR and AR may play a major role in both formal and informal education. Google Expeditions is already offering teachers affordable kits that let students use Google Cardboard to take virtual field trips to the Eiffel Tower or the American Museum of Natural History. VR and AR can change the meaning of “immersive learning” by letting students dive into small-scale systems (cells) or the very large (galaxies). These technologies can also lower the risk and cost associated with practicing advanced skills such as surgery, engineering or space exploration. In 2015 Brown University debuted an immersive 3D virtual reality room called the YURT (YURT Ultimate Reality Theatre) to accelerate science through training and  experimentation. AR/VR can support lifelong learning as well. The startup ScopeAR is tackling this challenge by adapting AR for the Do-It-Yourself community. Where people now turn to YouTube for tutorials on almost anything, in the future you might “scopein” an expert to walk you through a DIY project, step by step, seeing what you see and showing you what to do.

The power of VR/AR can be dangerous too. One writer has envisioned how augmented reality could be used for “racial filtering”: to avoid seeing people who are “other,” or to fuel harassment by tagging people with “augmented reality warnings.” Virtual reality is not only more effective than traditional media in triggering empathy, it is also more effective in desensitizing both men and women to rape and making people more conscious of the gap between their perfect avatars and their real-world bodies.

What This Means for Museums

Barry Joseph, associate director of digital learning at the American Museum of Natural History, has neatly parsed the taxonomy of rapidly speciating AR/VR tech. His “Mooshme Matrix of Place-based Augmented Devices” (see page 26) ranks each platform along two axes: from here (enhancing the user’s surroundings) to there (transporting the user to a different space); and from me (personal/solitary experiences) to we (shared, social experiences). Joseph acknowledges the threat posed by we/there technology like Oculus Rift: if people can be social in immersive, inspiring virtual environments, why come to a museum? But he proposes that such experiences can generate deep interest and inspire people to seek information in real life. The proliferation of AR tech expands the world of BYOD (Bring Your Own Device) and the ways in which people can mediate their own visits. Joseph is most excited, however, by we/here AR technology like HoloLens, which he calls the “sweet spot of museum engagement.” What if visitors could see, handle, manipulate and share digital doppelgangers of real objects, or share the attentions of a docent avatar? “It is why people travel to museums in the first place,” Joseph points out, “to have a place-based, shared experience with their friends and family.”

Museums, along with print journalism and classical music, have been steadily losing market share to other pastimes. To Joseph’s point about inspiring curiosity, increasingly sophisticated AR and VR will heighten the impact museums can make as they push their content out into the world via these platforms. Can AR/VR experiences provided beyond museum walls help win new audiences?

AR and VR will provide new ways to share and access expertise. Not every museum can have an in-house conservator, packer and shipper or mount maker. What if even the smallest museum could  “scope in” an experience specialist to walk staff through the process of cleaning objects or other specialized tasks?

Museums Might Want to…

• Experiment with offering visitors AR and VR experiences, using existing or original museum related content, to better learn which tools afford the best in-museum experience, and upselling opportunities like 3D IMAX films. Content production for VR is still not easy or cheap. However, many institutions are working with emerging production companies and artists at below-market rates while the major funding source for content development remains advertising companies and hardware-specific deals. By partnering with companies launching VR and AR programs, such projects can be accomplished on a nonprofit budget.
• Adopt existing tech and adapt it to their purposes. Why not take the NYT’s lead? Give every museum member a Google Cardboard headset and release new content on regular basis: behind-the-scenes tours, a preview of the new special exhibit, a mini-doc of the last paleontology dig or the year’s highlights delivered in a tête-à-tête with the director as you sit (or poke around) in her office.

Additional Resources

Barry Joseph has explored the “Mooshme Matrix of Placebased Augmented Devices” in two posts on his blog www. mooshme.org. Part one introduces the classification scheme and introduces the major players in the emerging AR/VR marketplace. Part two refines the scheme and comments on the NYT Google Cardboard project.

The Virtual Reality Journalism Report (Tow Center for Digital Journalism, 2015) traces the history of virtual reality, presents a case study in VR journalism, analyzes the potential of the emerging technology and presents recommendations for journalists seeking to work in VR. The report is of great applicability to museums as well.

Some scifi novels exploring the future of AR/VR:

• Neal Stephenson, Snow Crash (Bantam Books, 1992), takes place in part in the Metaverse—Stephenson’s vision of a VR
  version of the Internet.
• Ernest Cline, Ready Player One (Broadway Books, 2011), explores how VR may transform the next generation of humanity.
• Cory Doctorow, For The Win (Macmillan, 2010), addresses the economic and human rights implications of the intersection of real and virtual worlds.