This Could Be How We Watch Movies in the Future

Marvel Studios What If…? — An immersive story is hard to describe. Part interactive game, part narrative film, part 3D comic book, this game puts you — the viewer? The player? — at the center of a narrative that reimagines the fate of the superheroes and villains of the Marvel Cinematic Universe.

To discover And if…? You have to wear an Apple Vision Pro headset, which is considered “mixed reality.” That means it incorporates both virtual reality, or VR (which transports you to a different world), and augmented reality, or AR (where other videos are superimposed on the room you’re in). The story alternates between AR scenes in which characters appear to materialize in the user’s real room, and a series of dizzying VR landscapes set in the multiverse.

Some sequences involve passive viewing, such as one in which supervillain Thanos is put on trial for theft. But there’s also plenty of interactivity: a character explains how to use hand gestures, such as making a fist, to defend against enemies and cast magic spells.

It’s a very different experience than what you traditionally get on TV or in the movies, and industry insiders believe it will change the face of entertainment.

“It’s kind of creating a new canvas,” he said. Shereif Fattouh of ILM Immersive, executive producer of the immersive version of And if…?.

Although interactivity is at the heart of the experience, viewers can choose not to use it.

“There are a lot of traditional gamers who really want to shoot things,” Fattouh said. “And there are people who don’t play video games and just want to see a great story.”

Catering to a wide variety of tastes

Gamers have been using virtual reality systems for decades. But in the last decade or so, newer headsets with more powerful graphics and motion-tracking technology have begun to expand their audience.

Today’s entertainment offering caters to a wide variety of tastes. For example, Meta headset users can sit down with friends to watch an NBA basketball game through its Xtadium app, explore a haunted Irish castle in The Faceless Lady, a live-action VR horror series, or attend pop star Sabrina Carpenter’s recent immersive VR concert.

“Watching your favorite basketball game or seeing your favorite superhero is a completely different experience (in VR),” said Jason Thompson, creator and host of The Construct, a VR-focused YouTube channel. “It can’t really compare to watching things on a flat screen.”

Thompson said he uses apps like Bigscreen to watch traditional TV shows and movies with his headset. If he wants, it can become a social experience; users can chat with others during a viewing party or mute them if the conversation gets too intense. They can also change their environment, so that a living room couch turns into what appears to be a luxurious movie theater, complete with virtual popcorn. Thompson said he sometimes watches movies in Bigscreen’s bedroom, lying flat on a bed.

“The screen is actually mounted on the ceiling,” Thompson explains. “And you have to lie down to see it.”

Lying down to watch VR content isn’t just a fantasy. It also has a practical function. Most of today’s headsets are heavy and awkward to wear. Thompson said lying down to watch takes pressure off your head and neck.

“For virtual reality to excel, it has to become comfortable,” he said.

An industry that finds its marks

Tech players are working on it.

“Over the years, we’ve been improving the technology, making it easier to set up, more transparent to what you’re doing on other devices, and we’re seeing more and more people adopt it,” said Sarah Malkin, director of immersive entertainment at Meta, the current market leader in consumer VR. “We knew we were going to invest in what is essentially the future of computing, and it was going to take time.”

Malkin says Apple’s entry into the market (the Apple Vision Pro launched in February) is a good sign that headsets are becoming more mainstream, even if they cost anywhere from $300 to more than $3,000. However, Apple and Meta don’t disclose specific sales figures, so it’s hard to know for sure how the market is shaping up.

“The key to this market is consumer adoption,” said Ben Arnold, a consumer technology analyst at market research firm Circana. “Because that’s what makes app development more attractive.”

Adoption depends on both technology and content

Filmmakers say the technology also needs to be adapted to creatives, so they can tell better stories in virtual reality.

“The basic interaction methods have not yet been determined,” said Eugene Chung, VR film writer and director at Penrose Studios, the company behind several VR films, including Arden’s Awakeninga post-apocalyptic ocean adventure that won the inaugural Lion Award for Best Virtual Reality at the Venice Film Festival in 2017. “It should be as natural as using your iPhone. And we’re a long way from that.”

Chung said it’s easy for people to get frustrated with most current VR offerings for TV and film because users don’t know where to direct their attention in a given scene, or they want to fully interact with a character but often can’t.

“You see things happening, but things don’t react the way you think they will,” Chung said. “For example, you can’t just walk up to a character and talk to them about Shakespeare or ask them what they had for lunch.”

Still, he’s excited to continue exploring the creative potential of this new medium, especially as many of today’s young people are growing up as virtual reality natives.

“I have no doubt that this will be the future of all entertainment and, indeed, all computing,” he said.

Jennifer Vanasco I edited both the audio and digital versions of this story. Isabelle Gomez SarmientoI mixed the audio version.

With thanks to Will Mitchell and James Mastromarino