how movies can change the plot in real time, responding to viewers’ emotions

Most movies offer exactly the same viewing experience. You sit down, the movie starts, the plot unfolds, and you follow what’s happening on screen until the story ends. It’s a linear experience. My new movie Before we disappear – about a pair of climate activists who seek revenge on the corporate perpetrators of global warming – aims to change the viewing experience.

What makes my film unique is that it adapts the story to the emotional response of the viewer. Thanks to the use of a computer camera and software, the film effectively observes the viewer watching recordings of climate disasters. Viewers are implicitly asked to choose a side.

I decided to use this technology to make a film about the climate crisis to get people to really think about what they are willing to sacrifice to ensure a better future.

Storytelling has always been interactive: traditional oral narrators interacted and responded to their listeners. Film directors have been around for almost a century experimenting with interactivity – the last decade has seen an explosion of interactive content.

Streaming services give viewers the opportunity to choose their own adventure. However, leaving the viewer in control of the action has long been a challenge: it conflicts with narrative immersion, in which the viewer is drawn into the world created by the story.

One of the most outstanding recent experiments in interactive film, Netflix’s Bandersnatch, illustrates this clearly. This is where the action ends to ask the user what to do next – to interrupt the flow of the story and actively engage the viewer. Solving the problem of breaking immersion remains a key question for artists exploring interactive film.

The films I create and direct take a different route, using unconscious control to influence the film while the viewer is watching. My previous one brain controlled movies, The Moment (2018) AND The Disadvantages of Time Travel (2014), brain-computer interfaces (BCI) were used. These systems use computers to analyze electrical signals from the brainallowing people to effectively control the device with their minds.

Using brain data, viewers create unconscious edit film in real time – reinforcing the films’ stories of sci-fi dystopia and the wandering, dreaming mind.

However, the BCI interface requires specialized equipment. For “Before We Disappear,” I wanted to use technology that was more accessible to viewers and would allow the videos to be shared online.

Narrative control

Before We Disappear uses a regular computer camera to read emotional signals and instruct film editing in real time. To make this work, we needed a good understanding of people’s reactions to the videos.

We ran a few studies examining emotions the filmmakers intend to evoke and the way viewers visually portray emotions while watching. Using computer vision and machine learning techniques offered by our partner BlueSkeye artificial intelligencewe analyzed viewers’ facial emotions and reactions to film clips and developed several algorithms to use this data to control the narrative.

Although we have observed that viewers do not usually show much emotion when watching a video, BlueSkeye’s facial and emotional analysis tools are sensitive enough to pick up small enough differences and emotional signals to adapt the video to the viewer’s response.

The analytical software measures the movement of facial muscles and the strength of emotional arousal – that is, the level of emotion the viewer feels at a given moment. The software also assesses the positivity or negativity of emotions – something we call “valence“.

We’re experimenting with different algorithms where arousal and valence data influences editing decisions in real time, causing the story to reconfigure itself. The first scene is the reference point against which the next scene is measured. Depending on the answer, the narrative will become one of approximately 500 possible edits. In Before We Disappear, I use a non-linear narrative that offers the viewer different endings and emotional journeys.

An emotional journey

I see interactive technology as a way to expand the filmmaker’s toolkit, to further tell the story, and to allow the film to adapt to the individual viewer, which challenges the director and distributes his power.

However, emotional responses can be misused or have unforeseen consequences. It is not difficult to imagine an online system presenting only content that evokes positive emotions in the user. This can be used to create an echo chamber where people only see content that matches their preferences.

Or it could be used for propaganda purposes. What did it look like in the Cambridge Analytica scandal? large amounts of personal data were downloaded from Facebook and used in political ads.

Our tests aims to spark discussion on how user emotion data can be used responsibly with informed consent, while allowing users to control their own personal data. In our system, data is analyzed on the user’s device and not, say, in the cloud.

Big business, big responsibility

Unconscious interaction is big business. Platforms like ICT Tok AND Youtube use analysis of users’ past interactions on platforms to influence the new content they see there. Users are not always aware of what personal data is being created or stored, nor can they influence what algorithms present to them next.

It is important to create a system in which recipient data is not stored. Viewer videos and facial expression data should not be transferred or analyzed anywhere other than on the player device. We plan to release the film in the form of an interactive application that will be aware of the potential misuse of user data and will secure personal data on the device on which it will be viewed.

Adaptive films provide an alternative to traditional choose-your-own-adventure storytelling. When the story can change based on viewers’ unconscious responses rather than intentional interaction, their attention can be focused on the story.

This means they can enjoy a more personalized movie experience. It turns out that in the 21st century, old storytelling traditions can still teach us a lot.