how movies can change the plot in real time, responding to viewers’ emotions

Most movies offer exactly the same viewing experience. You sit down, the movie starts, the plot unfolds, and also you follow what’s happening on screen until the story ends. It’s a linear experience. My latest movie Before we disappear – a couple of pair of climate activists who seek revenge on the corporate perpetrators of worldwide warming – goals to change the viewing experience.

What makes my film unique is that it adapts the story to the emotional response of the viewer. Thanks to the use of a pc camera and software, the film effectively observes the viewer watching recordings of climate disasters. Viewers are implicitly asked to select a side.

I made a decision to use this technology to make a movie about the climate crisis to get people to really take into consideration what they’re willing to sacrifice to ensure a greater future.

Storytelling has at all times been interactive: traditional oral narrators interacted and responded to their listeners. Film directors have been around for nearly a century experimenting with interactivity – the last decade has seen an explosion of interactive content.

Streaming services give viewers the opportunity to select their very own adventure. However, leaving the viewer in control of the motion has long been a challenge: it conflicts with narrative immersion, in which the viewer is drawn into the world created by the story.

One of the most eminent recent experiments in interactive film, Netflix’s Bandersnatch, illustrates this clearly. This is where the motion ends to ask the user what to do next – to interrupt the flow of the story and actively engage the viewer. Solving the problem of breaking immersion stays a key query for artists exploring interactive film.

The movies I create and direct take a distinct route, using unconscious control to influence the film while the viewer is watching. My previous one brain controlled movies, The Moment (2018) AND The Disadvantages of Time Travel (2014), brain-computer interfaces (BCI) were used. These systems use computers to analyze electrical signals from the brainallowing people to effectively control the device with their minds.

Using brain data, viewers create unconscious edit film in real time – reinforcing the movies’ stories of sci-fi dystopia and the wandering, dreaming mind.

However, the BCI interface requires specialized equipment. For “Before We Disappear,” I wanted to use technology that was more accessible to viewers and would allow the videos to be shared online.

Narrative control

Before We Disappear uses an everyday computer camera to read emotional signals and instruct film editing in real time. To make this work, we wanted understanding of individuals’s reactions to the videos.

We ran a couple of studies examining emotions the filmmakers intend to evoke and the way viewers visually portray emotions while watching. Using computer vision and machine learning techniques offered by our partner BlueSkeye artificial intelligencewe analyzed viewers’ facial emotions and reactions to film clips and developed several algorithms to use this data to control the narrative.

Although we’ve observed that viewers don’t normally show much emotion when watching a video, BlueSkeye’s facial and emotional evaluation tools are sensitive enough to pick up sufficiently small differences and emotional signals to adapt the video to the viewer’s response.

The analytical software measures the movement of facial muscles and the strength of emotional arousal – that’s, the level of emotion the viewer feels at a given moment. The software also assesses the positivity or negativity of emotions – something we call “valence“.

We’re experimenting with different algorithms where arousal and valence data influences editing decisions in real time, causing the story to reconfigure itself. The first scene is the reference point against which the next scene is measured. Depending on the answer, the narrative will turn into one in every of roughly 500 possible edits. In Before We Disappear, I exploit a non-linear narrative that provides the viewer different endings and emotional journeys.

An emotional journey

I see interactive technology as a way to expand the filmmaker’s toolkit, to further tell the story, and to allow the film to adapt to the individual viewer, which challenges the director and distributes his power.

However, emotional responses can be misused or have unexpected consequences. It just isn’t difficult to imagine a web-based system presenting only content that evokes positive emotions in the user. This can be used to create an echo chamber where people only see content that matches their preferences.

Or it could possibly be used for propaganda purposes. What did it seem like in the Cambridge Analytica scandal? large amounts of non-public data were downloaded from Facebook and used in political ads.

Our tests goals to spark discussion on how user emotion data can be used responsibly with informed consent, while allowing users to control their very own personal data. In our system, data is analyzed on the user’s device and never, say, in the cloud.

Big business, big responsibility

Unconscious interaction is big business. Platforms like ICT Tok AND Youtube use evaluation of users’ past interactions on platforms to influence the latest content they see there. Users are usually not at all times aware of what personal data is being created or stored, nor can they influence what algorithms present to them next.

It is very important to create a system in which recipient data just isn’t stored. Viewer videos and facial features data mustn’t be transferred or analyzed anywhere aside from on the player device. We plan to release the film in the type of an interactive application that might be aware of the potential misuse of user data and can secure personal data on the device on which it’s going to be viewed.

Adaptive movies provide an alternate to traditional choose-your-own-adventure storytelling. When the story can change based on viewers’ unconscious responses moderately than intentional interaction, their attention can be focused on the story.

This means they can enjoy a more personalized movie experience. It seems that in the twenty first century, old storytelling traditions can still teach us loads.