One Measure of Happiness

An interactive Video Installation

Content
Content
Objectives
Concept

1. Installation
2.
The Video
3.
System Architecture
4.
Setup
5.
Project state (January 2003)
6.
Who we are

Detailed System Architecture

1. Data flow and timing

Figure 1: General Data Flow and Timing

2. High level Roles & Responsibilities

2.1 Sensation TBD
2.2
Perception TBD
2.3
Cognition
2.4
Emotion
2.5
Sprite Manager
Table 1:
Sprite Manager Selectors
Table 2:
Sprite Manager Events

3. Detailed Roles & Responsibilities

3.1 Sensation Details TBD
3.2 Perception Details TBD
3.3 Cognition Details
3.4 Emotion Details TBD
3.5 Sprite Manager Details
  3.5.1
Scene Selection
  Table 3 - Story Skeleton Phases
  Table 4 - Story Scene representation
  Table 5 - Sequence representation
  3.5.2 History
  Table 6 - History
  Table 7 – Memory Clips


Objectives

1.  To tell a coherent story interactively, fluently and cinematically through the interplay of three actors: a database-driven story model, an autonomous synthetic character, and an involved and active interactor.

2.  To create a compelling experience of intimacy between an interactor and a synthetic video character using familiar gestures through an affordable touch-screen interface.

Concept

1.  Installation

The installation is situated in a physical four-wall compartment. Interactors enter the compartment one at a time through a path of obstacles. Each obstacle involves a prohibition that must be transgressed in order to proceed. At the end of the path, a touch screen is embedded into a wall, with a prohibitive “Don’t Touch!” sign above it.

2.  The Video

The Video is composed of two composite layers: Foreground and Background. In the foreground layer we see a Close-Up shot of the protagonist, a young woman whose, face we can touch. The background layer is a back-projection of the protagonist’s thoughts, memories and feelings.

The protagonist invites us to touch her face. The way we touch, the location, manner and duration determines the level of intimacy and confidence we inspire in the protagonist. This, in turn, affects the way we progress through the story.

3.  System Architecture

The foreground and background layers are generated in real time by Macromedia Director, from an external clip database. The clips are assembled on the fly according to the program algorithm, which is comprised of three elements: A narrative and presentation manager, which is in charge of maintaining narrative progression, coherence and presentation; the protagonist, with her autonomous goals, emotions and behaviour; and an interaction manager, which monitors and interprets the interactor’s activity.

4.  Setup

5.  Project State (January 2003)

Software:
We are progressing in a modular mode. We’ve already completed the overall system architecture and algorithms and are now writing the first modules of the Sprite Manager in Macromedia Director Lingo. Cuncurrently, we are writing the pseudo-code of other modules, to be executed later.

Creative:
We have completed a synopsis and are now in an advanced stage of adapting it into a 30-page treatment. We expect to be casting by March 2003, in production by April 2003 and in a gallery near you by May 2003.

6.  Who we are

Investigators, producers and creative directors:
Noam Knoller & Udi Ben-Arie

Research Supervisor:
Amnon Dekel

Story and screenplay:
Galit Reuchman and Nelle Schaeffer

Architecture Engineer:
Maya Lotan

Software Manager:
Mirit Tal

Lingo Team Manager:
Gal Tushia

Lingo Team:
Udi Ben Arie & Noam Knoller


Detailed System Architecture

1.  Data flow and timing


Figure 1: General Data Flow and Timing

2.  High level Roles & Responsibilities

2.1 Sensation TBD

2.2 Perception TBD

2.3 Cognition

Evaluate and react to a gesture.

2.4 Emotion

Evaluates the user interactivity level, maintains the mood of the character, its attitude towards the user and the level of intimacy between them.

2.5 Sprite Manager

All selectors are responsible for collecting all necessary data for their selection process.

Memory Sprite

Character Sprite

Reaction Sprite

BG

FG

UFG

Select Memory & Speech Scene

Select Reaction Clip

Select Reaction Clip

Select Memory Scene

SelecMood Clip

Select Reaction Clip

Select Mood Clip

Select Mood Clip

Select Mood Clip

Select Reaction Clip

Select Mood Clip

Select Mood Clip

Table 2: Sprite Manager Events

Memory Sprite

Character Sprite

Reaction Sprite

User Activity

BG

FG

UFG

a scene with out a FG

Memory Scene

Mood clips

-

User Activity = No

(from Emotion)

Reaction Clip

Regular

Opacity<100%

Reaction (Type) = Regular

(from Cognition)

Interrupt

Reaction Clip:

Extreme

Opacity=100%

Reaction (Type) = Extreme

(from Cognition)

a scene with a FG

Memory Scene

Speech Scene

-

User Activity = No

(from Emotion)

Reaction Clip

Regular Positive

Opacity<100%

Reaction (Type) = Regular, Reaction (Tag) = Positive

(from Cognition)

Reaction Clip

Harassed reaction (u r bugging me!)

Opacity=100%

Reaction (Type) = Regular, Reaction (Tag) = Negative/Impertinent

(from Cognition)

Interrupt

Interrupt

Reaction Clip

Extreme reaction

Opacity=100%

Reaction (Type) = Extreme

(from Cognition)

3. Detailed Roles & Responsibilities

3.1 Sensation Details TBD

3.2 Perception Details TBD

3.3 Cognition Details
TBD reaction Connotation is the GestureArea + neg/pos

3.4 Emotion Details TBD

3.5 Sprite Manager Details

3.5.1 Scene Selection

Scenes are selected according to all the below, each weighted – the weight of each aspect can be changed by us.

  • Authentic to the character – The scene with highest authentic level derived from connotation to user gestures and the Association to character memories. TBD Might be a weight between different aspects listed below.

    • Connotation – A scene that has a connotation that fits the Reaction (Tag) and the Gesture (Area).
      (Reaction (Tag) à XXX Area Connotation)

    • Association – A scene that has association that fits the association of the last scene. TBD representation and process.

    • Start Mood - The scene Start Mood fits the Mood (Level)

  • Authentic to the story skeleton – according to the story phase.

  • Authentic to the Intimacy between the user and the character

    • Depth – The scene depth fits the I (intimacy Level). High depth scenes are presented only when high intimacy is achieved.

    • A drop in intimacy, reflected in low user activity, triggers Brooding Mood.

Table 3 - Story Skeleton Phases

Story Phase

Description

In

out

K

Introduction

Single Intro scene to the system played in loop.

user enters system

user touches start.

K1

Exposition

Several scenes exposing 4 aspects of the character and the story in no specific order. Selected out of several scenes, each covering at least one aspect.

end of Introduction

all aspects are covered exactly once

or eject

K5

Plan

The plan scene of the story. Selected out of several scenes.

end of Exposition

scene ends or eject.

K6

Body

Several interlaced sequences. Each sequences is composed of a procession of stages (crisis, local peak etc.) Each stage is represented by a selected scene.

 

Several sequences may be played in parallel. There is an incentive to close open sequences for the sake of coherence.

end of Plan

eject

K?

Peak & Relaxation

The peak and relaxation sequence of the story and interaction.

The scene is selected according to the K level and preceding scenes played, the mood and the Attitude Level towards the user.

Eject

   

Sprite watches
K – User knowledge of the character (story information) – not each scene contributes to the K level (e.g. scenes that open a scene-loop – contribute nothing, while closing scenes contribute the whole-lot).
I - Intimacy (Level)

Eject – triggered when either:

Property

Description

Value

example

ID

 

number

 

Phase

story phase of the scene

phases described above

 

Depth

How much intimate knowledge about the character does this scene provide?

number

 

Precedent

Specifies which other scenes must be presented before the scene can be presented (only for peak scene)

list of number (scene ID)

12

StartMood(Level)

 

number (same as Mood(Level) of Emotion)

2

EndMood(Level)

 

number (same as Mood(Level) of Emotion)

-2

XXX Area Connotation (XXX = face area from clip map)

The connotations of this area of the face in the scene

connotation

positive negative, neutral

(or null)

XXX Area Connotation Score

 

weight number (1-?)

2

Association

A list of the association of a scene with other scenes. Each has a weight.

list of associations (keyword, weight)

weight number (1-?)

mother, 4


Table 5 - Sequence representation

Property

Description

Value

example

ID

 

number

 

Sequence skeleton

The stages within a sequence

List of strings

 

Sequence Order

the order of the stages within the equence

List of strings

2

 

Sequence stage scenes

The possible scenes in every stage

List of numbers

 

A sequence is selected indirectly when its 1st scene is selected.
A sequence may be played in more than one order of stages, but only once (except for Brooding Mood scenes).
A stage within a specific order of a scene loop is filled by a specific scene, selected according to mood.

3.5.2. History

History maintains the order and frequency of the scenes, and monitors the incentives to close sequences.

Table 6 - History

scene.played

The number of times a scene has been played through

Variable

scene.repeat

the number of times a scene may be repeated.

Variable

Last scene

The last scene played

Number (of scene.ID)

open sequences

which sequences’ 1st scene or more has been played

List of Numbers (of sequence.ids)

Open sequence last scene

Which scene was last played in a sequence

Number (of scene.ID)

Open sequence incentive

How many scenes have been played since open sequence last scene

Number

 

General clip – composing the scenes (not reaction clips) - remember clip for more than one scene TBD

Table 7 – Memory Clips

Property

Description

Value

example

ID

     

Scene