Commedia Virtuale: From Theatre to Avatars

Ben Salem
Department of Industrial Design,
Technische Universiteit Eindhoven, The Netherlands.
b.i.salem@tue.nl

Abstract:
We are investigating face, hand and body expressions to be applied to avatars of a virtual environment to improve their communication capabilities and enrich and facilitate their perception by users of the environment . We report on our work based on obtaining inspiration from the world of theatre. In this perspective Commedia dell’Arte and Noh theatre have been the focus of our attention. We explore key features of Commedia dell’Arte namely improvisation, exaggerated gestures and expressive postures, and investigate how their adoption in the design of avatars can be useful for Collaborative Virtual Environments. With the same objectives we look at another theatre style, the Noh theatre. We investigate the variety of masks and the choreography complexity. The outcome is a visual language for avatars made up of postures, gestures and appearances. We have concluded this investigation with the production of an experimental theatre play involving real and virtual actors.

1. Introduction:
Our definition of a collaborative virtual environment (CVE) is one of an online community where users can participate in a variety of collaborations, and are rendered as avatars so as to be visible to each other. They are substantial advantages when people can see each other in terms of the efficiency of their dialogue. We are attempting to design avatars that would make the experience of participating into the CVE closer to real life. To do so the avatars have to perform easy to understand gestures, postures and expressions. The body language and behaviours are very important to deliver a rich combination of messages and clues about other’s presence in the environment. Such avatar design will also facilitate the awareness of other participants in the CVE, although it is relevant to point out that it is not just the avatars that will make participants aware of each others but also the interaction and dialogues between users (Robinson et al., 2001).
The Commedia Virtuale work is part of the ReLIVE (www.relive.org.uk) project. In Commedia Virtuale we first look at human communication and the importance of gestures, postures, and expressions for an effective exchanges.

2. Human Communication
Communication between people involves the exchange of a spoken discourse along side a series of expressions rendered by the face, the hands and the body. These expressions, gestures, postures, called Non-Verbal Communications (NVC) are used to express the mood, attitude, state of mind and social status. NVC can be used to express emotions and moods (anger, happiness), communicate interpersonal status (friend, acquaintance, stranger), support speech (emphasis, illustration), identity assertion (appearance, status, group membership) and to perform rituals (greetings, etiquette) (Argyle 1990). NVC is also used to strengthen the communication and support the spoken discourse (Olveres et al., 1998) and the exchange of information (Watanabe et al., 1999). The NVC relevant in this context are: clothing, overall appearance, body location and postures, walking stride, hand gestures, and facial expressions.
NVC is not simply a complement to speech but carry a message on its own which is quite distinct and help social relations (Chovil, 1992). When used in combination with speech they help establish flexible and robust communication (Olveres et al., 1998). NVC conveys real-time responsiveness between conversation participants, they let people influence others and acknowledge influence attempts (Wiemann, 1983). In other words, NVC is important to establish feedback between participants in the conversation and to express support to the speaker, agreement with the discourse and so forth.

2.1 Perception of Other Participants
We consider the awareness of others in a CVE as an essential component of an experience close to real-life. The awareness of other participants in the virtual environment is a combination of three components occurring in chronology (Warr and Knapper, 1968). They are pre-judgement, prediction and attachment. The first part is about attributing the person certain characteristics and quickly evaluating them. Pre-Judgment is about gaining first impressions of a person. The prediction is about generalising to all situations, the behaviour and characteristics of a person from what has been observing in one situation. The predictions are therefore directly the result of the prejudgement one makes of another person. The attachment is the emotional response one has to another person in terms of liking, sympathy, and so on. We propose to exploit these aspects of the perception of others by emphasising NVC that would reinforce both pre-judgement and attachment. NVC are essential to first impressions and are also important to the expression of emotions. We have investigated different NVC and have realised that it is regulated by social and cultural norms. While communication is better achieved when there is a slight deviation from the expected behaviour to reflect some personality and individual differences. In general, low-status avatars are expected to conform to a norm, and highstatus ones are expected to behave rather independently and outside predefined rules (Burgoon, 1983).

2.2 Body Location
Body location is about the distance between participants that should reflect the kind of relationship that links them. It is also about the invasion of territory (intimate for couples or in a medical practitioner/patient situation, personal for very close people, social for friends, professional with colleagues and public). The nearer one gets the closer relationship one is expected to have to avoid provoking discomfort.
During communication; participants establish a small territory in which their discourse will occur (Scheflen, 1972). The participant orientation, and the distances separating them as well as the amount of body contact and touching are all part of communication and the establishment of a social order between participants. NVC can be used to assert affiliation to the same group, dominance and submission, as well as territorial limits.

2.3 Role and Identity
A role is a set of appearances and behaviours characteristic of one or a group of persons in a context. In many cases the appearance defines the role, police officers, doctors and many others are expected to wear a uniform. A role is also a set of actions or performance that characterise the person or the group of persons observed. It is important to ignore those characteristics of a person that do not define a role, for example the eyes colour of a police officer. There are three role genres: social roles, (e.g. the middle class married couple), contextual roles (e.g. the patient in a hospital), and functional roles/roles associated with tasks, (e.g. the postman delivering mail). Three key elements of an avatar have to be considered during their design as the embodiment of an environment agent or participant. They are: the avatar role, its interaction mode, its representation and its personality (Churchill et al., 2000). Another important issue is the credibility and relevance of the avatar and the expression of their competence and expertise. The embodiment of avatars does also reinforce the environments users experience of presence (Gerhard et al., 2004).

3 NVC for Avatars
There are three kind of NVC, which we have used: appearances, postures and expressions. Some work has been done in the area of facial expressions and presentation engine for avatars (Muller et al., 2001). There also has been some work on the setting of a system for the autonomous behaviour for avatars (Cassell and Vilhjalmsson, 1999). In this paper we focus on the expressiveness of the avatars. To strengthen the perception of the NVC selected, we have adopted exaggerated features for the avatars we are designing. Caricature-like faces are easier to read and exaggerated gestures and postures are clearer and less ambiguous to understand. As a result, we are not interested by realistic or high-quality avatars but rather by simple, caricatured and expressive ones. Looking at CdA and Noh/Kabuki theatre the exaggerated features the characters possess in these theatre styles make them very relevant for our design choices.

3.1 Facial Expressions
Facial expressions are of great importance for the conveying of messages for social interaction and are one of the first forms of communication we rely on (Izard, 1979). They are used for close range encounter, and are particularly relevant during a conversation. Facial expressions can communicate effectively a multitude of emotions (Fabri and Gerhard, 2000). The key elements of a facial expression are the mouth at the root of any facial expression, the eyes which help modify an expression, and the brows to a lesser degree (Fleming and Dobbs, 1999).
We could use faces that can generate expressions or faces which by themselves portray some expressions, for example thanks to masks from Commedia dell’Arte (CdA) (see fig.3 and fig.4). The other advantage of masks is that they delivers a solution to the problem of confusion between facial expressions.

3.2 Hand Gestures
Hand gestures are a powerful means for non-verbal communication. Gestures are used for commands, dialogues, to quantify and describe, as well as to perform static and dynamic signs as with the American and British Sign Languages (ASL and BSL). They are also used to indicate objects and direction and to illustrate properties of objects such as size. Hence the suitability of hand gestures for expressive avatars. It is important to bear in mind that the viewpoint must be close enough to the avatar otherwise the gestures are difficult to see.

Author : Ben Salem

3.3 Body Postures
Human communication is also about the understanding of the other’s mental state, beliefs, desires, moods and personality (Bruce, 1994). Body postures are essential in interpersonal communication when people are trying to understand more than just the spoken discourse. There is no reason why avatars should not be capable of performing postures and gestures to emphasise their discourse, to inform about their status and tasks. Body postures are perceived at the farthest range they can be used for someone wishing to attract attention or indicate his status. Postures can also be used to express some emotions and the social status such as dominant and subordinate.

Author : Ben Salem

4 Theatre Styles and Characters
Within the scope of this project we have evaluated two theatre styles that are rich with choreographies of body movements to express emotions and statements. Already, CVEs have significant potential to deliver a new experience of drama, story and narration (Laurel, 1993). They have also been explored for the recording and performance of drama (Craven et al., 2001), and for the broadcasting of a theatre play (Matsuba and Roehl, 1999). As inspiration for the design of avatars we sought theatre and performance art styles where there exist a set of choreography and behaviour used for the expression of some aspects of the play, the characters, their emotion, discourse and importance. We have selected two theatre styles to investigate within CVEs. One is improvisational theatre of Europe of the 16th century, Commedia dell'Arte. The other is a traditional, rigid and highly regulated Japanese theatre style: Noh.

4.1 Commedia dell’Arte (CdA)
CdA comes from middle 16th century’s Italy it is at the origin of modern mime theatre. CdA is essentially improvised theatre, where a play is loosely outlined and the actors perform continuously renewed and changed acts from their our initiative and reacting to the audience. Improvisational theatre relies on creating interesting and fruitful scenarios to facilitate the actors’ improvisations (Klesen et al., 2001). Conversation topics, project related meetings and social events are good analogies between CdA in the theatre and CVEs. The actors would improvise all their own dialogues and performance, within a general framework the scenario. While performing the actors adopt a set of hand gestures, body postures and movements, which are highly exaggerated and caricature-like to facilitate their understanding by the audience.
In CdA there are many characters reflecting the society as a whole in terms of personality, desires, and relationship. Typical characters would be the old miser (Pantalone) , the alcoholic (Il Dottore), the vulgar (Arlecchino), the owner of premise (Brighella), and so on. Each character is well defined in terms of their name, costume and visual appearance, stance, gestures, relationships to others, status, mask, walk, movements, and functions (Rudlin, 1994).

4.2 Noh Theatre
Noh theatre is a classical 14th-15th centuries Japanese performance form which grew from court music, popular entertainment and religious practices. It combines dance, drama, music and poetry in a convergence of arts. A Noh play follows a three-part structure: Jo (introduction), Ha (development) and Kyu (scattering). There is traditionally a combination of Noh and Kyogen (comic part) performances during the same program. Noh play is a continuous flow and nothing can be separated as a single isolated element (Mitchell and Watanabe, 1994). During a Noh play the actors must remain focused, with the head high up. The head and the body are never relaxed. All the actor movements are slow and controlled as if the actor was facing great resistance from the surrounding air. This indicates roles that would not be as intrusive in the CVE as those of CdA based avatars.

4.3 Roles and Identities for a Virtual Environment
CdA and Noh/Kabuki are particularly relevant as they are theatre styles where the visual element of the play is given equal if not greater emphasis than the verbal. In CdA, for example, the actors mime the main subject of their dialogue simultaneously with their discourse (Grantham, 2000). In Noh/Kabuki theatre, the movements of the actors are highly regulated and tense, mesmerising the audience. Looking at the elements of CdA and Noh/Kabuki Theatres we have translated these into the making of our proposed Avatars.

Author : Ben Salem

5 Avatars Masks
A part from few exceptions facial expressions are culturally universal, and seven expressions are identified easily, they are: happiness, sadness, surprise, anger, fear, disgust/contempt and interest (Anderson, 1992). Some key features are used to generate these expressions. The masks from CdA and Noh/Kabuki theatres use these to render strong expressions that in turn portray the character, which don them.

5.1 Commedia Masks
CdA masks are always half masks. They are the most efficient and powerful way of giving the identity of the character (Grantham, 2000). In Commedia Virtuale we have made two kinds of masks, low and high resolution. Low-resolution masks are used for system agents such as Il Dottore as a help agent. The mask is modelled out of simple shapes, circles and cones. Yet the overall aspect is still preserved as shown in figure 3.

Author : Ben Salem

Author : Ben Salem

5.2 Noh Masks
The Commedia Virtuale system is designed for westerners, therefore Noh/Kabuki masks may not portray personalities easily recognisable in our society. We are therefore using these masks exclusively for the representation of system agents. We have selected the demons masks to represent agents delivery messages related to bandwidth bottlenecks, security, and system failures.

Author : Ben Salem

6 Improving on Current CVEs
As we advocate a consistency of the interaction mode when navigating in a VE system. Users should be prompted by the system through the CVE and not through pop-up text windows. This implies the need for avatars representing system agents that will appear in the environment to deliver messages to the user. We have looked at both CdA and Noh/Kabuki for inspiration regarding the appearance and also the behaviour of the proposed avatars. Table 2 lists the essential elements taken from these theatre styles and directly translated into avatar functions.

Author : Ben Salem

Author : Ben Salem

6.1 Conversation Circles
We have developed the concept of conversation circle as a structure in the virtual environment for social encounters and collaboration (Salem and Earle, 2000). A similar concept exists for both CdA and Noh/Kabuki theatres. The order of appearance, the position of each actor is relevant for the understanding of the plot. Furthermore, in CdA the position the characters stand in the stage and the way it moves around are of great importance in understanding his/her personality.

Author : Ben Salem

Author : Ben Salem

6.2 Collaborative work
In the case of object/data manipulation and visualisation we have selected a series of body postures from CdA. There is a comprehensive set of postures available that are related to object manipulation and inspection, dialogues and statements. For example Seeking, observing, inspecting, accepting, rejecting, and moving around objects are some of the most efficient ones from CdA. Mime theatre is after all quite good at giving the impression the actors are manipulating virtual objects (such as opening an invisible door, cleaning a window…)

Author : Ben Salem

6.3 Improvised situations
CdA is quintessentially improvisational theatre, rules of engagement are clear and allow for the spontaneous flow of dialogues and improvisation. These rules are based on a set of actions, speech/discourses and monologues. This would provide the user with a set of gestures, answers and behaviour useful in social interaction. For example a laughing sequence made of the initial understanding of the joke, and a series of laughter. Indeed CdA can be used as a design metaphor, a set of rules for CVE user representation and behaviour (Tuomola, 1999).

7 Theatre Play
To further investigate the use of CdA and Noh/Kabuki in CVEs, we have set up a Commedia dell’Arte play involving real actors and avatar. In other words we have attempted a reverse engineering approach by building an animation for an avatar based on CdA performance (figure 9.a and 9.b). We then included this animation in a CdA scene. This play combined a real actor and an avatar. The plot we have selected was about Pantalone complaining about a dirty yard in a hot afternoon. He then realises that his shadow has disappeared. The shadow in fact emerges in the background as an avatar (figure 9c.). The scene ends with Pantalone arguing with his shadow…

Author : Ben Salem

In sequence we see the avatar waking up (login –in) investigating his surroundings, observing an object, reacting to Pantalone acting. In brief some of the key elements of the behaviour of an expressive avatar within a CVE. Such bevaiour could be used for the creation of characters for virtual narrative systems such as teatrix (Paiva, 2001).

Conclusion
To be successful a CVE should deliver a better experience and be more efficient than a telephone conversation or email when engaging a collaboration with others. A Turing test need to be set-up, one assessing what best tool to use for collaborative work, a CVE or a meeting room. It is a test that so far has not been passed unequivocally. Commedia Virtuale is an inspiration from theatre styles in our drive to implement avatars that possess a certain degree of expressiveness, a useful quality for interpersonal communication. The animations based on avatars adopting behaviour from CdA, we have realised, have helped demonstrate that such avatars deliver clear messages about their status and role in the VE. Inspiring the design of avatar from the world of theatre can be productive, and it is easy to understand why CdA and Noh are popular and still successful theatre styles. The Commedia Virtuale is at crossing point between the world of entertainment and the world of virtual environments.

Acknowledgments
Commedia Virtuale is funded by a generous grant from Yorkshire Arts, RAL Program # 5-2080.

References
Anderson, A.H. (1992) The human communication research dialogue data base, Journal of child language 19, 711-716.
Argyle M. (1990) Bodily communication, Routledge, London, England.
Bruce, V. (1994) What the human face tells the human mind: some challenged for the robot-human interface, Advanced Robotics 8(4) , 341-355.
Burgoon, J.K. (1983) Nonverbal violations of expectations, In Wiemann J.M., Harrison R.P. (eds.) Nonverbal interaction, Sage Publications, Beverly Hills, CA, USA.
Cassell, J., Vilhjalmsson, H. (1999) Fully embodied conversational avatars: making communicative behaviors autonomous, Autonomous agents and multi-agent systems 2, 45-64.
Chovil N. (1992) Discourse-oriented facial displays in conversation. Research on language and social interaction 25, 163-194.
Churchill E., Cook L., Hodgson P., Prevost S., Sullivan W. (2000) Embodied conversational agents, MIT Press, Cambridge, MA, USA.
Craven M., Taylor I., Drozd A., Purbrick J., Greenhalgh C., Benford S., Fraser M., Bowers J., Jaa-Aro K.-M., Lintermann B., Hoch M. (2001) Exploiting interactivity, influence, space and time to explore non-linear drama in virtual worlds, In Proceedings of CHI 2001, ACM, pp.30-37.
Fabri M., Gerhard M. (2000) The Virtual student: user embodiment in virtual learning environments, In Orange G. and Hobbs D. (eds.) International perspectives on tele-education and virtual learning environments, Ashgate, Aldershot, England, pp. 32-55.
Fleming B., Dobbs D. (1999) Animating facial features & expressions, Charles River Media, Rockland, MA, USA.
Gerhard, M., Moore, D., Hobbs, D. (2004) Embodiment and copresence in collaborative interfaces, International journal of human-computer studies, Article in press, available at www.elseviercomputerscience.com.
Grantham, M., Playing commedia: a training guide to commedia techniques, Nick Hern Book, London, UK, 2000.
Izard C. (1979) Facial expression, emotion, and motivation, In Wolfgang A. (ed.) Nonverbal behavior: applications and cultural implications, Academic Press, New York, NY, USA.
Klesen M., Szatkowski J., Lehmann N., A Dramatised actant model for interactive improvisational plays, In de Antonio A., Aylett R., Ballin D. (eds.) Proceedings of Intelligent virtual agents 2001, Springer-Verlag, Berlin, Germany, pp. 181-194.
Laurel B. (1993) Computers as theatre, Addison-Wesley, Reading, MA, USA.
Matsuba S., Roehl B. (1999) “Bottom, Thou Art Translated”: the making of VRML dream, IEEE Computer Graphics and Applications 19(2), 45-51.
Mitchell J.D., Watanabe M. (1994) Noh and Kabuki: staging japanese theatre, IASTA Press, Key West, FL, USA.
Muller, W., Spierling, U., Alexa, M., Rieger, Th. (2001) Face-to-face with your assistant: realization issues of animated used interface agents for home appliances, Computer & Graphics 25, 593-600.
Olveres J., Billinghurst M., Savage J., Holden A. (1998) Intelligent, expressive avatars. In Proceedings of IEEE workshop on embodies conversational characters, IEEE, pp. 47-55.
Paiva, A., Machado, I., Prada, R. (2001) Heroes, villains, magicians,...: dramatis personae in a virtual story creation environment, In Proceedings of IUI'01, Santa Fe, NM, USA, 14-17 January 2001, ACM Press, 129-136.
Robinson M., Pekkola S., Korhonen J., Hujala S., Toivonen T., Saarinen M.-J. (2001) Extending the limits of collaborative virtual environments, In Churchill E., Snowdon D., and Munro A. (eds.),Collaborative virtual environments (CVEs): histories, perspective and issues, Springer-Verlag, London, England, pp. 21-42.
Rudlin J. (1994) Commedia dell’Arte: An actor’s handbook, Routledge, London, England.
Salem B. (1999) Gestures, In Noyes J. M. and Cook M. (eds.) Interface technology: the leading edge, RSP, Hertfordshire, England, pp.73-96.
Salem B., and Earle N. (2000) Designing a non-verbal language for expressive avatars, In Proceedings of CVE’2000 , San Francisco, CA, USA, September 2000, ACM Press, pp.93-101.
Scheflen A.E. (1972) Body language and the social order, Prentice Hall, Englewood Cliffs, USA.
Tuomola, M. (1999) Drama in the digital domain: commedia dell’arte, characterisation, collaboration and computers, Digital Creativity 10(3), 167-179.
Warr P., Knapper C. (1968) The perception of people and events, John Wiley & Sons, London, England.
Watanabe T., Okubo M. (1999) Virtual face-to-face communication system for human interaction analysis by synthesis, In Proceedings of HCI INT 99, 2, pp. 182-186.
Wiemann J.M., Harrison R.P. (1983) Non-language aspects of social interaction, In Wiemann J.M., Harrison R.P. (eds.) Nonverbal interaction, Sage Publications, Beverly Hills, CA, USA.