Commedia Virtuale: From Theatre to Avatars
Department of Industrial Design,
Technische Universiteit Eindhoven, The Netherlands.
We are investigating face, hand and body expressions to be applied to avatars of a virtual environment to improve their
communication capabilities and enrich and facilitate their perception by users of the environment . We report on our
work based on obtaining inspiration from the world of theatre. In this perspective Commedia dell’Arte and Noh theatre
have been the focus of our attention. We explore key features of Commedia dell’Arte namely improvisation,
exaggerated gestures and expressive postures, and investigate how their adoption in the design of avatars can be useful
for Collaborative Virtual Environments. With the same objectives we look at another theatre style, the Noh theatre. We
investigate the variety of masks and the choreography complexity. The outcome is a visual language for avatars made
up of postures, gestures and appearances. We have concluded this investigation with the production of an experimental
theatre play involving real and virtual actors.
Our definition of a collaborative virtual environment (CVE) is one of an online community where users can participate
in a variety of collaborations, and are rendered as avatars so as to be visible to each other. They are substantial
advantages when people can see each other in terms of the efficiency of their dialogue. We are attempting to design
avatars that would make the experience of participating into the CVE closer to real life. To do so the avatars have to
perform easy to understand gestures, postures and expressions. The body language and behaviours are very important to
deliver a rich combination of messages and clues about other’s presence in the environment. Such avatar design will
also facilitate the awareness of other participants in the CVE, although it is relevant to point out that it is not just the
avatars that will make participants aware of each others but also the interaction and dialogues between users (Robinson
et al., 2001).
The Commedia Virtuale work is part of the ReLIVE (www.relive.org.uk) project. In Commedia Virtuale we first look at
human communication and the importance of gestures, postures, and expressions for an effective exchanges.
2. Human Communication
Communication between people involves the exchange of a spoken discourse along side a series of expressions
rendered by the face, the hands and the body. These expressions, gestures, postures, called Non-Verbal
Communications (NVC) are used to express the mood, attitude, state of mind and social status. NVC can be used to
express emotions and moods (anger, happiness), communicate interpersonal status (friend, acquaintance, stranger),
support speech (emphasis, illustration), identity assertion (appearance, status, group membership) and to perform rituals
(greetings, etiquette) (Argyle 1990). NVC is also used to strengthen the communication and support the spoken
discourse (Olveres et al., 1998) and the exchange of information (Watanabe et al., 1999). The NVC relevant in this
context are: clothing, overall appearance, body location and postures, walking stride, hand gestures, and facial
NVC is not simply a complement to speech but carry a message on its own which is quite distinct and help social
relations (Chovil, 1992). When used in combination with speech they help establish flexible and robust communication
(Olveres et al., 1998). NVC conveys real-time responsiveness between conversation participants, they let people
influence others and acknowledge influence attempts (Wiemann, 1983). In other words, NVC is important to establish
feedback between participants in the conversation and to express support to the speaker, agreement with the discourse
and so forth.
2.1 Perception of Other Participants
We consider the awareness of others in a CVE as an essential component of an experience close to real-life. The
awareness of other participants in the virtual environment is a combination of three components occurring in
chronology (Warr and Knapper, 1968). They are pre-judgement, prediction and attachment. The first part is about
attributing the person certain characteristics and quickly evaluating them. Pre-Judgment is about gaining first
impressions of a person. The prediction is about generalising to all situations, the behaviour and characteristics of a
person from what has been observing in one situation. The predictions are therefore directly the result of the prejudgement
one makes of another person. The attachment is the emotional response one has to another person in terms of
liking, sympathy, and so on. We propose to exploit these aspects of the perception of others by emphasising NVC that
would reinforce both pre-judgement and attachment. NVC are essential to first impressions and are also important to the
expression of emotions. We have investigated different NVC and have realised that it is regulated by social and cultural
norms. While communication is better achieved when there is a slight deviation from the expected behaviour to reflect
some personality and individual differences. In general, low-status avatars are expected to conform to a norm, and highstatus
ones are expected to behave rather independently and outside predefined rules (Burgoon, 1983).
2.2 Body Location
Body location is about the distance between participants that should reflect the kind of relationship that links them. It is
also about the invasion of territory (intimate for couples or in a medical practitioner/patient situation, personal for very
close people, social for friends, professional with colleagues and public). The nearer one gets the closer relationship one
is expected to have to avoid provoking discomfort.
During communication; participants establish a small territory in which their discourse will occur (Scheflen, 1972). The
participant orientation, and the distances separating them as well as the amount of body contact and touching are all part
of communication and the establishment of a social order between participants. NVC can be used to assert affiliation to
the same group, dominance and submission, as well as territorial limits.
2.3 Role and Identity
A role is a set of appearances and behaviours characteristic of one or a group of persons in a context. In many cases the
appearance defines the role, police officers, doctors and many others are expected to wear a uniform. A role is also a set
of actions or performance that characterise the person or the group of persons observed. It is important to ignore those
characteristics of a person that do not define a role, for example the eyes colour of a police officer. There are three role
genres: social roles, (e.g. the middle class married couple), contextual roles (e.g. the patient in a hospital), and
functional roles/roles associated with tasks, (e.g. the postman delivering mail).
Three key elements of an avatar have to be considered during their design as the embodiment of an environment agent
or participant. They are: the avatar role, its interaction mode, its representation and its personality (Churchill et al.,
2000). Another important issue is the credibility and relevance of the avatar and the expression of their competence and
expertise. The embodiment of avatars does also reinforce the environments users experience of presence (Gerhard et al.,
3 NVC for Avatars
There are three kind of NVC, which we have used: appearances, postures and expressions. Some work has been done in
the area of facial expressions and presentation engine for avatars (Muller et al., 2001). There also has been some work
on the setting of a system for the autonomous behaviour for avatars (Cassell and Vilhjalmsson, 1999). In this paper we
focus on the expressiveness of the avatars. To strengthen the perception of the NVC selected, we have adopted
exaggerated features for the avatars we are designing. Caricature-like faces are easier to read and exaggerated gestures
and postures are clearer and less ambiguous to understand. As a result, we are not interested by realistic or high-quality
avatars but rather by simple, caricatured and expressive ones. Looking at CdA and Noh/Kabuki theatre the exaggerated
features the characters possess in these theatre styles make them very relevant for our design choices.
3.1 Facial Expressions
Facial expressions are of great importance for the conveying of messages for social interaction and are one of the first
forms of communication we rely on (Izard, 1979). They are used for close range encounter, and are particularly relevant
during a conversation. Facial expressions can communicate effectively a multitude of emotions (Fabri and Gerhard,
2000). The key elements of a facial expression are the mouth at the root of any facial expression, the eyes which help
modify an expression, and the brows to a lesser degree (Fleming and Dobbs, 1999).
We could use faces that can generate expressions or faces which by themselves portray some expressions, for example
thanks to masks from Commedia dell’Arte (CdA) (see fig.3 and fig.4). The other advantage of masks is that they
delivers a solution to the problem of confusion between facial expressions.
3.2 Hand Gestures
Hand gestures are a powerful means for non-verbal communication. Gestures are used for commands, dialogues, to
quantify and describe, as well as to perform static and dynamic signs as with the American and British Sign Languages
(ASL and BSL). They are also used to indicate objects and direction and to illustrate properties of objects such as size.
Hence the suitability of hand gestures for expressive avatars. It is important to bear in mind that the viewpoint must be
close enough to the avatar otherwise the gestures are difficult to see.
Author : Ben Salem
We have selected a vocabulary of general gestures such as those illustrated in figure 1. These gestures were selected
because of their relevance in a dialogue and their almost universal acceptance; they are part of a wider set of gestures
selected because of their relevance for a User Interface (Salem, 1999).
3.3 Body Postures
Human communication is also about the understanding of the other’s mental state, beliefs, desires, moods and
personality (Bruce, 1994). Body postures are essential in interpersonal communication when people are trying to
understand more than just the spoken discourse. There is no reason why avatars should not be capable of performing
postures and gestures to emphasise their discourse, to inform about their status and tasks.
Body postures are perceived at the farthest range they can be used for someone wishing to attract attention or indicate
his status. Postures can also be used to express some emotions and the social status such as dominant and subordinate.
Author : Ben Salem
Figure 2 shows that postures can be used to render simple states of mind. The postures are however limited in the
number of mood they can express. A too large number of postures and they become confusing to understand. CdA has a
comprehensive set of caricatured but sometimes ambiguous postures. The ambiguity is solved by the spoken discourse
concurrent to the posture as well as by the context where the posture occurred.
4 Theatre Styles and Characters
Within the scope of this project we have evaluated two theatre styles that are rich with choreographies of body
movements to express emotions and statements. Already, CVEs have significant potential to deliver a new experience
of drama, story and narration (Laurel, 1993). They have also been explored for the recording and performance of drama
(Craven et al., 2001), and for the broadcasting of a theatre play (Matsuba and Roehl, 1999). As inspiration for the
design of avatars we sought theatre and performance art styles where there exist a set of choreography and behaviour
used for the expression of some aspects of the play, the characters, their emotion, discourse and importance.
We have selected two theatre styles to investigate within CVEs. One is improvisational theatre of Europe of the 16th
century, Commedia dell'Arte. The other is a traditional, rigid and highly regulated Japanese theatre style: Noh.
4.1 Commedia dell’Arte (CdA)
CdA comes from middle 16th century’s Italy it is at the origin of modern mime theatre. CdA is essentially improvised
theatre, where a play is loosely outlined and the actors perform continuously renewed and changed acts from their our
initiative and reacting to the audience. Improvisational theatre relies on creating interesting and fruitful scenarios to
facilitate the actors’ improvisations (Klesen et al., 2001). Conversation topics, project related meetings and social events
are good analogies between CdA in the theatre and CVEs. The actors would improvise all their own dialogues and
performance, within a general framework the scenario. While performing the actors adopt a set of hand gestures, body
postures and movements, which are highly exaggerated and caricature-like to facilitate their understanding by the
In CdA there are many characters reflecting the society as a whole in terms of personality, desires, and relationship.
Typical characters would be the old miser (Pantalone) , the alcoholic (Il Dottore), the vulgar (Arlecchino), the owner of
premise (Brighella), and so on. Each character is well defined in terms of their name, costume and visual appearance,
stance, gestures, relationships to others, status, mask, walk, movements, and functions (Rudlin, 1994).
4.2 Noh Theatre
Noh theatre is a classical 14th-15th centuries Japanese performance form which grew from court music, popular
entertainment and religious practices. It combines dance, drama, music and poetry in a convergence of arts.
A Noh play follows a three-part structure: Jo (introduction), Ha (development) and Kyu (scattering). There is
traditionally a combination of Noh and Kyogen (comic part) performances during the same program. Noh play is a
continuous flow and nothing can be separated as a single isolated element (Mitchell and Watanabe, 1994). During a
Noh play the actors must remain focused, with the head high up. The head and the body are never relaxed. All the actor
movements are slow and controlled as if the actor was facing great resistance from the surrounding air. This indicates
roles that would not be as intrusive in the CVE as those of CdA based avatars.
4.3 Roles and Identities for a Virtual Environment
CdA and Noh/Kabuki are particularly relevant as they are theatre styles where the visual element of the play is given
equal if not greater emphasis than the verbal. In CdA, for example, the actors mime the main subject of their dialogue
simultaneously with their discourse (Grantham, 2000). In Noh/Kabuki theatre, the movements of the actors are highly
regulated and tense, mesmerising the audience. Looking at the elements of CdA and Noh/Kabuki Theatres we have
translated these into the making of our proposed Avatars.
Author : Ben Salem
In both theatre styles, the characters have a role and an identity easily recognisable. They are inspired from society and
tales we are all familiar with (the good against the bad, the weak as a victim of the strong, etc..). Because the theatre
styles have been refined and actualised over centuries, they are a comprehensive reflection of society. They illustrate in
a combination of subtle, exaggerated and dramatised ways all the characteristics of social interaction. Hence the
relevance of theatre as a source of inspiration.
5 Avatars Masks
A part from few exceptions facial expressions are culturally universal, and seven expressions are identified easily, they
are: happiness, sadness, surprise, anger, fear, disgust/contempt and interest (Anderson, 1992). Some key features are
used to generate these expressions. The masks from CdA and Noh/Kabuki theatres use these to render strong
expressions that in turn portray the character, which don them.
5.1 Commedia Masks
CdA masks are always half masks. They are the most efficient and powerful way of giving the identity of the character
(Grantham, 2000). In Commedia Virtuale we have made two kinds of masks, low and high resolution. Low-resolution
masks are used for system agents such as Il Dottore as a help agent. The mask is modelled out of simple shapes, circles
and cones. Yet the overall aspect is still preserved as shown in figure 3.
Author : Ben Salem
High-resolution mask are used for avatar of greater importance representing an application agent, e.g. Pantalone and the
environment manager, e.g. Brighella.
Author : Ben Salem
5.2 Noh Masks
The Commedia Virtuale system is designed for westerners, therefore Noh/Kabuki masks may not portray personalities
easily recognisable in our society. We are therefore using these masks exclusively for the representation of system
agents. We have selected the demons masks to represent agents delivery messages related to bandwidth bottlenecks,
security, and system failures.
Author : Ben Salem
*Photo from : www.pasar5.com/NOH_MASK/mask/tengu.html
Author : Ben Salem
The masks are self-explanatory, looking at them one can see that they portray demons bearing bad news.
6 Improving on Current CVEs
As we advocate a consistency of the interaction mode when navigating in a VE system. Users should be prompted by
the system through the CVE and not through pop-up text windows. This implies the need for avatars representing
system agents that will appear in the environment to deliver messages to the user.
We have looked at both CdA and Noh/Kabuki for inspiration regarding the appearance and also the behaviour of the
proposed avatars. Table 2 lists the essential elements taken from these theatre styles and directly translated into avatar
Author : Ben Salem
Author : Ben Salem
Figure 6 illustrate the postures selected
6.1 Conversation Circles
We have developed the concept of conversation circle as a structure in the virtual environment for social encounters and
collaboration (Salem and Earle, 2000). A similar concept exists for both CdA and Noh/Kabuki theatres. The order of
appearance, the position of each actor is relevant for the understanding of the plot. Furthermore, in CdA the position the
characters stand in the stage and the way it moves around are of great importance in understanding his/her personality.
Author : Ben Salem
Author : Ben Salem
6.2 Collaborative work
In the case of object/data manipulation and visualisation we have selected a series of body postures from CdA. There is
a comprehensive set of postures available that are related to object manipulation and inspection, dialogues and
statements. For example Seeking, observing, inspecting, accepting, rejecting, and moving around objects are some of
the most efficient ones from CdA. Mime theatre is after all quite good at giving the impression the actors are
manipulating virtual objects (such as opening an invisible door, cleaning a window…)
Author : Ben Salem
6.3 Improvised situations
CdA is quintessentially improvisational theatre, rules of engagement are clear and allow for the spontaneous flow of
dialogues and improvisation. These rules are based on a set of actions, speech/discourses and monologues. This would
provide the user with a set of gestures, answers and behaviour useful in social interaction. For example a laughing
sequence made of the initial understanding of the joke, and a series of laughter. Indeed CdA can be used as a design
metaphor, a set of rules for CVE user representation and behaviour (Tuomola, 1999).
7 Theatre Play
To further investigate the use of CdA and Noh/Kabuki in CVEs, we have set up a Commedia dell’Arte play involving
real actors and avatar. In other words we have attempted a reverse engineering approach by building an animation for
an avatar based on CdA performance (figure 9.a and 9.b). We then included this animation in a CdA scene. This play
combined a real actor and an avatar. The plot we have selected was about Pantalone complaining about a dirty yard in a
hot afternoon. He then realises that his shadow has disappeared. The shadow in fact emerges in the background as an
avatar (figure 9c.). The scene ends with Pantalone arguing with his shadow…
Author : Ben Salem
In sequence we see the avatar waking up (login –in) investigating his surroundings, observing an object, reacting to
Pantalone acting. In brief some of the key elements of the behaviour of an expressive avatar within a CVE. Such
bevaiour could be used for the creation of characters for virtual narrative systems such as teatrix (Paiva, 2001).
To be successful a CVE should deliver a better experience and be more efficient than a telephone conversation or email
when engaging a collaboration with others. A Turing test need to be set-up, one assessing what best tool to use for
collaborative work, a CVE or a meeting room. It is a test that so far has not been passed unequivocally.
Commedia Virtuale is an inspiration from theatre styles in our drive to implement avatars that possess a certain degree
of expressiveness, a useful quality for interpersonal communication. The animations based on avatars adopting
behaviour from CdA, we have realised, have helped demonstrate that such avatars deliver clear messages about their
status and role in the VE. Inspiring the design of avatar from the world of theatre can be productive, and it is easy to
understand why CdA and Noh are popular and still successful theatre styles. The Commedia Virtuale is at crossing
point between the world of entertainment and the world of virtual environments.
Commedia Virtuale is funded by a generous grant from Yorkshire Arts, RAL Program # 5-2080.
Anderson, A.H. (1992) The human communication research dialogue data base, Journal of child language 19, 711-716.
Argyle M. (1990) Bodily communication, Routledge, London, England.
Bruce, V. (1994) What the human face tells the human mind: some challenged for the robot-human interface, Advanced
Robotics 8(4) , 341-355.
Burgoon, J.K. (1983) Nonverbal violations of expectations, In Wiemann J.M., Harrison R.P. (eds.) Nonverbal
interaction, Sage Publications, Beverly Hills, CA, USA.
Cassell, J., Vilhjalmsson, H. (1999) Fully embodied conversational avatars: making communicative behaviors
autonomous, Autonomous agents and multi-agent systems 2, 45-64.
Chovil N. (1992) Discourse-oriented facial displays in conversation. Research on language and social interaction 25,
Churchill E., Cook L., Hodgson P., Prevost S., Sullivan W. (2000) Embodied conversational agents, MIT Press,
Cambridge, MA, USA.
Craven M., Taylor I., Drozd A., Purbrick J., Greenhalgh C., Benford S., Fraser M., Bowers J., Jaa-Aro K.-M.,
Lintermann B., Hoch M. (2001) Exploiting interactivity, influence, space and time to explore non-linear drama in
virtual worlds, In Proceedings of CHI 2001, ACM, pp.30-37.
Fabri M., Gerhard M. (2000) The Virtual student: user embodiment in virtual learning environments, In Orange G. and
Hobbs D. (eds.) International perspectives on tele-education and virtual learning environments, Ashgate, Aldershot,
England, pp. 32-55.
Fleming B., Dobbs D. (1999) Animating facial features & expressions, Charles River Media, Rockland, MA, USA.
Gerhard, M., Moore, D., Hobbs, D. (2004) Embodiment and copresence in collaborative interfaces, International journal
of human-computer studies, Article in press, available at www.elseviercomputerscience.com.
Grantham, M., Playing commedia: a training guide to commedia techniques, Nick Hern Book, London, UK, 2000.
Izard C. (1979) Facial expression, emotion, and motivation, In Wolfgang A. (ed.) Nonverbal behavior: applications and
cultural implications, Academic Press, New York, NY, USA.
Klesen M., Szatkowski J., Lehmann N., A Dramatised actant model for interactive improvisational plays, In de Antonio
A., Aylett R., Ballin D. (eds.) Proceedings of Intelligent virtual agents 2001, Springer-Verlag, Berlin, Germany, pp.
Laurel B. (1993) Computers as theatre, Addison-Wesley, Reading, MA, USA.
Matsuba S., Roehl B. (1999) “Bottom, Thou Art Translated”: the making of VRML dream, IEEE Computer Graphics
and Applications 19(2), 45-51.
Mitchell J.D., Watanabe M. (1994) Noh and Kabuki: staging japanese theatre, IASTA Press, Key West, FL, USA.
Muller, W., Spierling, U., Alexa, M., Rieger, Th. (2001) Face-to-face with your assistant: realization issues of animated
used interface agents for home appliances, Computer & Graphics 25, 593-600.
Olveres J., Billinghurst M., Savage J., Holden A. (1998) Intelligent, expressive avatars. In Proceedings of IEEE
workshop on embodies conversational characters, IEEE, pp. 47-55.
Paiva, A., Machado, I., Prada, R. (2001) Heroes, villains, magicians,...: dramatis personae in a virtual story creation
environment, In Proceedings of IUI'01, Santa Fe, NM, USA, 14-17 January 2001, ACM Press, 129-136.
Robinson M., Pekkola S., Korhonen J., Hujala S., Toivonen T., Saarinen M.-J. (2001) Extending the limits of
collaborative virtual environments, In Churchill E., Snowdon D., and Munro A. (eds.),Collaborative virtual
environments (CVEs): histories, perspective and issues, Springer-Verlag, London, England, pp. 21-42.
Rudlin J. (1994) Commedia dell’Arte: An actor’s handbook, Routledge, London, England.
Salem B. (1999) Gestures, In Noyes J. M. and Cook M. (eds.) Interface technology: the leading edge, RSP,
Hertfordshire, England, pp.73-96.
Salem B., and Earle N. (2000) Designing a non-verbal language for expressive avatars, In Proceedings of CVE’2000 ,
San Francisco, CA, USA, September 2000, ACM Press, pp.93-101.
Scheflen A.E. (1972) Body language and the social order, Prentice Hall, Englewood Cliffs, USA.
Tuomola, M. (1999) Drama in the digital domain: commedia dell’arte, characterisation, collaboration and computers,
Digital Creativity 10(3), 167-179.
Warr P., Knapper C. (1968) The perception of people and events, John Wiley & Sons, London, England.
Watanabe T., Okubo M. (1999) Virtual face-to-face communication system for human interaction analysis by synthesis,
In Proceedings of HCI INT 99, 2, pp. 182-186.
Wiemann J.M., Harrison R.P. (1983) Non-language aspects of social interaction, In Wiemann J.M., Harrison R.P. (eds.)
Nonverbal interaction, Sage Publications, Beverly Hills, CA, USA.