[DADeR/Empathy] “Embodied Simulation: Speaking Up Through the Bod” – Sarah Goorhuis

Concept: embodied simulation (empathy, inner mimicry, mirror neurons)
Case Study: Nao robot

Abstract

Embodied simulation can be utilized as an attainable tool for non-verbal communication between humanoid robot and human. When we see a robot execute actions and movements, the same neural systems are activated as when we are watching humans move: seeing a robot move will activate an embodied simulation within the spectator. Although this is most apparent in observing humanoid robots, this can happen with the most abstract robot design, as long as there is something recognisable about an action or movement that we can relate to. Being conscious about the simulation that is irrevocably taking place when a human perceives a humanoid robot move can enable a stronger sense of empathy.

Speaking up through the body

In 1967, psychologist Albert Mehrabian proved that when humans try to communicate feelings and attitudes to each other in an ambiguous way, only 7% of the message is decoded through language. The impact of the other 93% of communication is divided over tone of voice (38%) and body language (55%).[1] Even though these exact percentages only apply to a specific kind of situation, Mehrabian’s research had a huge impact on realising the importance of nonverbal communication. If such a big part of human communication depends on tone of voice and body language, how can robots effectively communicate with humans to reach their design goal?
Currently, artificial intelligent devices mostly rely on written or spoken language for their communication with humans. This makes sense, since nonverbal communication between humans is so incredibly intricate and layered. It means, though, that their options for communicating are quite limited. This lack of tools for nonverbal communication is especially pressing concerning robots that are applied in social situations. Here, the robot often does not only have a supportive function but actively has to work together with their human counterpart(s) to achieve their goal. For this collaboration, smooth communication is needed.
In this paper I argue that embodied simulation can be a solution to bridge this gap in communication. I propose that a conscious utilisation of the embodied simulation that happens when a human is watching a robot is an attainable tool for nonverbal communication in human-robot interaction. I will substantiate this proposal through an analysis of the performance of a specific NAO robot called S.A.M., designed by students of the Free University of Amsterdam.

Inner mimicry, mirror neurons and empathy

In the text “Movement’s Contagion: the Kinesthetic Impact of Performance” choreographer and scholar Susan Leigh Foster pursues answers to the question “what do you feel, physically, when you watch another body performing?”[2] She does so through an examination of the ‘kinesthetic sense’ humans have, which she defines as the sensory experience provided by our bones, muscles, ligaments, tendons and joints.[3]  A first answer to Fosters question can be found in the writings of John Martin in his book Introduction to the Dance.[4] Here, he describes the reflective phenomenon of yawning when someone else yawns, experiencing fatigue when we see someone else lifting a heavy object and puckering when we see someone eating a lemon. He defines this process as ‘inner mimicry’ and argues that it enables us to understand the situation we find ourselves in.[5] He states:

Signs of fatigue in another are translated into a sympathetic awareness in our own bodies, and all types of gesture and facial expression convey meaning to us automatically because we have felt similar muscular experiences ourselves and recognize the postural attitudes and their emotional connotations as having happened to us.”[6]

So through a ‘sympathetic awareness’ of the muscles we understand the actions and corresponding emotions of others. He continues by stating that within dance performances, it is the dancer’s sole duty to “lead us into imitating his actions with our faculty for inner mimicry in order that we may experience his feelings.”[7] So through ‘inner mimicry’ Martin relates kinaesthetic sensations, performance and empathy to each other.
Martins findings seem substantiated in 2002 through the discovery of mirror neurons by neurophysiologists Giacomo Rizzolatti, Luciano Fadiga, Leonardo Fogassi and Vittorio Gallese. In their text “From Mirror Neurons To Imitation: Facts and Speculations,” they point to the existence of neurons in multiple areas of the cortex that fire both when a certain action is executed and when the action is seen as executed by others.[8] The authors describe this relation between observed and performed action through a metaphor of ‘resonance’:

It is as if neurons in these motor areas start to “resonate” as soon as the appropriate visual input is presented. This “resonance” does not necessarily produce a movement or an action. It is an internal motor representation of the observed event which, subsequently, may be used for different functions, among which is imitation.[9]

This sounds very similar to Martins statement that while watching dancers move, “[w]e shall cease to be mere spectators and become participants in the movement that is presented to us.”[10] Mirror neurons provide neurological proof for Martins statement: even though we won’t physically be moving our body, the corresponding neural areas in our brain are active as if we are. Internally, we are moving with the dancers.
Rizzolatti et al propose that mirror neurons facilitate action understanding. Through the internal resonance as described above, the neural activity can be linked to a specific action or movement from our own (muscle) memory, which enables us to predict the consequence of that same movement through our own experiential knowledge. This interpretation suggests that the goal of the resonance in mirror neurons is to present us with an internal simulation of what another individual is doing.[11] This can enable us to ‘neurologically’ put ourselves in someone else’s shoes.
Vittorio Gallese further elaborates on this process in his text “Embodied Simulation and Its Role in Intersubjectivity.”[12] Here, several years – and many publications and research projects – later, he defines the underlying system of mirror neurons as ‘embodied simulation.’ He states that embodied simulation subconsciously “mediates our capacity to share the meaning of actions, intentions, feelings, and emotions with others, thus grounding our identification with and connectedness to others.”[13] Because the actions, intentions and emotions of others are registered through the same neural mechanisms that are activated when we experience similar emotions and sensations ourselves, we are capable of understanding the position of others. Gallese states that this is an important underlying component of empathy.[14]
With these findings, the link between perceiving movement and empathy has been identified and Martins writings on the kinaesthetic impact of dance on the spectator have been substantiated (albeit through a different terminology).

Why did I take you, my reader, down this road? I want to argue that when we see a robot execute actions and movements, the same neural systems are activated as when we are watching humans move: that seeing a robot move will activate an embodied simulation within the spectator, same as it does watching a dancer. I think this can happen with the most abstract robot design, as long as there is something recognisable about an action or movement that we can relate to. Especially, though, it takes place while observing human-like robots. In the next paragraph, I will introduce you to one of those: a humanoid NAO robot called S.A.M..

Meeting S.A.M.

S.A.M. is programmed by seven students from the 2021 Socially Intelligent Robotics course at the Free University of Amsterdam.[15] S.A.M. was designed to teach Dutch primary school children age six to eleven English words through a mix of verbal and non-verbal communication. The students set out to create three lesson plans, each plan consisting of the word, sound and movement for five animals, actions or feelings. At the end, the children were to be presented with a quiz.

I got to meet S.A.M. in November 2021, during a visit to the university. S.A.M. addressed their target audience – primary school children age six to eleven – mainly through speech, in an enthusiastic and encouraging manner (as far as intonation goes for a NAO robot). S.A.M.’s words were well-chosen to encourage children to engage and participate in the learning goal. The children were asked to touch S.A.M.’s head or feet to communicate if they need to hear a word again or if they are ready for the next part of the story. The rest of their engagement with S.A.M. was to be through speech.

For our current investigation into embodied simulation, I will focus on the third lesson plan the students devised: the story to teach children the English words for five different feelings. In this story S.A.M. experienced various events, such as meeting a scary dog and falling down on the street. S.A.M. then taught children the English translation of the words that fit the feelings they experienced during these events: ‘angry,’ ‘scared,’ ‘sad,’ ‘happy’ and ‘tired.’

To investigate how S.A.M. relates to the subject of  embodied simulation, I want to clarify why I perceive S.A.M. to be performing during this lesson plan. Marvin Carlson identifies different approaches to performance in his book Performance: A Critical Introduction.[16] The most fitting for S.A.M.’s performative situation seems to be the fact that the robot is designed to pretend to have experienced human situations and sensations. Carlson states:

The pretending to be someone other than oneself is a common example of a particular kind of human behavior to which Richard Schechner has given the title “restored behavior,” under which title he groups any behavior consciously separated from the person doing it – theatre and other role-playing, trances, shamanism, rituals. [17]

Here, a quality of performance is characterised as a distance between the self and the behaviour presented, similar to that between an actor and its character on stage. This is extremely applicable to S.A.M., since they never actually got scared from an angry dog or felt pain when they fell down. S.A.M is just pretending they did so they would be relatable to children, which would further their learning goal. With this, an interesting second layer comes to the foreground: naturally, S.A.M. is not pretending anything, the students that programmed S.A.M. are. Carlson states that according to Richard Bauman, “all performance involves a consciousness of doubleness, according to which the actual execution of an action is placed in mental comparison with a potential, an ideal, or a remembered original model of that action.”[18] In S.A.M.’s case, this doubleness is multi-layered since it is the programmers that are constructing this copy of an ideal, a remembered original model of a certain action. S.A.M. is just the medium through which to present this copy.
So S.A.M., through their programming, is performing restored behaviour. With this, we have already arrived at an advice towards robot design in itself: realising that a robot such as S.A.M. is in fact performing opens up the possibility to make use of performance research considering the stage, (multidisciplinary) storytelling, spectatorship and so much more. Utilizing this research can help shape the design of a robot to be more effective. In the current investigation, though, we are not there yet.

Are we moving yet?

With everything we have thus far talked about in mind, I want to take a closer look at S.A.M. Specifically, at the movements that accompanied the events and feelings S.A.M. described. S.A.M. used a broad range of motion, clearly separating each feeling in movement and using gestures that were very relatable. For ‘angry,’ S.A.M. stood as tall as they could and placed their hands on their hips, for ‘scared’ S.A.M. made quick and sudden movements with their head and made themselves smaller by bending their knees and crouching down. ‘Sad’ was depicted through loosely hanging arms and a head that slowly bowed down to the front. ‘Happy’ showed S.A.M. moving their arms and legs as if dancing and ‘tired’ was accompanied by a hand moving to and from their mouth to depict yawning.
These movements are very cleverly chosen. I dare to say every human has had similar responses in posture and movement while experiencing the corresponding feelings. This means all humans have muscle memory related to these feelings. This easily enables embodied simulation. When we see S.A.M. putting his hand at their mouth, the mirror neurons in our brain simulate this movement. Through this embodied simulation, we can recognise S.A.M.’s movement as similar to our own when we cover our mouth when we yawn. This enables us to understand what S.A.M. is supposedly feeling: we recognise S.A.M. as being tired. I want to argue that in this process, it does not matter that S.A.M. does not actually have muscles or that S.A.M. cannot actually experience feeling tired. As long as we can somewhat recognise the movements as human-like, or can imagine them that way, embodied simulation will still take place.
This process of embodied simulation might seem very simple or even obvious, but it is actually quite nuanced. If S.A.M. verbally told us they were tired while depicting, for example, the movement they executed with the word ‘happy,’ our embodied simulation of S.A.M.’s movement would not match our own experience of feeling tired. Here, communication would most likely fail. I want to explain this by going back to Mehrabian’s research. Considering the fact that this situation is similar to Mehrabian’s investigation of ambiguous communication between humans concerning feelings, we know that we will only rely on speech for 7% of our meaning-making process. For the other 93%, we look to S.A.M.’s tone of voice and body language. [19] Since S.A.M. does not have a human body, these non-verbal cues are very limited. What we do have though, is our embodied simulation of S.A.M.’s movements. I want to argue this is where we will look for the true meaning of S.A.M.’s message. Since this message (‘happy’) does not fit the word S.A.M. wants their target audience to learn (‘tired’), the teaching objective will fail.
It is important for S.A.M.’s design that the communication does not fail. For a useful learning process, the information has to be presented to the primary school children in a clear and understandable manner. The children will engage with the teaching material more easily if they can identify with what is being taught. So, it is in the designer’s best interest to facilitate smooth communication between S.A.M. and the children. Enabling an appropriate embodied simulation to take place within the spectator, through well-chosen robotic movements and expressions, will help S.A.M. to achieve their learning goal.

Looking at the bigger picture

S.A.M. does not exist in a vacuum of course. There are many other robots programmed similarly to  S.A.M. that need a smooth communication process to take place in order to achieve their design goal. I propose utilizing embodied simulation as an attainable tool for non-verbal communication between a humanoid robot and a human. It can enable humans to understand and empathise with the robot it is interacting with, which will place them in a position to subconsciously collaborate with the robot.
With ‘utilising embodied simulation in robot design,’ I mean being conscious about the simulation that is irrevocably taking place when a human perceives a humanoid robot move, and using it to your advantage. This is also advantageous for robot designs different from S.A.M.. Utilising embodied simulation could benefit robots that are designed to battle loneliness among elderly, through enabling a stronger sense of empathy and thus strengthening the bond between elderly and the robot, subsequently making them feel less alone. It could be an asset to robots that are designed to help people rehabilitate from physical injury, through exercising with them and enabling a better imitation of the movement-goal. It could assist robots that are designed to perform, facilitating more control over the preferred effect of the performance on the spectator. As Rizzolatti et al state, embodied simulation is “an internal motor representation of the observed event which, subsequently, may be used for different functions, among which is imitation.”^[20] Through these different functions, the possibilities for robot design are endless.
My hypothesis is that this also applies to non-humanoid robots. This calls for further investigation, which I encourage anyone who is passionate about the relation between embodiment, robot design and the performing arts to do. Through investigations like this, findings from the field of performance studies can provide an aid towards robot design.

A closing remark

You might have noticed I called S.A.M. ‘they’ instead of ‘it’.[21] This is because for me, S.A.M. became a more of a person than an object after experiencing their story. The movements that were implemented enabled me to identify with the feelings S.A.M. distinguished and to empathise with them. For me, it did not matter if S.A.M. was a person or not. I imagined myself in their shoes and subsequently understood how anyone must have felt in such a situation. In this paper, I hope to have showed that you would have too.

[1] Albert Mehrabian and Susan R. Ferris, “Inference of Attitudes from Nonverbal Communication in Two Channels.,” Journal of Consulting Psychology 31, no. 3 (1967): 248–52.

[2] Susan Leigh Foster, “Movement’s Contagion: The Kinesthetic Impact of Performance,” in The Cambridge Companion to Performance Studies, ed. Tracy C. Davis (Cambridge University Press, 2008), 46–59.

[3] Idem, 46.

[4] John Martin, Introduction to the Dance (New York: W. W. Norton, 1939).

[5] Idem, 47-8.

[6] Idem, 48.

[7] Martin, Introduction to the Dance, 53.

[8] Giacomo Rizzolatti, Luciano Fadiga, Leonardo Fogassi, and Vittorio Gallese, “From Mirror Neurons to Imitation: Facts and Speculations,” in The Imitative Mind: Development, Evolution and Brain Bases, ed. Andrew N. Meltzoff and Wolfgang Prinz (Cambridge: Cambridge University Press, 2002), 249.

[9] Idem, 253.

[10] Martin, Introduction to the Dance, 47.

[11] Rizzolatti et al, “From Mirror Neurons to Imitation,” 258.

[12] Vittorio Gallese, “Embodied Simulation and Its Role in Intersubjectivity,” in The Embodied Self: Dimensions, Coherence and Disorders, ed. T. Fuchs, H.C. Sattel, and P. Henningsen (Stuttgart: Schattauer, 2010), 77–92.

[13] Idem, 77.

[14] Idem, 81.

[15] For privacy reasons they will remain anonymous.

[16] Marvin Carlson, Performance: A Critical Introduction (London & New York: Routledge, 2004).

[17] Idem, 3.

[18] Carlson, Performance, 5.

[19] Mehrabian and Ferris, “Nonverbal Communication,” 252.

[20] Rizzolatti et al, “From Mirror Neurons to Imitation,” 253.

[21] I prefer ‘they’ over ‘him’ or ‘her’ to avoid the unnecessary gendering of sub- and objects.

Bibliography

Carlson, Marvin. Performance: A Critical Introduction. London & New York: Routledge, 2004.
Foster, Susan Leigh. “Movement’s Contagion: The Kinesthetic Impact of Performance.” In The Cambridge Companion to Performance Studies, edited by Tracy C. Davis, 46–59. Cambridge University Press, 2008.
Gallese, Vittorio. “Embodied Simulation and Its Role in Intersubjectivity.” In The Embodied Self: Dimensions, Coherence and Disorders, edited by T. Fuchs, H.C. Sattel, and P. Henningsen, 77–92. Stuttgart: Schattauer, 2010.
Martin, John. Introduction to the Dance. New York: W. W. Norton, 1939.

Mehrabian, Albert, and Susan R. Ferris. “Inference of Attitudes from Nonverbal Communication in Two Channels.” Journal of Consulting Psychology 31, no. 3 (1967): 248–52.

Rizzolatti, Giacomo, Luciano Fadiga, Leonardo Fogassi, and Vittorio Gallese. “From Mirror Neurons to Imitation: Facts and Speculations.” In The Imitative Mind: Development, Evolution and Brain Bases, edited by Andrew N. Meltzoff and Wolfgang Prinz, 247–66. Cambridge University Press, 2002.