[DADeR/Sound] “Sound” – Herwin van Ee

Concepts: sound (noise, vocality)
Case Study: Nao robot

Abstract

How could the conceptualization of ‘sound’ in theatre and performance studies be applied to improve robot design? This paper emphasizes the importance of sound in creating an identity for the Nao robot. By giving the Nao robot a custom-tailored sound identity, it can be very much possible to cross the uncanny valley as theorized in the case study. This can be done either by customizing the voice of the robot and/or the sounds it produces while moving around or carrying out tasks.

Nao robot interacting with children. Image by Leader Edutech.

Introduction

Since my recent introduction into the theoretical world of theatre and performance studies, I have been amazed by the role of music and sound, or rather the lack of attention to aurality. Hearing, as one of the two major senses – five for those who argue taste, smell, and touch to be as important as the other two – which is provoked in theatre and performing arts, deserves to get nearly as much attention as sight, in my humble opinion as a musicologist. That being said, the introduction of music to the theoretical world of the performing arts is fairly new. Through my experience in musicology, I have found this to be the result of an idiom known as the musical work as an object. This has been dominating not only our scientific field but rather the entire musical universe since the 19^th century (Cook 2014, 1). Lydia Goehr wrote a seminal piece on this philosophy called The Imaginary Museum of Musical Works: An Essay in the Philosophy of Music (1992). In more recent years, there has been a paradigm shift in the philosophy of music by authors such as Nicholas Cook. His book Beyond the Score: Music as Performance (2014) has made him a key player in the paradigm of music as performance.

In my experience is the world of theatre and performing arts one where through multimodality multiple senses create one larger experience. Therefore, while performing studies on the aforementioned art forms, it is crucial to involve sound as much, or nearly as much, as the visual aspects. This expansion of performance does not stop here. Performance studies can prove helpful in other work fields such as robot design. After visiting and working together with robot programmers and designers, what struck me most was the amount of attention put into teaching the robot to perform several tasks. What I mostly missed was attention and thought put into why the robot should perform these tasks, how they were performed, and thoughts on ethics. I left with the feeling that sound design and choices based around sound did not even remotely seem as important to them as programming the robot’s movements. This is why I have chosen sound as the main topic of my paper, as part of the experimental database ‘Dramaturgical Aid for Designing Robots’ (DADeR).

This paper will consist of two parts. In the first part, I will be discussing the role of sound and its conceptualization in theatre and performance studies. The second part will place the concept of sound in the world of robot design. In this third and last part, I will be analyzing the concept of sound as discussed in the first part as used in a case study. For this case study, I will be discussing how the use of sound can help create an identity for the Nao robot. The overall goal is to show the importance of sound within both theatre and performance studies as well as the programming and design of robots.

Sound in Theatre and Performance Studies

History of sound in theatre and performance studies

According to Patrice Pavis in the preface of Theatre Noise: The Sound of Performance, historical moments in the world of theatre and performing arts collide with new phases in performing arts. Until the turn of the millennium, “theatre was previously understood to be visual (Pavis 2011, x).” However, this is changing. An important first notion is that “the sonic aspect certainly always had its place, but it tended to serve the visual arrangement.” But because of sounds “uncontrollable subjectivity (2011, xi),” it is fairly more likely that sound co-operates with visuals instead of merely serving it. This is because sound and visuals are everywhere at every time in the world around us. They are the world around us, creating unlimited amounts of accidental sync points. “All the more, since the art of the stage, or that of music theatre, manages to combine visuality and aurality, not as the accumulation or integration of signs in a single, common spatial or sonic volume, but as a confrontation between the two structures, prompting sound and image to see or to hear each other differently (2011, xii).”

Theatre Noise

Lynne Kendrick and David Roesner discuss the existence of sound in theatre through the concept of theatre noise. The concept of noise is quite fitting because of its place in the world of sound. Theatre noise encompasses all organized sounds which are in a traditional sense not thought of as music or speech (2011, xii). When we acknowledge the existence, and thus the importance of sounds in theatre other than music and speech, we can only conclude that we live in a world of constant noise. This theatre world of aurality “is a place where friction can be thematized, explored playfully, even indulged in: friction between signal and receiver, between sound and meaning, between eye and ear, between silence and utterance, between hearing and listening.” To put it simply, there are “a plethora of possible noises in performance and theatre (Kendrick and Roesner 2011, xv).” Because these noises and their diversity are finally getting the attention they deserve, a whole new researchable world opens up. For example, it is now possible to analyze the noises of robots from a theatre and performance studies perspective. Theatre noise “articulates the reach of auditory spaces (2011, xv).”

Vocality

Alongside noise, vocality is, because of its role in the robot case study, another type of sound that I will discuss. Vocality is not just sending a message through your voice to another recipient. “This encompasses the plurality of matter and utterance, body and grain, sounds and silences, and noises that might constitute the production of and audience to ‘voice’ (2011, xxii).” Kendrick and Roesner argue, through a chapter of Marc Larrue, that the arrival of sound technology in theatre caused “the first historic split between the voice and the body (Larrue 2011, 20-21).” However, in the case of robotics, this is quite the exact opposite. The mechanical body of a robot is, by definition, unable to produce an analog bodily voice. Because robots do not have a sound box like a guitar or vocal cords like a human, they cannot produce a voice on their own but merely reproduce sound. Thus, in the world of robotics, sound technology created a merger of voice and body.

Aural spaces and Intrusive noise

In Intrusive Noises: The Performative Power Of Theatre Sound, Katharina Rost argues how and why noises affect the audience in theatre performances. This is important because “noises are not only used in order to illustrate or amplify the onstage action, but far more than that, to create an atmosphere, rhythms, and highly complex arrangements that have a powerful impact on the audience. (Rost 2011, 45).” In this quote we read that noise not only has the quality to create aural spaces that affect audiences but at the same time accentuate onstage action. This is because of the intrusive effect of noise. “The intrusive sonic effect does not just add certain information or give some emotional background stimulus; instead it has or obtains the power to touch the listener in a directly physical way and to capture the audience’s attention, with or against their will (2011, 45).”

Noise should not be seen as a duality but rather a spectrum of multiple degrees and types of noises. This makes sense, because of how we perceive noise, through “perceptual activity (i.e., hearing, listening to, being distracted etc.) (2011, 46),” should be perceived as a spectrum as well. This understanding of noise and listening leads to the conclusion that noise is not just a binary sound, but a sound rich in its diversity that can be consumed in multiple ways. Another important characteristic of intrusive noise is that the way it intrudes bodies depends on the context, the aural spaces, in which the intrusive noises exist (2011, 46-47).

Case Study

Nao Robot

The Nao Robot is a humanoid robot created by SoftBank Robotics, which has been working on the robot since 2005. This humanoid robot has 25 degrees of freedom, seven touch sensors, 4 directional microphones, two 2D cameras, and has the ability to speak and recognize twenty different languages. This fairly small robot, just under sixty centimeters from head to toes, is also fairly easily programmable (SoftBank Robotics, n.d.). The design of the Nao Robots body is not the only dimension in which it is humanoid. Its function, as mentioned earlier, all appear in similar places where humans would have them. The eyes are the camera sensors, it has touch sensors in multiple places across his body and it emanates sound from its head. These are not a complete copy of the human body, but in my opinion, this helps see the robot as its own entity instead of just a copy of humans.

The robot’s voice and playback features are not the only sounds emanating from this robot. When the robot moves any of its possible moving body parts, noises can be heard. For instance, the stretching of arms or legs makes mechanical sounds. This is similar to the cracking sounds of human joints, which sound completely different from robots. Because these types of noises appear with similar actions but at the same time sound quite different, the robot enforces its own sound identity through its actions. The noises coming from the robot’s movements are not the only way in which the robot creates its own sound identity. The Nao robot has a programmable voice. Here you can customize parameters such as language, pitch, volume, and the sound of its voice. Alongside speech, the robot also has the ability to playback sound files. Unlike in theatre and performing arts, as mentioned earlier, sound technology embodies the robot and provides it with the ability of vocality. The digital nature of the robot should be seen as a perk towards creating its own identity instead of just another gap with humans.

The Uncanny Valley

In what ways should programmers try to shape the robot, and its sound identity, to become more like humans? This is a trick question because, in my opinion, it should not. The uncanny valley, a concept theorized by robotics professor Masahiro Mori, is the reasoning behind my opinion. Basically, this concept argues that robots that look and behave more like humans, generate more affinity in interactions with humans until a specific point. Here, the robot becomes too similar to humans and therefore scares humans, generating negative affinitive responses. However, there is a point after this valley of affinity, where robots begin to generate more positive affinitive responses again. These are robots who do not necessarily look human but nevertheless feel very human in interaction. I think that the nao robot is currently somewhere relatively close to the uncanny valley. It looks humanoid and can perform several tasks just like humans, but it is not as much as a copy to make it become unpleasant or scary. However, this is very much possible if designers and programmers make behave and look even more like humans. I do believe that the Nao robot has the potential to cross the uncanny valley. The main way to achieve this is not by making it more humanlike, but rather less humanlike in its behavior. One way in which this could be done is through giving the robot its own identity. Another way in which this can be done is through creating sounds for auditory localization. Elizabeth Cha et al theorize that robots who make audible sounds while, for instance, walking around help humans identify where the robot is. This improves the interaction between humans and robots (Cha et al 2018, 440-41). If robot programmers and designers put more thought into the why and how of sound design instead of solely focussing on the what question, then crossing the uncanny valley could prove to be easier. 

Conclusion and discussion

The world of theatre and performance studies saw a great evolution in its identity around the turn of the millennium. Before this turn, these studies were dominated by an emphasis on the visual aspects of theatre and performing arts, which was so large that even aural elements were theorized through visual aspects. This paradigm shift opened up a lot of potential for new studies on the use of sound. Sound is subcategorized by Kendrick and Roesner as music, speech, and noise. According to them, noise and its intrusive nature should not be seen as a binary opposition but rather as a spectrum. This does the right to the diversity of intrusive noises. The main point in the world of sound where theatre and performance studies can provide dramaturgical aid for designing robots is in providing tools to think about robots and sound. When designing and programming robots to try and improve the levels of their interaction with humans, it is vital to first understand what makes us human and separates us from other entities. In my opinion, this is identity. As humans, we spend our whole life creating and living out our own identities. We also provide entities who do not necessarily create their own identities, such as animals, with a fitting identity we can relate to. The designers and programmers of the Nao robot can give it its own identity. By giving the Nao robot a custom-tailored sound identity, it can be very much possible to cross the uncanny valley as theorized in the case study. A couple of ways in which this can be done is by customizing the voice of the robot and/or the sounds it produces while moving around or carrying out tasks.

Because this narrative on sound in theatre is still relatively new, there is a big chance that upcoming papers and other works of literature will create shifts in our understanding of the role of sound in theatre again. Thus revisiting this paper in a couple of years will most likely result in other or even more closely specified outcomes.

Bibliography

Cha et al. 2018. “Effects of Robot Sound on Auditory Localization in Human-Robot Collaboration.” Paper presented at The 2018 ACM/IEEE International Conference on Human-Robot Interaction, Chicago, IL, USA, March 5-8. https://doi.org/10.1145/3171221.3171285.

Cook, Nicholas. 2014. “Between art and science: Music as performance.” Journal of the British Academy 2, (March): 1-25

Kendrick, Lynne and David Roesner. 2011. “Introduction.” In Theatre Noise: The Sound of Performance, edited by  Lynne Kendrick and David Roesner, xiv-xxxv. Newcastle upon Tyne : Cambridge Scholars Publishing.

Larrue, Jean-Marc. 2011. “Sound Reproduction Techniques in Theatre: A Case of Mediatic Resistance.” In Theatre Noise: The Sound of Performance, edited by  Lynne Kendrick and David Roesner, 14-22. Newcastle upon Tyne : Cambridge Scholars Publishing.

Pavis, Patrice. 2011. “Preface.” In Theatre Noise: The Sound of Performance, edited by  Lynne Kendrick and David Roesner, x-xiii. Newcastle upon Tyne : Cambridge Scholars Publishing.

Rost, Katharina. 2011. “Intrusive Noises: The Performative Power of Theatre Sounds.” In Theatre Noise: The Sound of Performance, edited by  Lynne Kendrick and David Roesner, 44-56. Newcastle upon Tyne : Cambridge Scholars Publishing.

SoftBank Robotics. N.d. “Nao.” Accessed Junuary 26, 2022.
https://www.softbankrobotics.com/emea/en/nao