The eBoy instrument

(1)

C

by

Boye Riis jr.

THESIS

for the degree of MASTER OF ARTS

Department of Musicology Faculty of Humanities

University of Oslo

October 21, 2009

(2)

(3)

Acknowledgements

First, I would like to thank my supervisor, Professor Rolf Inge Godøy, for good support and valuable feedback during my thesis. I would also like to thank Professor Marcelo Mortensen Wanderley for supervising me during my study exchange at McGill University, during the fall of 2008. Thanks to Bjarte Knappen Røed for valuable feedback on design and usability testing. Thanks to St˚ale A. Skogstad for helping me with 3D modeling and printing.

A great number of people have given me invaluable support during this project. Also, I will give thanks to the people at University of Oslo and McGill University, especially people at Robin and the FourMs lab, and the Music Technology Area at McGill. A huge thank you to my family for great support during my ups and downs while working on this project.

(4)

(5)

Abstract

In the context of controllers for sound synthesis, the keyboard and the mouse are limited as controllers. This has motivated me to develop two hand-held controllers. In this two-part Master’s project, the first part is the practical development of a digital musical instrument for real-time processing I have named the eBoy Instrument. The second part is this thesis, in which I document and present a theoretical evaluation of the eBoy Instrument.

An iterative methodology was used as a framework for the design process, in which two final control concepts were developed. The first concept is easy to use, while the second concept is harder to use. Analysis of sensor data and sound output has shown that concept number two has a better coupling between action and sound. Qualitative and quantitative evaluations have shown that 11 participants with a high degree of musical training selected concept number two as the best solution to control sound synthesis.

A theoretical evaluation, supported by observations of 11 participants, indicates that the eBoy Instrument functions well as a controller of sound synthesis.

(6)

(7)

List of Figures

2.1 Control surface for Different Strokes and Predators . . . 14

2.2 Usability Curve . . . 15

3.1 DMI representation . . . 20

3.2 Examples of alternative touch controllers . . . 22

3.3 Timbre space . . . 24

3.4 A DMI with a three layer mapping . . . 26

3.5 OpenSoundControl namespace . . . 27

3.6 Lifecycle model . . . 31

3.7 An iterative process . . . 32

3.8 Hurley’s sandwich-model . . . 38

3.9 A musical instrument interaction model. . . 38

4.1 The iterative design framework . . . 42

4.2 Prototype 1 . . . 46

4.3 Prototype 2 . . . 47

4.4 Dimensional Space . . . 49

4.5 Connection point for both concepts . . . 51

4.6 Paper sensors . . . 52

4.7 Range in volt curve . . . 53

4.8 Image of both controller concepts . . . 55

4.9 Electronic circuit of both concepts. . . 55

4.10 The eBoy Instrument framework . . . 57

4.11 The controller window of both concepts . . . 59

4.12 Breadboard position . . . 61

(12)

4.13 The synthesis window. . . 63

4.14 Sound-producing actions figure . . . 66

4.15 Mapping of action type Hit forward . . . 67

4.16 Mapping of action type Stir . . . 68

4.17 An example of action-sound couplings . . . 68

4.18 Action-Sound couplings of Hit forward . . . 71

4.19 Sonogram of both concepts . . . 72

5.1 Pictures from the test . . . 78

5.2 Participant comfort ratings. . . 80

5.3 Participant action-sound coupling ratings . . . 81

5.4 Participant realistic sound ratings . . . 81

(13)

List of Tables

2.1 Tasks for this experiment. . . 16

4.1 Three concepts . . . 45

4.2 Final two concepts . . . 48

4.3 Max abstractions . . . 59

4.4 Technical control parameters . . . 60

4.5 Synthesis abstractions . . . 61

4.6 Technical synthesis parameters. . . 64

4.7 Classification of action types . . . 66

4.8 Sound feature dimensions . . . 67

5.1 Musical tasks for this experiment . . . 78

(14)

(15)

Chapter 1 Introduction

In this chapter, I will first give the reader an introduction to control of sound synthesis¹ and various key issues when designing new musical instruments in the digital domain.

The term affordance will be discussed to illustrate how it can be used as a valuable tool in the design of both traditional and digital musical instruments.

1.1 Control of sound: A personal story

When I was 17, I started playing the double bass in a jazz band, and I have been a performing musician ever since. Before I started playing the double bass, I was already playing the electric bass, and it was not so difficult to learn to play a second instrument, although it took me some years to get used to the new, bigger instrument with all its intonation issues. An acoustic instrument, such as the double bass, offers the musician good control of sound and great sonic possibilities, as well as specific constraints. When I started using the computer in music, I was a bit disappointed. Suddenly, the control I had over its sound was limited compared to that of the double bass.

According to Wessel and Wright (2002), control of sound synthesis is very different from that of traditional musical instruments. When I play the double bass, for instance, my body and fingers are in direct contact with the instrument’s body and sound source.

The latter is traditionally a result from plucking or bowing the four strings, and the

1For those of you who may not be familiar with music technology terminology, the control of sound synthesis is equal to the control of sound when using the computer.

(16)

resulting sound corresponds to the action carried out. While in control of sound synthesis, however, the controller and sound production are usually separated as two independent devices, and these devices may be unified with different kinds of mapping strategies.

Miranda and Wanderley (2006) define this constellation as a digital musical instrument, which I will present in more detail in chapter 3. Nevertheless, when musicians hear ”I play the computer”, they often associate it with a keyboard and mouse, office tools that may not work well for control of sound synthesis. Often the sound result does not correspond to the action carried out; hitting “enter” with a gentle touch gives a vigorous sound, for instance. In live performance, musicians often prefer to interact with other control devices than a keyboard and mouse, devices which will give them more precision, lower latency, a broader range of physical actions, and better couplings between action and sound, which in turn may give the audience a better experience.

My initial goal with this Master’s project is to investigate the control aspect of sound synthesis, and this implies that I will focus more on the control device and its mappings than the sound synthesis. The design strategy used in this project is a so-called iterative methodology, which can be defined as a repetitive process containing different design strategies. In this process, the user and the designer are working together as a team, often with the aim of developing more user-friendly products (Wickens et al. 2004). The design literature used in this project merges from human factors and human computer interaction, which aims at the interaction between people and technology. I define this technology as that of everyday objects, and that such everyday objects should be easy to use because they are a part of our everyday life. I personally believe that the design of musical instruments (whether traditional or digital) has other goals than the design of everyday objects. I will now present some reflections on human factors and Human Computer Interaction in order to understand how the design of everyday objects may be used for designing musical instruments.

1.2 Designing a musical instrument

As a point of departure for this project, I started with a human factors point of view.

The reason for this was that human factors and human computer interaction are of-

(17)

ten mentioned in the computer music literature (Kiefer et al. 2008a;b, Mulder 1998, Wanderley and Orio 2002), or may also be employed because we are using the computer.

Before discussing how a musical instrument may be designed, I will present human factors and the design of everyday objects. I believe that these methods can be used as a framework for the design of new instruments. First, human factors can be seen as a special field, which aims at the interaction between humans and technology. Often human factors is associated with the term ergonomics, and the Oxford dictionary defines ergonomics as

“the study of people’s efficiency in their working environment” (Stevenson et al. 2005).

In this thesis, this working environment can be defined as the concert scene or a place where there is some sort of musical creation. Moreover, the literature looks at “(. . . ) Ergonomics and Human Factors as having closely overlapping goals with HCI, being concerned with the interactions among humans and other aspects of a system in order to optimize human well-being and overall system performance” (Sharp et al. 2007, 11).

Thus, human computer interaction (HCI) can be defined as the study of interaction between people and computers.

In any case, designing a musical instrument from a human factors point of view is a complex activity that presents several issues. These issues may be different kinds of feedback, communication, and coordination situations between fellow musicians that the designer must take into consideration. In the book The Design of Everyday Things, Donald Norman (2002) stresses the importance of conceptual models, constraints, and affordances when designing everyday objects. Everyday objects are something that is a part of our daily life, such as doors, coffee makers, water taps, and advanced kitchen systems. These artefacts should be quite easy to use precisely because they are a part of our daily life.

In order to understand the design concept by Norman from a musical instrument point of view, you can consider a well-known musical instrument such as the classical guitar. The strings could be understood as having affordances (action possibilities):

they allow the musician to generate sound by plucking the strings, and the fingerboard can be understood as being the constraint for pitch: thus the musician may lose the possibility of controlling the pitch outside the fingerboard. According toNorman, a good conceptual model contains effective use of affordances and constraints, and suggests that

(18)

conceptual models, constraints, and affordances may be used as a frame of reference to design everyday objects, which may also aim at usability or ease of use: “Usability is primarily the degree to which the system is easy to use, or user friendly” (Wickens et al.

2004, 59). Normansuggests that designers should move their focus from technology to the needs of the user, and therefore employ a so-called user-centered design: This is “. . . a philosophy based on the needs and interests of the user, with an emphasis on making products usable and understandable” (Norman 2002, 188). In order to develop such a user-centered design, Normanagain stresses the importance of a good conceptual model, constraints, and affordances.

The term affordance has been used in various ways, especially inside the HCI community (Sharp et al. 2007). A clarification of this term is needed to avoid further confusion.

The term affordance was coined by the perceptual psychologist J.J. Gibson (1986) to refer to the actionable properties between the environment and an actor (a human or an animal). Affordances are relationships, they exist everywhere in the environment, and they do not need to be visible, known, or desirable. Gibson is interested in how we perceive the environment with direct perception: “The affordances of the environment are what it offers the animal, what it provides or furnishes, either for good or ill” (ibid., 129).

The existence of an affordance is independent of a person’s past knowledge and experience, but the person’s ability to perceive the information may be based upon culture or experience. I will now provide you with an example of an affordance. Try to imagine sitting in the office and observing a pen. You are most likely to pick up the pen and start writing. All this happens automatically because we have a good understanding of its action possibilities, thus the pen affords writing. We do not observe this pen as a com- position of different materials such as metal or ink; we directly perceive the pen as a unit and start writing. There must be perceptual information that specifies the affordance for it to be directly perceived. This direct perception is only present when the person has the necessary writing skills. Thus, the pen may not have affordances for a person without writing experience.

InThe Design of Everyday Things,Normandisagrees that the existence of affordances is only dependent upon the action capabilities of a person. He introduces so-called per-

(19)

ceived affordances, meaning that the existence of affordances may also be depended upon the person’s perceptual capabilities: “The term affordance refers to the perceived and actual properties of the thing, primarily those fundamental properties that determine just how the thing could be used. (. . . ) A chair affords (is for) support and, therefore, affords sitting. A chair can also be carried” (Norman 2002, 9). In another quote, he states a clear distance to Gibson by coupling affordance with past knowledge and experience:

The notion of affordance and the insights it provides originated with J.J Gibson, a psychologist interested in how people see the world. I believe that affordance result from the mental interpretation of things, based on our past knowledge and experience applied to our perception of things about us (ibid., 219).

According to Norman, the affordance of an object should suggest its usage: “Affor- dances provide strong clues to the operations of things” (ibid., 9). Therefore, a person should know what to do only by looking at everyday objects. More advanced everyday objects may have labels or instructions (e.g. advanced kitchen systems), but not simple everyday objects (e.g. coffee makers). In other words, if the person does not understand or needs to repeat the action, the affordance has failed and the action was inefficient.

Thus, perception by a person may be involved in characterizing the existence of a per- ceived affordance. Moreover,perceived affordances are of little interest if they are invisible to the person. According to Norman, it is the designer’s responsibility to make sure that all possible actions are visible, and hence the existence of an affordance is highly dependent upon the person’s possibility to perceive them. In contrast, a Gibson affordance is always there to be perceived and it is the person’s responsibility to be able to perceive it. Furthermore, if that person fails to perceive it, Norman suggests that the affordance failed and may be considered as a bad affordance. In design, the perceived affordance might be manipulated to match the perception of the person. Gibsondisagrees with the view that affordances can be modified to fit with the person’s ability to perceive them by defining affordances as invariants:

The affordance does not change as the needs of the observer changes. The observer may or may not perceive or attend to the affordance, according to his needs, but the affordance, being invariant, is always there to be perceived. An affordance is not bestowed upon an object by need of an observer and his act of perceiving it. The object offers what it does because it is what it is (Gibson 1986, 139).

(20)

This quote clearly points out that a person cannot modify or add an affordance; the affordance is there to be perceived. The ability to perceive the particular affordance depends upon the person’s past knowledge and experience. Direct perception is possible when there is an affordance and the information necessary to specify that affordance.

From my point of view, it may seem that the concept of perceived affordances tells us something about usability; the characteristics of an affordance may be based upon the person’s past knowledge and cultural background. I have now presented two different views on the term affordance. I will now explain why I prefer Gibson’s affordance by presenting affordances of traditional musical instruments and why these reflections are important in the design of digital musical instruments.

1.3 Affordance of traditional musical instruments

Musical instruments offer the musician a set of different affordances, which performers will use differently. The double bass, for instance, has a complex set of affordances. This instrument is difficult to master and, as stated above, it takes a lot of practice to handle a very basic musical task. The musician produces sound by setting the string into motion, and this sound will be amplified by the instrument’s body. The musician can manipulate the pitch by moving the left hand’s fingers around the fingerboard. This is where the affordances are presented to the musician. In order to perceive new affordances, musicians always experiment with new sets of styles and techniques, and this results in hours of practice. The latter is closely linked with the notion of ”effort,” which is often represented in the performance (complex control) of musical instruments (Mulder 1998), and research has shown that complex control is the factor which makes musical instruments interesting to interact with (Hunt et al. 2003).

How do action affordances appear in digital musical instruments? According toKvifte (1989), playing a musical instrument (either new or old) is a complex operation, which combines instrumental control, music theory, and musical sound. Hence, one can use classification and playing technique of new and old instruments as a frame of reference to explore new action possibilities of instruments, either in the acoustic or digital domain.

Nevertheless, the double bass, for instance, has been played in various ways throughout

(21)

its existence, and one may say that bowing has been the most frequently used playing technique for this instrument. As a result from hundreds of years with experimentation and different playing styles, new playing techniques have been discovered, such as slapping. The characteristic of slapping is a percussive sound, in which the musician hits the string with his right hand. This new percussive affordance has always been a part of the instrument, and the musician has been able to perceive this new affordance as a result of learning, playing technique, classification of instruments, and knowledge about sound features. Thus, affordances of musical instruments may be perceived as a result from time spent on learning playing technique (effort) and knowledge of instrument classification.

In this project, I look at usability as the degree to which the instrument design is user- friendly. The user in this project is a musician with a high degree of musical training, and from personal experience as a professional musician, I find instruments user-friendly when there is a good conceptual model, specific constraints, and with such knowledge I have a stronger ability to perceive and understand the affordances of the instrument.

With this project I hope to spark a different approach to the design of digital musical instruments by including the user as a part of the design team. Therefore, this project focuses heavily on the user’s experience, instrument classification, description of action-sound features, and a good conceptual model. The latter emphasizes the use of different kinds of interaction metaphors, which may be based on traditional musical instruments. These factors will hopefully contribute to making digital musical instruments user-friendly.

1.4 Thesis outline

In this thesis I will mainly focus on the control dimension and spend less time on sound synthesis, which will become more of a tool for testing my hypothesis.

Chapter 2 contains a preliminary study on usability based on the testing of two software- based musical instruments. This test is a qualitative observation study on 5 participants with a high degree of musical training and programming skills. The aim of this test is to show that the keyboard and mouse are limited as control devices for sound synthesis. This will motivate me to design two moveable hand-held

(22)

controllers.

Chapter 3 presents a theoretical review of digital musical instruments where the control surface is in focus. First, a classification of different control surfaces will be presented, before a reflection on sound production and different mapping strategies.

A critical reflection on design and HCI is carried out, and methods for DMI design is also presented. In the end, the action-sound perception loop is presented to illustrate active action-sound couplings in musical instrument interaction, and how this theory also can be used as a guidance to design action-sound couplings in digital musical instruments. I shall occasionally refer to these theoretical aspects in later chapters, and discuss them in chapter 5 along with the results from development and tests.

Chapter 4 gives a brief overview of the design and technical assembling of the eBoy Instrument. An iterative methodology is used as the framework for this design process, which is divided intogoals,concept development,detailed design, and usability testing. First, two prototypes were developed, which resulted in two iterations.

Then two final concepts were developed. Later on, a more technical description of hardware and software will be presented. A description of action-sound features will also be presented, in which analyses of sensor data and sound output are presented to illustrate the action-sound couplings of the eBoy Instrument.

Chapter 5 contains a comprehensive musical test, which evaluates the eBoy Instrument by employing 11 participants with a high degree of musical training on traditional acoustical instruments. The aim of this test is to compare two control surface concepts and select one of them based on qualitative and quantitive data collection from the 11 participants. I will provide the reader with a description of the action- sound features of the eBoy Instrument. The relationship between design, HCI, and the eBoy Instrument will be discussed.

Chapter 6 will sum up all the work on this thesis and further work of this project.

Appendix A gives an overview of experiment materials used in this project.

(23)

Appendix B lists the contents on the accompanying CD-ROM, which consists of the materials used in chapter 4 and 5 to develop the eBoy Instrument.

1.5 Abbreviations

HCI = Human Computer Interaction DMI = Digital Musical Instrument

Max = Max/Msp/Jitter - abstractions will be written with boldtext OSC = Open Sound Control

GUI = Graphical User Interface HID = Human Interface Device FSR = Force Sensing Resistor

(24)

(25)

Chapter 2 A preliminary case study of usability

I shall here present a preliminary study on usability of two software-based musical instruments. The aim of this chapter is to demonstrate possibilities and limitations of the keyboard and mouse as the control dimension of two software-based musical instruments, and to motivate the development of two hand-held controllers.

In this case study, I have chosen to employ the so-called usability test. The usability test is a methodology for evaluating products on participants, and it gives an indication as to how real participants use the product. Before presenting a more detailed description of this test, I will present the two instruments which are to be tested.

The two instruments to be tested are Different Strokes¹ by Zadel (2006) and the Predators from ixiQuarks² by Magnusson (2006). The ixiQuarks is a software family containing several instruments and audio effects for live improvisation. They both represent so-called software-based musical instruments where the musical control takes place between the computer control surface (usually a keyboard and mouse) and a GUI containing abstract objects, figures, or drawings. The keyboard, mouse, or additional digitalized tablets may control these software-based musical instruments. For the ixiQuarks, MIDI controllers can also be used to control only filters and effects. In this test, I have decided to use the laptop as the musical instrument and only the built-in trackpad and keyboard as the control surface. The Predators is based on moving abstract objects around on the screen, and interaction between these abstract objects will result in different sounds.

1http://www.music.mcgill.ca/~zadel/differentstrokes/

2http://ixi-audio.net/content/download/ixiquarks/index.html

(26)

(a) Different Strokes (b)Predators (ixiQuarks) Figure 2.1: An illustration of keyboard and trackpad as the control surface.

Different Strokes, on the other hand, is based on freehand drawings of different strokes, and these strokes are mapped to sample playback. The interaction between a participant and the control surface for both instruments can be seen in Figure 2.1.

The primary aim of the usability test is to get direct information from how participants are using e.g. software-based musical instrument design, and what their exact problems are with the concrete design tested (Nielsen 1993). This test is conducted in a controlled environment, and this environment may be a usability laboratory or a room, where the performance of products is measured with pre-made tasks. Moreover, there are two important factors before considering a test design: Reliability: did the experimenter get the same results when repeated?, and validity: did the results reflect the usability issues one wants to test?

When recruiting participants, it is important to first consider that expert users are faster than novice users, and 5-12 participants are recommended for a test to be valid.

However, this small number of participants will only give an indication as to how the design is actually working. The aim is not to get reliable statistical data, but rather to get direct input from users to detect usability issues during a design process, for instance.

In the context of website design, Nielsen (2000) has developed a mathematical model to find usability problems due to the number of test users [N(1−(1−L)ⁿ]. The N is the total number of usability problems in the design andL is the proportion of usability problems discovered while testing a single user. Plotting the curve for [L = 31 %] gives

(27)

0 3 6 9 12 15 0 %

25 % 50 % 75 % 100 %

Numbers of Test Users

Usability Problems Found

Figure 2.2: Usability curve, from that ofNielsen (2000). Please see text for explanation.

the following result as shown in Figure2.2. Moreover, the curve tends to flatten out when there is more than 5 users, and because of this, it is possible to detect approximately [85 %] of the usability issues with only 5 users. Testing more than 5 users may not result in new valuable insight for the experimenter because you will most likely keep observing the same usability problem several times. Thus, Nielsen argues that it is better to test 5 users three times than testing 15 users only once. This requires less users, which in turn gives more time to correct usability problems for a second test.

2.1 Method

Participants. Five participants were recruited, and they all had a high degree of musical training on traditional musical instruments as well as programming skills. The participants were tested under similar conditions at the Music Technology Area at McGill University Montreal.

Procedure. This usability test was structured into four stages as suggested by Nielsen (1993): preparation, introduction,the test itself, anddebriefing. In the preparation stage, several pilot tests were carried out in advance to prevent flaws in script, test design, technical issues, and instructions with people from the lab. During the introduction stage, the participants were told to fill out a form on arrival. The experimenter briefed the par-

(28)

Task 1 Activate the instrument

Task 2 Navigate your way around inside the instrument Task 3 Change sounds or sound samples

Task 4 Improvisation

Table 2.1: Tasks for this experiment

ticipants with all necessary information regarding the two instruments to be tested and familiarization with the operative system used. Moreover, participants were encouraged to feel free to verbally express themselves during the test procedure. During the test stage, the participants carried out the same tasks for both instruments. Four tasks were carried out containing a varying degree of complexity, from familiarization to improvisation, which can be seen in Table 2.1. Together with instructions, they were given the first task and had the opportunity to pose questions to the experimenter regarding the test procedure, because the experimenter may not respond to questions during the test procedure. During this stage, the aim is to observe how capable the control surface is of supporting the participant’s expectation. If the participants are asking questions such ashow is this working, the experimenter might not answer directly, but may respond by asking did you expect this to happen?

The first task was to activate the instrument, create sound, and then quit the instrument. The purpose of this task was to give the participants a sense of how the instruments were working in general. In task 2, the participants carried out several sub-tasks such as selecting different sounds, effects, and moving objects or drawing different strokes. In task 3, they were changing sounds for each instrument, and in task 4, the participants were told to improvise based on previous tasks. After the test, a debriefing was conducted, in which each participant completed a questionnaire concerning matters such as the operative system they were using, musical performance, digital music performance, and programming experience.³ This information is valuable for the design process later on in this thesis.

The participants were seated in a room in a front of a laptop computer, and a hi-fi stereo system was used to generate load sound. Both programs were running on a Mac- book Pro 15” laptop using OS X 10.5. A video camera was placed beside the participants

3All test materials can be located in Appendix A.

(29)

exclusively to record the hand interaction with the laptop and to record spoken words for further transcription.

2.2 Results

Only qualitative data were collected during this test, and according to Nielsen’s mathematical model, the test should make it possible to detect approximately [85 %] of the usability issues. Only the most important words were transcribed from the video record- ings. Based on video observations most of the participants performed all tasks very well.

After studying the spoken words and video observation of each participant, I found that both instruments were providing possibilities as well as limitations. The most positive aspect of both instruments is that the control surface and the instrument are the same physical device. Hence, one can download the software from the Internet and use the instrument right away without the need of additional units to control the sound synthesis. However, from participants’ comments and observations, this control dimension has more limitations than possibilities. Participant 3 wished for a larger control surface than the built-in trackpad: “The trackpad does not lend itself to this application very well because you are trying to make some sort of expression stroke very limited what you can do. Probably a mouse will be better.”

Furthermore, in musical performance, timing is a very important function and participant 2 commented this: “For the Strokes one, Different Strokes. As what I felt like I wanna try to make something that is more regular in timing and the fact that I can’t control timing seems bad, but here, maybe because there is no timing mechanism at all.”

This timing mechanism might have been easier to control if the control surface itself were bigger and thus might give the participants the possibility of using larger arm movements.

Hence, control of timing on a little control surface, such as the built-in trackpad, may not be a good design solution. The area is too small and for the Different Strokes, the control surface may correspond to the size of the screen resolution, e.g. a touch screen containing both GUI and control surface. From video observation, all participants sat down to perform all tasks, and one may believe that an additional moveable controller would give the performer the possibility of moving while playing the instruments.

(30)

2.3 Conclusion

The evaluation of both instruments provided an indication as to how participants with a musical background are interacting and thinking about two software-based musical instruments. As stated by participant 3, the trackpad may not be suitable for a musical expression task where the participant accounted for larger arm movements. Moreover, in the above results, the last task was confusing for some participants, for the reason that a particular musical expression may be very difficult to master for this test without the necessary practice time. The principle of this usability test was based upon the testing of commercial products, and this procedure may not be suitable for testing musical instruments. I assume that this test was possible to implement because both instruments were easy to use. Due to this ease of use, the Predator was considered more a toy than a musical instrument by most of the participants.

My conclusion from this case study is to highlight the importance of employing musical tasks that reflect a true musical situation. This should stand as a fundament for musical instruments that are easy to use, and the more complex ones. A usability test of musical instruments may not be based upon commercial everyday objects such as the coffee maker or advanced kitchen systems. From my point of view the usability test can be used as a frame of reference for evaluating musical instruments.

The keyboard and mouse are limited as controllers. Hence, I wish to develop a hand- held, moveable controller, namely the eBoy Instrument. However, on my way towards to the description of the eBoy Instrument, I shall discuss some issues of DMI, design, HCI, and the action-perception loop in the next chapter.

(31)

Chapter 3 Theory

Here, I will present three theoretical aspects that will stand as the framework for this thesis and the development of the eBoy Instrument. The aim of this chapter is to give the reader an understanding of key aspects when developing digital musical instruments for real-time performance.

3.1 Digital Musical Instrument (DMI)

A DMI can be defined as a musical instrument containing two independent units such as a controller¹ and sound production as shown in Figure 3.1. In this figure, primary feedback represents the physical interaction between input actions and the controller, while the sound production represents secondary feedback or sound output. These two units are related by different kinds of mapping strategies, which refers to the relationship between the output from the controller and the input of the sound production unit (Miranda and Wanderley 2006). The term controller is associated with one or more sensors that are assembled in the same unique physical device. Furthermore, the ”controller”

is the input channel of the DMI, and it is here that the physical interaction occurs between the DMI and the performer’s actions, such as hand movements and body or hand manipulation. As previously stated, the action control and sound production are two independent units. In traditional musical instruments (e.g. the double bass), however, this separation is impossible because the strings of the double bass represent both action

1Input device in HCI

(32)

Sound Production Controller

Primary feedback

Secondary feedback

Action input

Mapping

Sound output

Figure 3.1: DMI representation, adapted from Miranda and Wanderley (2006, 3). Please see text for explanation.

control and sound production.

First, it is essential to present a classification of controllers. In this section and later on, I will use the term controller instead of gestural controller when referring to the physical control unit of the DMI. Later on, reflections on sound production and different mapping strategies will be presented.

3.1.1 Controller

Miranda and Wanderley (2006) have created a taxonomy of available controllers and suggest that a controller can be classified into three main categories. This taxonomy will help us understand more of different controllers and their qualities. Moreover, to prevent confusion regarding terminology, I will use the term alternative controller instead of alternate controller, which may be use in some references.

Augmented musical instruments : These controllers consist of existing traditional musical instruments that may be equipped with additional sensors for controlling sound synthesis. One example can be a trumpet with sensors such as several FSR’s (Force Sensing Resistor) and accelerometers. The advantage of this type of controller is that the performer does not need to learn new playing techniques.

Instrument-like/inspired controllers : In the design of instrument-like controllers the mission is to model existing instruments as closely as possible. The advantage of this design approach is a familiar appearance, in which the musician does not need to learn any new playing techniques. One example of such controllers is the MIDI

(33)

keyboard, which, for instance, may look like a piano. When designing instrument- inspiredcontrollers, the designer has to concentrate more on the control surface than on trying to model the whole instrument. Moreover,instrument-inspired controllers can also be sub-classified as controllers that model acoustic instruments; e.g. the piano keyboard with weighted keys, and simulation of mechanical functions that are controlled by force feedback technology. Controllers inspired by acoustic in- struments: e.g. instruments that do not intend to reproduce the real instruments exactly.

Alternative controllers - As a contrast to the above classifications, alternative controllers are not directly inspired by existing musical instruments. These controllers may model objects and artefacts from our everyday life. However, this serves the designer some challenges, such as mental models, interaction metaphors, playing technique, classification, and standardization. Mulder (1998) suggests a structur- ization of these controllers into three categories:

Touch controllers: These controllers may augment everyday objects (e.g. a plastic material), and one example of such an alternative controller is the T-Stick developed by Malloch(2008). The control surface of this DMI consists of a large plastic tube equipped with several sensors, and the performer can use the entire surface of the plastic tube to control sound synthesis in real-time in a fixed space. When using a touch controller, the performer always needs to be in total physical contact with the control surface to achieve sonic results, and hence the controller provides the performer with total physical contact. Another good example of an alternative controller is the GyroTyre by Sinyor and Wanderley (2005). The control surface consists of a spinning bicycle wheel, equipped with sensors to control sound synthesis in real time. The sound is controlled by the speed of the wheel, which gives the performer continuous proprioceptive feedback.

Both controllers presented here can be viewed in Figure3.2.

The DMI’s presented above represent touch controllers that augment everyday objects, and during the last decade, the use of office tools (mouse, keyboard, and digitalized tablets, for instance) has become more popular due to low costs. These controllers are usually equipped with USB connections that make it easy to attach the controller to

(34)

(a)The GyroTyre (b)The T-Stick Figure 3.2: Examples of alternative touch controllers

any computers using the well-known HID protocol. The control surface of this software- based laptop instrument may consist of a digitalized tablet or keyboard and mouse, as presented in the previous chapter. Controllers from the computer game industry have become more popular in recent years, especially around the programming environment of Max (Wright et al. 2003), in which the user can attach a game controller to the computer and develop customizable patches to control sound synthesis. It is also possible to develop so-called home-made or cheap controllers. Jensenius et al. (2006) have built a low-cost controller named the Cheapstick. In order to construct such a low-cost controller, the inner components of game controllers may be used as the interface along with FSR’s made of paper.

Expanded-range controllers: Compared to touch controllers, these alternative controllers may not require or require only limited physical contact with the control surface.

The Theremin, for instance, will only have a limited range of sound-producing actions.

The control surface of the Theremin consists of an electric sensory field, which is gener- ated by two antennas. To produce sound, the performer needs to move his hands inside this electric field. Thus, the performer can perform movements outside the sensory field without musical consequences.

Immersive controllers: Compared to touch and expanded-range controllers, these alternative controllers have few or no restrictions on sound-producing actions. These controllers are best suited for adaption of specific action capabilities and the needs of the

(35)

performer. Thus, a performer can wear a body suit equipped with sensors to track all human movements, and in this way, immersion is achieved. When using this alternative controller, the performer is always inside the sensory field and cannot ”escape” from it.

The above controllers take into account different types ofphysical control surfaces, while immersive controllers take into account the spatial state of different virtual control surfaces and mapping strategies are being used. According to Mulder (1998), the spatial state of the immersive control surface can be grouped as follows:

1. Internal controllers: The spatial state of this control surface is the same as the physical shape of the human body itself. The performer can wear a body suit equipped with sensors.

2. External controllers: The spatial state of this control surface is considerably dif- ferent from the physical shape of the human body. The spatial state is separate from the human body, and it may be difficult to implement a physical shape. One pertinent example is the use of a data glove, which allows the performer to use the position between two fingertips as a control parameter for musical control.

3. Symbolic controllers: This control surface is not visible at all, and it is recommended to use formalized actions (e.g. sign language or conducting movements) to produce sound. An example might be a hand movement for trigging musical sounds (e.g.

musical sound from a drum set).

According to Miranda and Wanderley(2006), there exist a few more controllers using actuators (haptic feedback simulation, for instance), and due to space limitations I will only present their names: haptic music controllers,high-end motion trackers,collaborative and web-based controllers, and adaptive controllers. I will also mention software-based musical instruments, in which the laptop itself is the control surface. It is here that the physical interaction between action inputs and control surface occurs. The musical creation is mostly based on navigating inside a GUI.

(36)

"LESS BITE"

"BRIGHTER"

Figure 3.3: This figure illustrates a two-dimensional timbre space, based on that of Wessel (1979). The black arrows and dots indicate a timbre distribution. The horizontal dimension indicates the quality of the ”bite” in the attack, while the vertical dimension indicates the bright- ness of the spectral content.

3.1.2 Sound production

As shown in Figure 3.1, the sound production unit produces sound output or secondary feedback. This can be done by using different kinds of sound synthesis models. Wessel (1979) suggests that perceptual research on musical timbre can be used as a model, or what is sometimes called a ”metaphor” for designing musical control of sound synthesis.

In order to do so, one can develop a set of rules based on results from timbre discrimination studies, which can be a two-dimensional timbre space, as shown in Figure3.3. Moreover, this figure can be expanded to represent a two-dimensional space GUI inside a DMI, in which a slider may control the dimension that is related to the shape of the spectral energy distribution, while a button may control the dimension that is related to the attack rate of the timbre. Building a control dimension based on a perceptual model, such as the two-dimensional timbre space, may provide some sort of predictability for the user.

Thus, a timbre space as control dimension of sound synthesis may create a good coupling between action input and sound output in DMI performance.

(37)

3.1.3 Mapping

I have now presented classification of controllers, mostly for physical control of sound synthesis, and some reflections on sound production. When designing new musical instruments, the designer is faced with several challenges, such as which sensors to use, how will the musicians use the instrument, which sound will it make, and which sensors should control which aspects of the sound. The connection between the controller and sound production is mapping, as shown in Figure 3.1, and mapping is one of the most important factors when developing new instruments: the glue that connects the physical actions with the sound synthesis – and thus mapping defines the control dimensions of the instrument. The interesting question here is: how complex should the mapping be?

Should the designer focus on ease of use, or more complex mapping strategies?

Much work has been done on mapping, andHunt and Wanderley (2002) present how complex mapping strategies (inspired by acoustic musical instruments) can be used when developing DMI’s for expert users. The term “complex mapping” can be understood not only by how many parameters that exist, but also by how the mapping is carried out.

In several case studies, complex mappings turned out to be more intuitive in the end, in which expert users may achieve more control of the sonic result. An example of typical mapping or so-called single layer mapping, is technical control parameters (transducers) that are directly connected to technical sound parameters, e.g. a pressure sensor controls a pitch or amplitude parameter.

To develop more intuitive mapping, a 3-layer mapping approach is suggested by (ibid.). Compared to a single mapping layer, a third mapping layer focuses on couplings between action-sound types², while first and third layers are device-specific (controller and sound synthesis). In this thesis, action types are defined as descriptive parameters, e.g. a specific arm movement with reference to the human body, while sound types may be perceptual meaningful parameters, e.g. timbre as presented above. As shown in Fig- ure 3.4, these action-sound types are mapped with one-one mappings; however, these simple one-one mappings may already consist of complex, cross-coupled technical parameters in the first and third layers. From now on, the second mapping layer will be defined

2The expression ”Gesture-sound semantics” may be used in some references (Malloch et al. 2008).

(38)

control parameters

Controller

action types sound types

Sound production

synthesis parameters

First Mapping Layer

(Technical)

Third Mapping Layer

(Technical) Second

Mapping Layer (Action-Sound couplings)

Action input Sound output

Primary feedback

Secondary feedback

Figure 3.4: This figure illustrates a DMI representation with a 3-layer mapping approach, derived from that of Malloch et al. (2008). The mapping between action types and sound types are one-one mappings, which already may consist of complex cross-coupled mappings of technical parameters in the first and third layers.

as action-sound couplings.

Malloch et al. (2008) have adopted this three-layer mapping approach by developing a mapping tool for collaboration between DMI designers and musicians called the os- cMapper. The aim of this tool is to create more intuitive solutions to construct different mappings for DMI’s without requiring a high level of technical knowledge, making the technical data more transparent. This mapping tool can make it easier for connection between control devices and sound synthesis, in which the user focuses on action-sound couplings rather than mappings between technical parameters. Furthermore, a GUI is constructed to create, modify, and disconnect mappings between data streams and synthesis parameters. Another goal is to enable a plug-and-play approach, in which the user can easily plug in a device, and the system will then recognize it. This is especially important for alternative controllers, which often consist of one unique prototype. The first and second mapping layers in the oscMapper tool are based upon the Gesture Descrip-

(39)

/controller

/mouse/1/x <value>

/mouse/2/x <value>

/mouse/3/x <value>

Figure 3.5: An example of an OpenSoundControl namespace.

tion Interchange Format (GDIF), which is a result from a collaboration between different research institutions (Jensenius 2007).

One of the aims of this format is to improve the structure of incoming data, which can either be real-time control of sound synthesis (as the mapping tool presented above) or streaming and storing data simultaneously when analyzing musical movements. Often the problem is the use of different devices that have different protocols. Developing a standards format or device library, e.g. based on device classification, may simplify this challenge. By using this standardization, the user can plug in a device and the system will recognize it. Developing a general description for alternative controllers is more challenging compared to established controllers, since they mostly consist of a single prototype.

The representation and communication of GDIF is linked with OSC (OpenSound- Control), which is a protocol containing a sender and a receiver with a given namespace address. This namespace can be structured as a hierarchy, in which lower and higher data are separated with the forward slash (/) as illustrated in Figure 3.5. Furthermore, this protocol is optimized for broadband speed, by means of fast transportation of device data.

Hence, OSC is a suitable communication protocol, not only for its namespace advantage, but also for communication between devices in real-time situations. Malloch et al.(2008) stress the importance of a strong OSC namespace when using the oscMapper tool.

According toJensenius(2007), the GDIF is structured into these these levels, however, I shall only present the layers which are most relevant for DMI design according to Malloch et al. (2008).

Raw layer is passing raw data from the connected device (e.g. a micro-controller), and the namespace should reflect the original connected device. The data in this level are not processed or scaled into meaningful ranges. An example of a simple device,

(40)

in which the n is the ID number given when using the oscMapper tool:

/<device name>.n/raw/rotation/xyz <3 values - no units>

Cooked layer is similar to the raw layer, but the data are passing different types of filtering operations (using median and/or low pass filter to remove noise and spikes in data). The data is normalized, or given a unit value according to the input provided by the sensor. A gyroscope may end up as:

/<device name>.n/cooked/rotation/xyz <3 values - deg/s or rad/s>

Descriptive layer is used to code data resulting from raw and cooked layers. It is suggested that this layer is structured into device layer, body layer, and environment layer. Compared to the levels above, the device layer may describe how the processed data are in relation to a device, while the body layer may describe them in terms of the human body, e.g. holding the device with either your left or your right hand. In the context of DMI building, the instrument layer will provide instrument- related actions. In this example, the action type ”jab” is given a normalized unit:

/<device name>.n/instrument/jab <0. - 1.>

According toJensenius(2007), functional and meta layers are also a part of the GDIF format, but this will not be of relevance to this project.

3.2 Design, HCI, and DMI

I have now presented and defined what DMI is. Furthermore, I would like to present different design methods based on findings in the DMI and HCI literature. Before presenting design methods for DMI development, I will here present some reflection on design and HCI.

3.2.1 What is design?

Designing either physical or graphical artefacts is a major challenge for the designer, as he/she has to deal with multiple processes that are likely to create food for thought.

These thoughts and ideas will eventually end up as a product. It may be an ingenious

(41)

product, but not for users who may want something else or are not able to use it. When designing new technology (a digital musical instrument), it is important to include the user (musician) in the design process so that the finished product can meet the user’s needs and requirements. Design is about creating artefacts for the community, but what, actually, is design? Design seems to be everywhere; thus, in research, scientists may need to design their own instruments for experiments, and HCI developers need to design prototypes to be tested by users.

Fallman(2003) intends to address what design ”is” from a HCI perspective, and how design is related to HCI, by presenting three competing accounts:

• The conservative

• The romantic

• The pragmatic

Theconservativeis a design situation based on standard methods such as requirements and guidelines. Here, the designer is what we may call impersonal and inside a glass box;

thus, there is no space for individual creativity. This means less freedom as an individual designer, an area where development of practical artefacts based on scientific problem solving is the setting. This consists of three major stages: First, the designer analyzes the problem solved, second, he/she synthesizes a solution, and third, he/she evaluates the outcome. All these stages are based on well-documented methods, and ought to be carried out in a linear manner. On the other hand, the romantic stems from art, music, and drama. The designer can be considered as the creative genius. Here, the resulting artefact is based on aesthetics and abstract thinking where tools and aesthetics are woven together, but the difficulty here lies in the separation of these two. In stark contrast to the impersonal,conservative, rationally thinking designer, theromantic(individual) designer is the creative genius who may develop non-practical artefacts. The design process is guided by the designer’s taste and values, and this gives the romantic designer a mystical character. According to the romantic designer, design is not about developing useful, practical artefacts, but rather an art form. As a contrast to both previous accounts, the pragmatic account is inspired by pragmatism and holds that design is always carried

(42)

out in a specific design situation. Since every design situation is different, the ability to deal with and understand the outcome of different practical situations is more important than specific theories and methodologies. They believe that artefacts should emerge from studying human activities, culture, experience, emotions, and so forth. The pragmatic designer may use iterations, and because of this, the process may be called hermeneutic in the sense of a recursive or circular procedure, reminiscent of the so-called ”hermeneutic circle” in humanities research.

3.2.2 Design and HCI

According to Fallman (2003), thesketch can be seen as a valuable dialogue between the designer (consultant), other designers, and customers. Sketching is not only a communication tool, but also an archetypical design activity. Hence, it allows the designer to shape new ideas. These ideas can be expressed with a simple pen and paper, without the need of physical constructions or technology. In HCI, however, there are different kinds of ideas that may be difficult to express with the help of only pen and paper. These ideas may only be expressed with physical interaction, audio design, haptics, and so forth. In HCI, sketching is termed prototyping, which allows the HCI designers to develop card- board modeling, visualization, and programming languages. Hence, the prototype is the valuable communication dialogue between the HCI designer and costumers.

The term prototyping is also defined as “(. . . ) a limited representation of a design that allows users to interact with and explore its suitability” (Sharp et al. 2007, 530).

This prototyping activity can be divided into so-called low fidelity and high fidelity prototyping. Low fidelity prototyping is a paper-based activity where users can test easy design solutions early on in a design process, while high fidelity prototyping uses materials expected to be found in a final product, with an emphasis on testing technical issues.

Moreover, Sharp et al. (2007) suggest that a design situation can be divided into conceptual and physical design. In conceptual design, the emphasis is on how people will interact with the artefact, while physical design focuses more on detailed description of the concrete artefact, such as how many sensors are needed, and the layout of a software screen. There are no rigid borders between these two design activities.

(43)

Identify needs/

established requirements

(Re)Design

Build an interactive

version

Evaluate

Final product

Figure 3.6: A simple interaction design lifecycle model, bySharp et al. (2007, 448).

In a larger perspective, Fallman(2003) suggests that the role of design in HCI can be divided into two different activities:

• Design-oriented Research: In this activity, knowledge is the main motivation, which is usually conducted by academic research. The primary aim is to explore new knowledge outside current paradigms by studying the design artefact in use. There- fore, the resulting design artefact itself will be considered more a tool for research than a finished commercial product.

• Research-oriented Design: This activity may better illustrate the relationship be- tween consultants and costumers, where production of new artefacts is the main focus. This makes problem-solving an important part of the activity. Hence, developing commercial artefacts for everyday use is the main motivation for this activity.

3.2.3 Design methods for DMI

So far, I have outlined what design is and explained its relationship to HCI. I will now present the iterative design methodology. The word iterative is defined in the dictionary (Stevenson et al. 2005) as something relating to or involving iteration, and the word

(44)

Evaluation Synthesis

Analysis

Repeating

Conceptual design Physical design Usability testing

DMI

Figure 3.7: This figure illustrates an iterative process. The dotted lines indicate the freedom to interpolate between all stages and there are no strict rules where to begin.

iteration is defined as the repetition of a process or utterance. According to Sharp et al.

(2007), in the field of interaction design, this process is regarded as a circular process or a design lifecycle model, as seen in Figure 3.6. Iterative design is a design methodology based on cycling processes of prototyping, evaluation, analysis, and the refinement of a work in progress: A process that involves designers and users, and together they create new, alternative design.

Nevertheless, Fallman (2003) suggests that the concept of iteration may be added to the conservative account to solve the problem of using multiple methods, theories, and guidelines in a linear matter. By adding a non-linear iterative process to the conservative account the designer has the freedom to interpolate between the stages of analysis, synthesis, and evaluation as illustrated in Figure 3.7.

A review of current available methods for the design of DMI’s will now be presented, and the iterative design methodology can be seen as the main loop of the design process.

It contains different methods, which may be repeated in smaller iterations. It needs to be said that the methods presented here are grounded in the field of HCI and human factors, and I will do my best to link these methods to musical contexts.

Know the user: Early focus on professional musicians is the fundamental importance in a user-centered design process. If you don’t have users, there is no user-centered process; hence, it is here that the designer enters the stage of analysis: defining and collecting the users for the chosen project. Different empirical measurements can be carried out to understand how they are thinking in regard to the designer’s project.

Magnusson and Mendieta(2007) has carried out a survey of how musicians with different musical backgrounds and differences in age thought of performing music, either with

(45)

acoustic or digital musical instruments. Their goal was to take a qualitative approach and use the given empirical data to design their own DMI’s in a better way.

Know the task: One task would be be performing on a musical instrument in a musical situation or environment with or without fellow musicians, or performing musical tasks in evaluation of DMI design.

Know the context: Wanderley and Orio(2002) suggest a list of possible musical contexts for performing interactive computer music; they define the context asmetaphors for musical control, in which DMI’s are used, and they can all vary depending on the context of use. The list of these contexts consists of 11 different musical situations (ibid., 69).

Due to space limitations, I will only present two different musical contexts as examples.

A musical context can either be note-level control; a musician is manipulating a controller in real-time that affects basic musical features such as pitch and loudness control, sound processing control, orpost-production activities; a performing musician may control sound synthesis with live electronics. These two musical situations are good examples of possible contexts in interactive computer music.

Whenuser,task, andcontextare defined, the designer may need to define requirements and needs regarding the upcoming DMI. A requirement can either be thatthe instrument should generate sound in x seconds or the graphical interface should not contain cognitive overload. Sharp et al. (2007) emphasize that these requirements act as usability criteria or guidelines early on in the iterative process, and these requirements can be collected by empirical data from interviewing chosen users.

These guidelines or heuristics are the inner core of an iterative process, in which an expert examines the design upon Human factors criteria. This is for the designer only, and it is a cost-effective method. In DMI design, it is very important to create a good design plan if one wants to succeed. Cook (2001) suggests a set of practical principles for designing DMI’s, and how these principles can guide the designer to consider more than one factor, e.g., only technical parameters. It may also be important to take into account other considerations such as human factors and artistic needs. Three main design principle categories will be presented according to Cook. Due to space limitations, only some principles will be presented here:

• Human/Artistic principles – Some players have spare bandwidth, some do not, or

(46)

smart instruments are often not smart.

• Technological principles – MIDI = Miracle, Industry Designed, (In)adequate, or wires are not that bad (compared to wireless).

• Other principles – Everyday objects suggest amusing controllers or existing instruments suggest new controllers.

These principles do not intend to end up as design rules, but act more as guidelines throughout the design process.

Another method in use is thetask analysis, a method that lists up all possible tasks in order to create an overview of the design state (Wickens et al. 2004). This is a powerful tool for detecting design remarks, such as too many tasks are taking place to activate the interface, or other important factors. Mulder (1998) mentions that music involves simultaneous control of many parameters, and thus, a task analysis can be fruitful to create an overview of musical tasks (e.g. as a tool to develop musical tasks for a usability test).

However, the methods mentioned so far are best suited for the design of everyday objects, for which the aim is often efficiency and simplicity. Wanderley and Orio (2002) suggest a strategy for designing and evaluating DMI’s, borrowing tools from HCI and knowledge from testing existing traditional musical instruments. A well-known task in testing input devices in HCI is the so-called Fitt’s law or target acquisition: The speed is adjusted to reach the goal. In musical contexts, however, such evaluation might be trivial, since musicians might consider more temporal features such as timing and rhythm, and this may be difficult to measure in speed. Wanderley and Oriobelieve that simple musical tasks can be used as a starting point in the evaluation of DMI’s. To construct a musical evaluation test, a set of guidelines is suggested in order to achieve a good understanding of what a musical task requires. These guidelines are mostly related to the control of DMI’s in real-time, the first context example mentioned above.

• Learnability - The time it takes to learn the DMI, is an important point to consider in the development of musical tasks. When learning a traditional musical instrument, the time needed to fully understand its musical features might be up