If I had a tendency, at the beginning of this PhD study to think of computer/performer interaction as merely a technical problem, the research involved in this study has made me reconsider the question behind the project itself. Through the projects that I have completed, as well as the ones that I am currently working on, I have focused on investigating the experience of interaction between the different agents involved in the production of musical content. In the case of etherSound, it was the interaction between the user (and/or listener) and the sound; in the project Negotiating the Musical Work, it is between the performer and the composer; and in the composition `Repetition Repeats all other Repetitions' (for which the collaborative project Negotiating the Musical Work is a pre-study), it is between the performer, the score, the computer and the sounds produced by means of the computer. I now believe that it is only through a thorough understanding of the very nature of interaction in the context of musical production, that an artistically relevant and perceivable mapping mechanism between input and output in an interactive system can be achieved. The updated research question now reads: Can an interactive system that uses the sound as its object of interaction provide the necessary premises for an integration of digital and analog sound sources on the level of both sound (as it is perceived) and performance (as it is experienced), and, furthermore, how can significant features of human/human and human/sound interaction in the context of musical production inform such a system?
At first glance, the following selection of artistic works and projects, belonging to the larger investigation of sound and interactivity, may seem disparate or heterogeneous. However, if the focus is placed not so much on the content of each of the projects, but rather on the larger frame within which they exist, i.e., on the premises and outsets, then the projects form a more homogeneous collection of studies. The intention is to create a platform for the investigation of the limits for interaction and sound, and the result is intended to be an artifact of artistic output. Apart from the three projects presented below, a number of improvisations and smaller scale works have been, and will be, produced within my PhD project.
etherSound was commissioned by curator Miya Yoshida for her project The Invisible Landscapes and was premiered in August 2003 at Malmö Art Museum in the city of Malmö, Sweden. The curatorial concept for The Invisible Landscapes project was the use of cellular phones as a means of experiencing and creating artistic expressions. The principle idea behind etherSound came to be an attempt at developing an instrument that can be played by anyone who knows how to send an SMS (Short Messages Service) from a cellular phone. In the version displayed at The Invisible Landscapes, all messages sent to a specified phone number were received by an Internet server, parsed for its content as well as the phone number it was sent from and the date and time it was received. This information was written into a database which was queried at regular intervals by a computer running a control as well as a text analysis application (written in Java [J2SE 1.4.2, 2004,J2EE 1.4.1, 2004]), with sound-synthesis software (Max/MSP [Zicarelli, 2001] running a Csound orchestra [Boulanger, 2000]). For every new message, the data was downloaded, processed and analyzed by the control program, and turned into control signals, which were then sent to the sound-synthesis engine. Every message generated one sonic object that would last for up to two minutes. The response was very direct with a clear causality between the input and the output of the system - a received SMS would result in an immediate and perceivable change in the sound (see http://www.henrikfrisk.com/index.jsp?metaId=music&id=music&about=1&field=name&query=etherSoundfor some audio examples).
There are two states in which etherSound may operate. One is as a stand-alone, interactive sound installation and the other as a vehicle for improvisation. In the latter, one or several performers improvise along with the sounds of the installation while the audience contribute actively to the performance by sending text messages. etherSound is an investigation of some of the aspects of interaction between the listener, the sounds created and the performing musicians, and also of the formal and temporal distribution of the music that this interaction results in [Frisk and Yoshida, 2005,Yoshida, 2006,Frisk, 2005].
etherSound also has significance in relation to the discussion of the man/machine relationship, described above. Perhaps the mobile phone is one of the machines of recent years that humans have most successfully and naturally integrated in their lives. Furthermore, although the traditional roles of the agents involved in the production of the music in etherSound were shifted or distorted, there is no doubt that the programming, i.e., the actual code that constitutes the synthesis and the control program, is the score - if a score exists at all. In any event, these programs contain the only apparent work identifying instructions, to use the language of Stephen Davies [Davies, 2001]. When technology is put in the center in this way, the technique (in the broad sense of the word) that is required of the user/listener is of a different kind. Traditionally there is an intimate connection between social class, level of education and cultural interests [DiMaggio and Useem, 1978,Bourdieu, 1979] which affects cultural consumption. Despite the fact that the connection between social class and mobile phones is likely to be of a different nature than that between social class and arts consumption, interactivity and collaborative art in themselves may help to counteract the exclusiveness of contemporary art and music. Perhaps it can contribute to creating conditions for classless and unprejudiced participation in the arts without compromising the content and the expression. Roy Ascott, in addressing the issue of `content' in art involving computers and telecommunications writes:
In telematic art, meaning is not something created by the artist, distributed through the network, and received by the observer. Meaning is the product of interaction between the observer and the system, the content of which is in a state of flux, of endless change and transformation [Ascott, 1990].Following this line of thought, it may be concluded that the need for a thorough insight into the history of art or electronic music is no longer a prerequisite for understanding a collaborative, interactive work of art. This limits the advantages of the educated listener and makes room for new interpretations of the term `understanding' in the arts.
I am undertaking this project, which is not yet completed, in collaboration with guitarist Stefan Östersjö. It consists of three distinct parts:
Primarily, we discuss the musical work prior to its ultimate notation and prior to its performance; we discuss the musical work within the context of Western 'art music' tradition, in which musical notation has an ontologically crucial function. The study deals exclusively with music for solo instrument and live electronics. Our purpose is to acquire a deeper understanding of the underlying processes in the communication between the composer and the performer as well as their respective roles. Through an improved understanding of the musical interaction between the two parties involved in the creation of the work, we also hope to better understand the necessary conditions for a successful interaction between the performer and the electronics. We have developed a hybrid method of investigation, which involves musical semiology, qualitative method involving hermeneutics, and verbatim transcriptions of the video documentation.
Musical semiology has been constructed with the intention of providing tools for analytical understanding of the musical work in its entirety - not only in terms of analyzing formal structures or details in the construction of the work, but also examining its socio-cultural context. Attempting to move to a more basic level of organization than that of musical notation may help to further clarify the issue in relation to a wider sphere of knowledge. For the analysis of the composer/performer interaction we used the tripartite model suggested by Nattiez and Molino:
...recognizing, elaborating, and articulating the three relatively autonomous levels (poietic, neutral and esthesic) facilitates knowledge of all processes unleashed by the musical work, from the moment of the work's conception, passing through its 'writing down', to its performance. [Nattiez, 1990]In short, according to Nattiez the poietic phase is the complex series of activities that are part of the construction of a musical work, the esthesic phase is the reconstruction of the message and the neutral level is the trace left by the poietic (or the esthesic) processes.
The conclusions we draw from the first stage of the study, which has many implications for both the second and the third phase, is that both the creative and interpretative activities oscillate between poietic and esthesic processes. Taken in this context, in etherSound for example, one may dispute whether one can talk about a single work as an ontological unit or even a single originator in the role of 'composer.' We also found striking examples of creative misunderstandings between the agents involved in the collaboration which led us to the perhaps somewhat exaggerated conclusion that, in communication noise is not a problem. As has already been mentioned, we are used to thinking of a computer-based interactive system as a cybernetic system in which information is transmitted from point A to point B, and where great care is taken to avoid noise in the transmission. In our joint project we will attempt to avoid the kind of binary oppositions that require a clean control-signal path (such as the pressing of a pedal) in the design of the interactive system. Obviously this will also affect the way the instrumental part is written.
timbreMap is a software development project that attempts to allow for direct interaction with sound itself rather than with an abstract classification of sound. This is the part of my doctoral project that I was initially inclined to look at as the central goal. I have already referred to my growing frustration with the way in which the tools available to me for letting a computer interact with a performer in real time did not satisfy my needs as a composer. One example of a widely-used tool in electroacoustic music with live instruments is something that is referred to as ``pitch-tracking''. What this process attempts to achieve is the transformation of a (monophonic) audio signal into discrete pitches. Aside from the fact that this is a difficult task, the information gleaned by this system is only useful if the pitch class representation is a meaningful and substantial parameter in the intended totality of the musical output. In much of my music it is not.5With timbreMap I have attempted to construct a system that uses self-organizing feature maps and chained neural-networks to track the relative change of timbre in an audio stream and make this information available for interaction. As was mentioned above, 'connectionism' or neuron-like computing is in itself a move away from the binary representation of numbers, approaching what may be called an attempt at modeling continuous processes. It is a special purpose machine that is closely linked to the artistic enterprise that created the need for it. In that sense, although in a more abstract way than the programming of etherSound, its code is part of the notation of the possible pieces it may give rise to.
In the time that has passed since I started my doctoral project I have come to realize that, although timbreMap is a big part of the totality of my project, both in terms of time invested and its significance to the whole, its most intriguing aspect may be the mapping of the output of the system and the musical stimuli to which it gives rise. The mapping must be related to large-scale empirical studies, such as those mentioned in this article, but also has to be tested in the specific case-studies. From etherSound I learned that successful mapping involves a certain amount of pedagogy - knowledge creates anticipation and expectation. The studies performed within the project Negotiating the Musical Work opened up the idea of the `creative misunderstanding' and a semiological analysis of the communication within an interactive system.