Select category or view all .

Interactive Systems - current status

Henrik Frisk
DMus student
Malm Academy of Music - Lund University

14th September 2004

1 Outline

I will use this presentation to evaluate the current status of my project (so I hope to be among friends...). In my doing this I hope to touch upon some problems and challenges that I think we all have in common. However, I will also talk about some less general subjects in order to give you the full picture of my project.

In my project I am exploring different modes of musical interaction between musicians and computers (analog systems and digital systems) - in other words, to create a framework for a seamless integration of electronic sounds and acoustic sounds.

The reason is that I am simply not satisfied with the tools available. As I will get in to later in my presentation, the way that we learn to deal with musical means of expression is always geared towards a certain aesthetic preference, or tradition if you wish. As soon as you start breaking the boundaries between genres, styles or modes of expression, the models offered to you are no longer relevant.

At this point I am spending much time at reverse engineering, so to speak, a theoretical framework for the conclusions that I have come to more or less by intuition. Having said that, I may add that this is the difference between being an artist and doing artistic research.

Apart from the music that will be part of my project, an important part of it are the actual tools I design (software) and the knowledge produced is everything that these two aspects boil down to.

2 Computer Interaction

This, in many ways extremely large, interdisciplinary and multidimensional field I am attempting to limit to deal with two specific cases (I say attempt becasue these two contexts can and will be combined, varied and distorted.):

  • Improvising musician interacting with a computer.
    The music is not known, so the interface will have to adapt. Ultimately, the interface should have a memory so that the system can learn to predict the outcome of a given musical situation.
  • Musician playing a precomposed score interacting with a computer.
    The music is known. In this case the interaction is a matter of aligning the computer part with the instrumental part.

In either case the computer acts as an instrument or as an extension, or resonance tied to the acoustic instruments and generates sonic events but it is also a listener and a musician. Special care has to be taken to evaluate the implications of these roles. Should the computer be a passive or active listener? Should it be able to take initiatives that can alter the outcome of the music?

As a composer, how do I view the computer?

  • As one additional musician
  • As a way to simply alter the sounds produced
  • ...

As an improvisor should I

  • look at the computer as a co-musician
  • or as a predictible sound altering device

Ultimately, I would like to evaluate and try all different possibilities, but most importantly, I am convinced that these aspects are crucial to the outcome.

3 The challenge

All sounds produced by a musician playing an acoustical musical instrument are subject to constant change inflicted by a number of factors. Some of these are controlled by the musician, directly or indirectly, and some are unforeseen and tied to the time and space of the performance of the sound. Together they create the dynamic that we have learned to recognize as a musical sound. Whitout them we are left with a static physical sensation. (I would go so far as to say that it is the nonpredictability of a sound that makes it a musical sound.)

This constitutes an analog system (res extensa). There is a continous change between sound and no sound and only the changeing can be described.

The underlying logic behind a computer is rather the opposite; on all levels (whether it is perceptual or not) it is binary, either on or off and even if we can model continuous signals digitally the fundamental difference will always be there (res cogitans). The challenge is to find ways to integrate the two systems in a flexible larger system that can be perceived, audibly and analytically, as a whole.

My belief is that it is the mode of interaction between the two that is the key. By creating a common ground or an interface where they can co-exist on equal terms they become part of the same logical type on the next higher level. I will argue that the actual method chosen for the interaction is not merely technical detail, but something that has to be paid a great deal of attention in order to achieve the merging of the parts.

But in order to be successful at merging the parts, I need to define what needs to be merged but this I will come back to later.

This far I have presented you the background and the problem I am addressing in my project. I had learned through my own experiences with working with the computer as a musical instrument, alone and in combination with acoustical instruments that the techniques for integrating the computer in a musical context commonly used by the community was not ideal for me. Within the framework of my project, a lot of the work I have done so far has been to seek to create a theoretical foundation in order to describe what I have arrived at intuitively - necessary in order to design a system more fitted to my needs.

4 Perspicio - (se igenom)

Before I move on I would like to show an example. The idea behind this piece was to create the audible variation of looking at an object through water - as the surface moves the edges of the object becomes distorted. I use this piece in my project in order to illustrate the problems that will rise when

I use a standard way of linking the computer to the music performed. The computer is extracting the pitch from one of the saxophones and can follow the score. This way it is easy to synchronize the computer part with the instrument part. To a large extent, this technique and the tools needed to implement it has been developed by IRCAM in Paris. It is based on the paradigm Suivi de la partition established by its director Pierre Boulez.

If we look at this model schematically it becomes obvious that the interaction moves in a non-connected feedback loop. The interaction is based on the representation of the music and not the sound itself. In other words the interaction is not connected to the intended output of the system and little has been done to actually integrate the continuous entity and the non-continous entity.

If we look at the flow of information hierarchically it may look like this pane. The jump between the levels in the computers attempt to synchronize itself to the score becomes a problem. This jump results in what Gregory Bateson calls an error in logical typing[1].

5 Logical types

Looking at the problem from the perspective of logical typing seems to me to be a helpful tool. Gregory Bateson adapted Bertrand Russels theory of logical typing to describe interaction between biological systems, specifically he used it to develop his double bind theory. He talks about the contrast between learning about the particular and learning about the general and that the larger case is of a totally different nature from that of the particular[1](p. 115).

This is in principle my situation here: the note event (particular information about one event) was generated from a sound (general information about a class of events).

6 Suggested Solution

If, instead of transforming the information back and forth from the representation of the music (the score), the sound becomes the interface, I beleive the first step towards a more transparent communication is taken. There is a constant two-way communication between the sum of the sound produced by the two enteties.

This, being an attempt to describe a very complicated interaction in a very schematic way, is obviously not the whole truth. But it is the beginning of my understanding of the processes at action, and as such the beginning of the continuation.

7 Implications

However, this suggested solution implies that it really is the sound itself that is the object which is not at all necessarily the case. Dalhaus means that it may very well be the score that is the object, but this we will come back to later.

Another implication is that integration of electronic sounds and acoustic sounds is really desirable. If used intelligently, the contrast between digital and analog sounds may be a powerful expressive effect. However, if the merging of the parts cannot take place, it remains an effect and as such, rather limited. Also, the fact that I learn to integrate the two parts, doesn’t mean that I cannot bring out the contrast.

Implied is also that the interface in some ways, however limited, models the way a human ear perceive of sounds - sounds, not music. The techniques that I am studying to accomplish this is psycho-acoustics, cognition, machine learning, ANN, Artificial Intelligence and others. In fact, this is a very active area for scientific research. There are many applications where it would be desirable for a computer to be able to distinguish timbres.

8 Alternate Aproach

If I can successfully integrate the musician and the machine to the level of them becoming part of the same system, one can alternatively look at the two as parts of a structure already present within the musician. The result of this model (which I feel is desirable) is that the computer then no longer is a musician, but part of a decision system that is ruled by the consciousness of the musician.

9 Theory Map

This is a map that displays different theoretical apraoches to the problem. From top down is the mathematical/logical way. Right to left is the phnomenological way. From bottom up is the sociological way and from left to right I don’t know what to call.

10 Knowledge

This map is a way to try to visulize all the different subparts of my project. Play improvisation...

What is free? Free from what?

11 Drive

I will end this presentation with playing back a recording of one of the latest pieces that I have completed. As you can see from the score, the piece is contained on one page and offers a lot of freedom to the performer. In this piece the structures are fixed in the computer part, which, in this case, as opposed to Perspecia, is very predictable.


[1]   Gregory Bateson. Mind and Nature. Hampton Press, 1979.