(transcript of a talk at On Borders and Edges)

(maja introduces xinwei)

i am very happy to be back with my foam friends, and ill take an opportunity to talk a bit about the kind of work that motivates us.

maja mentioned previously the personal reasons as to why some of us are doing what we are doing. she mentioned in passing something we share very much; the desire to work in ordinary life, this ordinary reality, and to bring a little bit of the marvelous into ordinary reality. this is one way in which we can understand mixed reality, why do we want to mix, in any sense, to mix some of the magic, the marvelous in to everyday life? i think we can all use some of this magic, some sunlight in the winter, and its in that sense of desire that i would like to try to do this work.

i would like to show some of the kind of environments that maja referred to, the responsive spaces that foam has been working on, with a group that i started, called 'sponge'

(shows tgarden video)

so these are some of the playspace, the tgarden responsive space, that we have built over the last 4 years or so. they are playful spaces, the idea is that we aim to make marvelous spaces, these are certainly not ordinary spaces. we give people a chance to come into such a space and we, in this case, they were asked to choose from some fantastic costumes, which allowed them to feel a little bit more playful. evelina, cocky eek and some other designers, working with maja, were responsible for these fantastical costumes which freed the people. because people were asked to put on, basically a new body, this gave them the opportunity to put on a new body, as well as a new voice when they enter such a space, and when they enter such a space they could then play in an intimate setting, a small room, much smaller than this space, in a groups of 2, 4, maybe 5 people, and begin to improvise with these new bodies. how could you move differently? how could you sound differently? if you were in such a space, in a mixed reality? now these were small spaces, the first in 2001, later foam did these large environments, amazing environments, which were the size of a circus.

(plays txoom footage)

this is a circus, the txoom project, every one has their own pronunciation of the word. ive taken the liberty to give you an idea of the scale on which we can built these spaces, they can be small for a small number of people, or much larger.

these are different kinds of marvelous spaces in which everything comes together, alok mentioned 'mise en scene', the 'mise en scene' would be the different kinds of technologies you would use to make such a mixed reality. it includes not just digital, computer, video and sound, but of course costumes, lighting, regular theatre lights, curtains, carpentry, all the kinds of technology required to make such a space, like a theatre space.

so for he last 3 years at Georgia tech, ive been working in a laboratory form, try to work with some aspects of these kinds of 'mise en scene', what are some elements of 'mise en scene' that we can use to take ordinary, everyday spaces and make them more marvelous? and some of the aspects of these kind of spaces id like to focus are the aspects that involve the body. most importantly, your bodies, our bodies, are live. we are very much interested in live situations, where people are together, like us, here in this space, together, live. no internet connection between us, we are in principle always able to touch each other, to interfere with another person's sense of the event, that is tangible, material and involves gestures. these are the aspects i would like to explore, im exploring very much with the topological media lab. of course, if you say something is live, what does it mean to say something is live? something that foam, and all the people on this panel are interested in. for example when i knock on the table, the table responds immediately to me, its live, a live response, but if you have a computer involved the response might take 20 seconds to come back. computers are not as fast as physics, as Kristina used the word 'physics', the world responds to my actions according to its physics. if you scratch yourself, you will find that the skin, the cloth, is responding directly to you. it is in this sense that we aim to build our technology, it should respond immediately. so there are different challenges in this.

(shows picture)

going behind the scenes here, this is how a computer would see the space, there are various cameras, sensors, and the computer systems themselves, doing their work and producing video and sound. this is only a part of the story, this is not the complete 'mise en scene', it doesn't tell the story about fabrics, about tangibility.

(shows tgarden video)

this is from an early foam/sponge project, the music is composed by Joel Ryan, at steim. its very bouncy, quite aggressive, when people listen to this music, they tend to jump, move. on the player's chest is an accelerometer, measuring the movement of the body, modifying the sounds, but also being used to influence the graphics on the floor. the graphics are very simple, but what is important about the mesh is that it closes and opens very much tied to the motion of the body. the mesh movement was not coded into the computer's program, the motion came directly from the physical world, which means that the computer graphics, or animation was not done in software, it was merely reflecting the physical motion of the body. the result is that it gives the floor a sense of elasticity. this is a way we have been working alot, we want to take the affordances of real life, real flesh, real cloth, real wood, as much as possible and then augment it with just a little bit of computer work. just enough to make a difference, we start from the real world, then add something to make it marvelous.

a violin is a very good example of a responsive instrument, its wood, it computes its response immediately, its very nuanced and it has a whole performance practice around it. this gives us an avenue to explore larger spaces and back down to smaller things. we have been thinking along the lines of responsive things, in a very nuanced way. something you can maybe pick up an play right away, but in addition to this, you can pick it up again and again, and get better and better at it. think of the violin, you can make a sound with it immediately, it might be ugly, but anything you do will make a sound, but this is not the case with a computer. take a PDA, you might pick it up, it could break, give a syntax error, tell you that you cannot do this. but with a violin , or a swimming pool, you jump in and something will happen, it will make a noise, but it may take you 20 years to learn how to play or swim in a beautiful way. this is the kind of rich environment we would like to build, on with very thick aspects. you might ask why foam wants to work in a circus, or go to a village somewhere to explore a real village in a real environment, and try to enhance that in some way? because, in some way we what to have a type of dense, thick, rich experience. not complicated, but rich.

(shows video)

in this case, with my student, we are using tinyOS microcomputers on the wrist, the idea here is to think about gestures that we might be able to play with. there are very fluid gestures and the membrane between people is a synthetic rope, we do not like screens , it is not satisfactory to us. we would like to be able to take some of these images and suspend them in air.

(shows video)

like in this example, any motion you make will create a pattern in the image and this allows people to improvise, so motion becomes smoke, smoke becomes colour and ripples, the smoke could become like water, then dissipates. so we are able to take live video and treat it as if it were smoke, we are able to take light, and turn it into ripples.

so what is at stake? supposing we have these things, and they become more ubiquitous. one way to respond, is that we are facilitating new forms of performance, but not just performance for experts.

(shows video)

here are some examples of expert performers, Laetitia Sonami has been working with 'the ladies glove'. Pamela z, has been playing for many years, they have both been practicing for many years with their particular instruments.

but, i would like to suggest, that now, because these machines are becoming more common, then maybe we can make these kinds of responsive instruments for people to play with in a way that is marvelous.

(show fashion show video)

here is something i do not want to do, this is a fashion show that my students did with professional models from Los Angeles. pure spectacle. we did it because we got support from Intel and Motorola, but its not what i want to do in personal work. what i would like to do, is something you can see in the outtakes.

(shows outtake)

this is a rehearsal, for the fashion show, you can see what is happening and it explains why i like to make such instruments. this is the first time these 2 models have put on these machines and she moved her wrist in a way she naturally would. the sensors are sensitive to tilt, so the music is in some way tuned to the wrist movement. so its predisposed to do something interesting with this kind of motion. so here you can see her discover this, it allows her to play with a few movements, she laughs. this was the first time they had laughed all day.

(shows tgarden video)

this is another example of again, a very beautiful moment of epiphany, where this person notices the rooms sound changes, she ask 'how did i do this?' she plays, she begins to improvise, to find a repertoire of gestures, thru which she can begin to play the room as a whole. this is a more nuanced way of discovering how one can play a more complex field with a continuum of improvisations.

so what we are beginning to do in the laboratory, is take these lines of research, for example in the domain of video we have developed different kinds of video substances, playing with light treating video as liquid.

(show bungee video)

we have just finished doing a room with 5 bungee cords across the room, in this case we are looking at very playful, yet very large and robust instrument made from bungee cords. you can move into it and play them as an instrument. the idea is to make them talk, the instruments is sensitive, you can stretch it and make it talk. this sound is not from the ropes twanging, but is totally synthesised from the machine, making it revoiced taking the physics from the real world. why should we go to all this trouble? that revoicing allows us to give it a voice that is very different, like a magic voice, a roar, something that can be used to talk.

in about a year or so, id like to be able to pull these elements of the 'mise en scene' back together, and start making another series of magic environments. in small, medium and large.

as i said earlier, we are not interested in screens, we are looking at ways we can make things appear on ordinary cloth or fabrics, of course we cannot do it right now, its too fast, bu we begin to work with cloth and material because ultimately we want to bring it to work in a live situation on bodies, in a playful way, to work with gestures. so i that sense im thinking of these explorations might lead to a kinesthetic play which also has aesthetic, social and ethical consequences.