Dances with Machines

Artist David Rokeby builds machines that watch us, make music with us, speak to us and free-associate on our behalf.

By Rebecca Zacks

The movements of the lanky man on the videotape mesh perfectly with the undulating rhythms and cascading tones that accompany his dance. As the music swells, his gestures grow pronounced and emphatic; as the sound dwindles to the pulse of a synthesized bass or the flutter of an electronic clarinet, his motions diminish to the twitch of a hand or the slow sweep of an arm. The choreographer, it seems, must have worked closely with the dancer and the composer to make such a seamless piece. The reality is more complex: This dancer is, in fact, also choreographer and composer, choosing his moves on the fly while simultaneously making the music to match in an intimate collaboration with a video camera and a homemade computer system.

Sprawled shoeless on the living room floor in his Toronto home, 38-year-old David Rokeby watches the 28-year-old version of himself on a small TV set. Though his worn jeans, wire-rimmed glasses and only slightly scruffy hair make him look like the math professor his parents wanted him to be, Rokeby has instead become an internationally known interactive artisthis multimedia installations invite gallery goers and exhibition attendees to become active participants in the artistic process.

In language that shifts easily between the professorial and the poetic, Rokeby explains both the technology and the artistic intentions behind his work. In many ways, his career sounds like that of a researcher. Rokeby thinks of each of his installations as an experiment; observing the hundreds of thousands of people who have participated with his pieces has given him an invaluable opportunity to learn about humans, machines and the very complicated relationships between them.

Through these artistic explorations, Rokeby has begun to understand how peoples interactions with computers change as technogadgetry becomes more and more common. And he has uncovered some ways that machines can subtly distort human perceptions. After years of investigating such ideas, Rokeby worries that our increasing interaction with the Internet and intelligent technologies might cause us to devalue some of the attributes that make us human. So while others work toward a transparent interface between person and machine, Rokeby aims to expose the quirks, foibles and rough edges of that relationship. Because Ive programmed a lot, because Ive built computers, I know what its like to write a program and then watch people deal with it, and watch how my decisions change peoples experiences, says Rokeby. For me, its important that I somehow articulate the importance of that act.

Rokeby played the videotape of his dance on a sunny January afternoon to demonstrate his best-known project: Very Nervous System. The name is an umbrella term for an ongoing series of installationsthe projects technological roots date back to some fiddling around with light sensors and a synthesizer that Rokeby did in the early 1980s. Over the years, Rokeby has used the technology behind Very Nervous System not only in his artistic endeavors, but also to support them; reduced to its initials, VNS is an image-processing device he builds and sells to performers, composers, researchers and other artists.

What VNS does, essentially, is translate the motion captured in a live video image into a digital signal. That signal can, via a Macintosh computer, drive electronic equipment such as synthesizers, video players and lightsall in real time. In a typical Very Nervous System installation, a body moving in the cameras field of vision becomes an integral part of the work, triggering and modulating sounds or other effects.

Rokeby develops software and hardware for projects such as Very Nervous System with little outside help, and no formal technical training. As a teenager growing up in southern Ontario in the 1970s, he taught himself programming in order to indulge a fascination with electronic music and computer graphics. At 19, with an offer on the table for a lucrative but uninspiring job in data processing, Rokeby instead embarked on a five-year planhe would focus on the things that interested him and avoid those that smacked of career. If it didnt work out, he figured, he could always go back to school and get a computer science degree.

After a stint at the Ontario College of Art, almost five years to the day after he hatched his plan, Rokeby received an invitation to show his work at the Venice Biennale, arguably the worlds premier art show. The list of his artistic honors has grown steadily since.

Rokeby isnt the only artist exploring the gray area between the body, the mind and the computer (see sidebar Virtual Plants), but he began doing this kind of interactive work long before most of the other artists currently on the scene, says Finnish media scholar Erkki Huhtamo, a visiting professor in the department of design at the University of California, Los Angeles. Whats more, Huhtamo says, Rokeby is one of few to have constructed his own technological tools. Hes wonderfully capable of doing that, says Huhtamo, but on the other side he has applied those tools for various artworksa career that combines these two sides meaningfully and interestingly is rather rare.

Virtuosity in both technology and art has given Rokeby a unique perspective on the evolving ways people relate to machines. Audiences of the early installations, shown at a time before many people used much computing power outside arcade video games, were more open to the raw experience, Rokeby recalls. They focused on the physical, and felt as though they were bumping into invisible objects that made noise. As time went on, though, people became more interested in the gee-whiz technical aspects of the installations, and in tying the experience into a rapidly expanding computer culture.

But even as PCs became ubiquitous and virtual reality and interactive media attained buzzword status, there was an ecstatic quality to how people reacted to the piece. Very Nervous System is very exciting, Rokeby says. To show it is very satisfying on a certain level because people love it, and come up to you and tell you that its brilliant and fabulous and it has changed their way of looking at something. But Rokeby began to worry that his work was too exciting, that people were so blown away by the real-time physical experience that they werent stopping to ask the questions he had hoped they would: What happened between me and the machine? What does that mean for my relationship with my word processor?

To Rokeby, the answers to these questions have implications far beyond artistic concerns. He noticed that people tended to credit Very Nervous System with more than its fair share of responsibility for certain effects; they might, for example, synchronize their movements unconsciously to a particular preprogrammed rhythm in the mix but believe it was the technology that adjusted to them.

Given peoples general sense that machines are very smart, Rokeby says, they have a strong tendency to attribute the smarts to the machine, even if its their own smarts reflected back. Rokeby believes that as interactive technologies, particularly the Internet, begin to play a central role in communications, commerce and civic activities, the sense of where the control is and where the intelligence is becomes more significant, more politically and socially important.

To get away from the distracting excitement of Very Nervous System, in the 1990s Rokeby began working on pieces that were less physical and lacked the frenzied feedback between audience and computer. In 1995, he started showing a piece that turned his early installations inside out, giving audiences the chance to watch the computers image-processing operation as it happened and to see what the machine had been seeing for all those years. One of the things that was always weird about Very Nervous System, Rokeby says, is that it is a surveillance system, but no one ever felt threatened by itpeople didnt feel like they were being watched.

So in Watch, Rokeby created an overtly voyeuristic experience. Video projectors shine two images side-by-side, each a processed version of a surveillance cameras view of a nearby public space. In Very Nervous System, the computer extracts motion from a video signal by comparing one frame with the last and determining which pixels have changed, but that whole procedure is invisible to the viewer. The image-processing techniques used in Watch are a dissection of VNSs internal workings. On one side only the things that are moving show up, white ghosts gliding through a black void; the other side shows only whats still, a seemingly normal but frozen black-and-white video image.

To these images, Rokeby adds a soundtrack: The occasional noise of a camera shutter or electronic beeping interrupts soft hypnotic sounds of breathing, a heartbeat and a ticking clock. Its a reminder, Rokeby says, that there might be something wrong with spying on people in this way.

Watch also serves as a reminder of how different the world can look when seen through varying technological lenses. In the early days of developing the piece, Rokeby aimed the camera out his studio window at a busy intersection. The two different video filtersone catching motion, the other stasisbecame socioeconomic filters: In one image, members of a vibrant crowd moved swiftly about their business, in the other, panhandlers appeared to be sitting quietly alone on a deserted sidewalk.

Rokeby again draws from art a lesson about the impact of technology on our perceptions. The image-filtering techniques he employs in Watch are very similar to those used to compress video for storage or transmission. (Programmers save digital space by recording or sending only the changing pixels in successive frames of a moving image.) The more we use such techniques in daily life, he says, the more we wear inherently biased lenses. Rokeby says he is particularly concerned by the large number of design decisions being made by programmers in startup companies working on intense deadlines, with very little experience of philosophy and politics.

Though the insights Rokeby has gained through his art may put him in a better position to make such programming decisions, he has no desire to tie himself to his own startup company. He builds and sells only a few VNS units a year, though many more people would like to get their hands on one, according to Todd Winkler, a music professor at Brown University. In the computer music world, his system is very well known and people talk about it, want to learn about it all the time, says Winkler, who has used his own VNS setup for more than three years in installations, performances and demonstrations. Still, Winkler understands Rokebys decision to focus primarily on art rather than commerce. Getting into the business of making little metal boxes that everybody in the world wants could really consume you completely, Winkler says.

On the contrary, what is consuming Rokeby these days is his latest project, The Giver of Names. Its a concept that came to the artist almost instantaneously on the day after his birthday in 1990. The idea was there would be a computer and objects and you could present the objects to the computer and it would talk about them, he recounts. Realizing this seemingly straightforward notion, however, has taken the better part of the decade.

Part of the motivation behind The Giver of Names was what Rokeby, perhaps presciently, saw as a shift in the interplay between people and technology. As he wrote in an e-mail quoted in the catalogue for the 1998 premiere of The Giver of Names, in the 1980s it was the body that was most challenged by the computer....In the 90s it seems to be the notions of intelligence, and consciousness.

Rokeby worries that as we grow accustomed to such phenomena as intelligent agents on the Internet and computerized phone systems, we may devalue certain human attributes. To talk to that computerized receptionist, for example, we often have to exaggerate and mechanize our speechthe change in enunciation is a subtle dumbing-down process. So rather than trying to make The Giver of Names a flawless facsimile of human thought, Rokeby wanted to leave it rough, exposing the quirky textures of a strictly mechanical intelligence rather than using clever programming to paper them over.

In action, The Giver of Names is quirky indeed. The installation space is spare: A video camera aims at a black pedestal around which a variety of objects are strewn. Off to one side is a Macintosh G3. Visitors can select objects from the pile, or items theyve brought with them, and arrange them on the pedestal; the computer captures an image and processes it, identifying colors, outlines and shapes. The system then begins a mechanical version of free-association, pulling up words that are somehow connected to the details culled from the image. The Giver of Names state of mind in this process is a relational database of 100,000 objects, words and ideas.

An object on the pedestal, Rokeby explains, is like a pebble dropped in a pond of memory, and the associations are like ripples moving away from the initial object and exciting or stimulating different parts of the memory. The words most stimulated in this process become the palette from which the computer chooses in forming sentences that appear on the computer screen. At the same time, male and female voices fill the installation space as they utter the words.

Presented with a soda bottle and an apple, for example, the system might pick up on the red of the apple and the shape of the bottlethese would probably stimulate the word wine, among others, says Rokeby. As for the sentence, it could be anything from The wine spilled to something completely off the wall like Red aliens from inner cities flopped sumptuously on the wine-stained sofa.

Early on, The Giver of Names tended to talk about war. The systems fixation on generals and grenades prompted Rokeby to consider the fact that many of the databases he used were developed for military-sponsored artificial intelligence and natural-language processing research. Its kind of interesting, he says, that the tools used to train artificial intelligences about language will inevitably have a strong defense bias, because the best resources right now were funded by the Defense Department.

Rokeby is the first to admit that such specific lessons arent likely to be obvious in his artworks, that most people wont listen to The Giver of Names talking about a piece of fruit and say, Gee, I should really think about the effects of military funding on the future of artificial intelligence. But by seeing ourselves in collusion with and in contrast to the mechanical perceiving, thinking and speaking systems that Rokeby builds, we can all begin to think about, as he puts it, how much of what we do is basically mechanical and how much of what we do does imply something richer and more complicated. And Rokeby takes great satisfaction in the unique intensity with which interactive art allows him to communicate such ideas. Not everyone gets the point of each installation, he says, but when they get it, boy do they get it.

Rebecca Zacks is an associate editor at Technology Review.

-- scott (hma5_5@hotmail.com), February 24, 2000