Golan Levin keynote at Cybersonica

To understand "visual music" (performing):
- a formula which combines both sound and image into a holistic union,
- or a striclty visual, temporal form, analogous to but separate from music.

Levin's work belongs mainly to the first "formula" but he often uses both.

Audiovisual performance systems

In his 1927 book Color-Music: the Art of Light, Adrien Bernard Klein wrote "It is an odd fact that almost everyone who develops a color-organ is under the misapprehension that he, or she, is the first mortal to attempt to do so."

Researchers, artists, scientists regularly claim they were the first to invent the Color Organ. Untrue as the story shows:

The earliest known device for performing visual music was built in 1734 by Louis-Bertrand Castel.The Ocular Harpsichord coupled the action of a harpsichord to the movement of transparent tapes. In 1844 D. D. Jameson's "color organ" filtered light through liquids of various colors and reflected it off metal plates onto a wall. Frederic Kastner's 1869 Pyrophone opened flaming gas jets into crystal tubes to create both sound and image. Levin went on giving many examples (see a list of them in the extract of his thesis proposal) and highlighted two instruments : Thomas Wilfred's Clavilux (1920's, pictured here), and Oskar Fischinger's Lumigraph (1948). The Clavilux filtered light through several stages of multicolored glass disks, the instrument produced only images, no sound at all, Wilfred was commited to that. The Lumigraph interupted colored beams of light with a flexible fabric surface.

Introducing the computer. The computer:
- transcends the limitations of physics, mechanics, optics,
- overcomes the control/generality tradeoff of physical systems,
- takes advantage of the unique affordance of computation: iteration and simulation; conditional testing and constraints (the difference between a calculator and a computer is that the computer has the ability to ask "what if"), data storage.

Visual interfaces to sound on the computer - Common paradigms:
- score displays,
- control-panel displays,
- flow-based networks,
- reactive "widgets" (spriters, romplers),
- navigable terrains (virtual land-mines),
- cellular automata (sonified life).

1. Score-based interfaces. They are extremely efficient but:
- they generally have a diagrammatic space instead of a painterly one,
- they rely on a coded visual language of mostly arbitrary graphical conventions. You need the key!

2. Control panel interfaces. They lack the gratifying tactility and the two-handedness of the 70's synthesis.

3. Widget interfaces (object metaphors) have material properties, physical properties (like "how high") and contextual properties(such as "how near to an object").

Reactive widgets: when taken individually have limited malleability, are quickly exhaustible (canned media), the malleability is achieved through sheer number (e.g. 10,000 widgets), future in granular resynthesis and wavelets.

4. Navigable space interfaces (terrain, architecture and map metaphors) where you navigate in a 3D world. E.g. Lab AU, and Playstation's Vib-ribbon (picture above).

5. Automata interfaces (rules systems). e.g. the work of Tom Betts.

6. Flowchart interfaces (networked metaphors).

Challenges and pitfalls
- Randomness: how can someone know that the system really responds?
- the taste of mathematical systems, instead of focussing on whether it is meaningful,
- cartesian and diagrammatic mappings,
- modal interactions instead of giving people feedback on where they are,
- ROM-based solutions.

Some exciting new directions:
- custom physical sensors,
- physical sound actuation,
- alternative imaging (physical pixels, lasers),
- new contexts (phone, PDA, putdoor, furniture),
- better user models (learning systems),
- software cannibalisation (game mods),
- etc.

The second part of Levin's talk was a presentation of his own work.