Location: CHB, Berlin (D), The Loop, Chicago (USA)
Client: Budapest Music Center, Balassi Institute
Format: Sound Reactive Visuals
Year: 2017

We have been asked to make real-time visuals for a series of electronic sound performance that is giving tribute to the legacy of Laszlo Moholy-Nagy. There is an ongoing retrospective exhibition this year that is running around the United States, including New York, Chicago and Los Angeles. This event is one of the first comprehensive collection from the works of the Hungarian born artist, so is a great and very inspiring thing. Moholy-Nagy was experimenting with a diverse set of media, from drawing, photo collages to paintings, scultpures, photograms and moving images. Observing this wide set of activity led to an interesting collection of his works.

Inspiration, sketching
We started experimenting a lot with shapes, colors and compositions that can be found within his works, then continued combining those, based on the most typical colors that can be found in our subjectively pre-selected picture set. For the color extraction, we were already reusing code that we developed for one of our other previous installations. Then we introduced movements, methods, such as noise, l-systems and tree growing into the creative flow. During the process of sketching and doodling, the basic idea was about to form: having a three dimensional space, where each sound from each musician has an affect on the space or adds some visual element to the scene.

The Visual Scene

Each instrument has different visual characteristics and, by the advancing of the sounds, a three dimensional sculpture is being shaped together. Apart from adding elements to the sculpture, the wind instruments got an extra feature: when they are played, the camera of the whole scene is moved around (based on its intensity), so the composition is set in motion.

Adding / removing elements can be smoothly adjusted from a custom built user interface, where the performer can select the desired visual shapes along the color palettes assembled for the set. Since the musical material was full improvisation, we had to define a parameter space, where the system can run fully automated if needed, everything is controlled by the sounds and the built in behaviours, but we also had to leave some space for intervention for the visual performer: there is some possibility to change colors and scene intensity, freezing the movements, controlling the 3d camera through the interface.

Sounds & Analysis
The music consists of three individual tracks played live by three musicians: wind instruments (László Gőz), dulcimer (Miklós Lukács) and electronics (György Kurtág Jr). A multichannel external sound card handles all the incoming audio channels, with custom software running spectral analysis on each input. Spectrum, mfcc, onset detection and averaged gain are the main sonic characteristics being used for direct visual manipulation. The tone of the sounds also has a special importance. We pre-trained three different tones for each instrument during the rehearsals, using supervised learning algorithms. At the end, the system is trying to predict which tone the musician is playing at the moment, and it selects the corresponding visual set of shapes, based on the predicted class. Noisy, cracking sounds produce different visuals compared to clean, harmonic ones in the scene.