Javascript Sound Experiments - Filtering [ProcessingJS + WebAudio]
I am working on a sound installation project where we are building our custom devices to measure distance (using wireless radiofrequency technology) & create sounds (using DAC integrated circuits with SD card storage, connected to a custom built analogue filter modul) through simple electronic devices. The building of the hardware with Marton Andras Juhasz is really exciting and inspirative in itself, not to mention the conceptual framework of the whole project. More hardware related details are coming soon, the current state is indicated on the video below:
The scope of this entry is about to represent filtered audio online, natively in the browser. I have to build an online representation of the system where the analog hardware modules can be seen and tried out within a simplified visualization (that is mainly to indicate the distances between the nodes in the system). Apart from the visualization, sound is playing a key role in the online representation, so I started playing with the Web Audio API , where a simple sound playback can be extended with filtering, effecting using audio graphs. The state of web audio might seem a bit confusing at the moment (support, browsers compatibility etc). To minimize confusion, you can read more about the Web Audio API here.One is for sure: the context of web audio is controlled via JavaScript, so it can fit well into any project that is using high level JavaScript drawing libraries, such as ProcessingJS.
The basic structure that we do here is
1. Load & playback soundfiles
2. Display information, create visual interface
3. Use control data from the interface and drive the parameters of the sounds
Load & Playback soundfiles
I found a really interesting collection of web audio resources here that are dealing with asynchronous loading of audio, preparing their content for further processing by storing them in buffers. Sound loading and storing in buffers is all handled with the attached Javascript files. If you need to switch sounds later, you can do it asynchronuously by calling the loadBufferForSource(source to load, filename) anytime anywhere on your page.
Display Information, create visual interface
Connecting your processingjs sketch to the website can be done by following this excellent tutorial. You should make functions in your Javascript that are “listening” to processingjs events. You can call any function that can accept any parameters. One important thing is that you have to actually connect the two systems by instantiating a processingjs object as the following snippet shows:
var bound = false;
function bindJavascript()
{
var pjs = Processing.getInstanceById('c');
if(pjs!=null)
{
pjs.bindJavascript(this);
bound = true;
}
if(!bound) setTimeout(bindJavascript, 250);
}
bindJavascript();
We have some work to do in the processingjs side, also:
// create javascript interface for communication with functions that are "outside"
// of processingjs
interface JavaScript
{
void getPointerValues(float x, float y);
void getFilterValues(float filter1, float filter2);
void getGainValues(float gain1, float gain2);
}
// make the connection (passing as the parameter) with javascript
void bindJavascript(JavaScript js)
{
javascript = js;
}
// instantiate javascript interface
JavaScript javascript;
Use control data from the interface and drive the parameters of the sounds
After connecting the html page with our sketch it is possible to use any action in the sketch to control our previously loaded sounds. This way we separate the whole audio engine (handling, storing, etc) from the visual interface (the processingjs canvas with rich interactive contents) by linking the two with a few functions to pass some parameters. These binding functions might remind someone to the OSC way of connecting of two softwares where functionalities are staying clearly isolated within the different systems, only the parameters are traveling between them represented as numbers, labels or other types of symbols.
This example at the moment works only with Chrome (new stable releases). According to the developer docs it should work with the new Safari (WebKit) browser but I couldn’t make it. Hope this will be fixed soon…
Try out the LIVE DEMO (you need latest stable Google Chrome
)
Download full source code (zip) from here
Sand Experiments
The last few weeks I was out of the circles & routines of my regular life, did not use my ordinary tools, I was visiting and contemplating different reality concepts during my journey in the far east. Self confidence, clean associations, purity arise without the regular routines that are wired so deep in me. To avoid regular routines: I travelled far and was doing something else.
Some days I was playing with sand. I saw some crabs digging holes into the sand to avoid attacks on the beach. They left really interesting patterns on the sand. I started cutting holes into a cardboard, pouring sand onto the surface of it, then observing the naturally organized patterns that are exactly the same that we find in the -these days so overhyped- algorithms of voronoi tessellations and diagrams.
After all it is really interesting that some basic computational calculations can be done with some natural forces (gravity and density in this case). Take time for example: sandglass is one of the simplest exact visualizations of time that we can find in nature.
The other example is a visible process instead of a “compiled” result. The sand is dancing here. It is in continuous movement, an ever-cycling entity, acts like a liquid, morphing shapes of mountains and oceans become visible between the interaction of the grains. It is a sort of low-tech reaction diffusion system. These methods are also used so widely in the world of computer graphics as natural simulations. The system here consists of sand, oscillating air pressure and gravity: a speaker is placed under the surface that is resonating at 60Hz.
Javascript Procedural Sound Experiments: Rain & Wind [ProcessingJS + WebPd]
If working with sound on the web, it is common to use sound samples for events in games or rich content websites, but it is rarely possible to make procedural sound synthesis for different situations. In games there are often easier to use artificial, procedural sounds for rain, glass breaking, wind, sea etc. because of the ease of changing their parameters on the fly. I was really excited when I discovered a small project called WebPd that is about to port the Pure Data sound engine to pure JavaScript and make it possible to use directly on websites without plugins. This script is initiated by Chris McCormick and is based on the Mozilla Audio Data API that is under intense development by the Mozilla Dev team: at the moment it is working with Firefox 4+.
Also - being a small project - there are only a few, very basic Pd objects implemented at the moment. For a basic sound synthesis you can use a few sound generator objects with pd (such as osc~, phasor~, noise), but there is no really timing objects (delay, timer, etc) so all event handling must be solved within the Javascript side and only triggered synthesis commands can be left for the Pd side. If you are familiar developing projects with libPd for Android, or ofxPd for OpenFrameworks, you may know the situation.
So here I made a small test composition of a landscape that has some procedural audio elements: rain and wind. The rain is made with random pitched oscillators with very tiny ADSR events (which is in fact only A/R event that is called within the Javascript code and the actual values are passed directly to a line~ object in Pd). Density of the rain is changing with some random factor, giving a more organic vibe for the clean artificial sounds. The wind is made with a noise object that is filtered continuously with a high pass filter object. The amount of filtering is depending on the size (area) of the triangle that is visible in the landscape as an “air sleeve”.
If you would like to create parallel, polyphonic sounds, it might be a good idea to instantiate more Pd classes for different sound events. When I declared a single Pd instance only, not all the messages were sent through the Pd patch: messages that I sent in every frame (for continuous noise filtering of the ‘wind’) ‘clashed’ with discrete, separate messages that were sent when the trigger() function has been called for the raindrops. So I ended up using two Pd engines at the same time in the setup():
pd_rain = new Pd(44100, 200);
pd_rain.load("rain.pd", pd_rain.play);
pd_wind = new Pd(44100, 200);
pd_wind.load("wind.pd", pd_wind.play);
This way, accurate values are sent to each of the pd engines. try out the demo here (you need Firefox 4+)
For user input and visual display I am using the excellent ProcessingJS library which makes possible to draw & interact easily with the html5 canvas tag directly using Processing syntax. The response time (latency) is quite good, sound signals are rarely have drop outs, which is really impressive if you have some experiments with web based sound wave generation.
Permutation
Permutation is a sound instrument developed for rotating, layering and modifying sound samples. Its structure is originally developed for a small midi controller that has a few knobs and buttons. It is then mounted on a broken piano, along several contact microphones. The basic idea of using a dead piano as an untempered, raw sound source is reflecting to the central state of this instrument in the context of western music during the past few hundred years and so. Contact mikes, prepared pianos as we know are representing a constant dialogue between classical music theory and other more recent free forms of artistic and philosophical representations. The aim of Permutation however is a bit different from this phenomenon. It is a temporal extension of gestures made on the whole resonating body of a uniquely designed sound architecture (the grand piano). In fact it is so well designed that it used to be as a simulation tool for composers to mimic a whole group of musicians. A broken, old wood structure has many identical sound deformations that it becomes an intimate space for creating strange and sudden sounds that can be rotated back and folded back onto itself.
The system is built for live performances and had been used for making music for the Artus independent theatre company. The pure data patch can be downloaded from here. Please note it had been rapidly developed to be used with a midi controller so the patch may confuse someone. The patch contains some code from the excellent rjdj library at Github.
Aether
Aether is a wireless remote application developed for collecting spatially distributed sound memories. Its aim is to collect and use sound events on the fly. The client interface is shown on the left, the performance environment we are using it in can be seen on the right, a diagram of the setup is on the bottom of the image.
While walking around a larger space (that is coverable by an ordinary wireless network) one can send / receive realtime data between the client app and the sound system. All the logic, synthesis, processing is happening in the network provider server (a laptop located somewhere in space) while control, interaction and site-specific probability happens on the portable, site dependent device. The system was used for several performances of the Artus independent theatre company where action, movement had to be recorded and played back immediately.
The client app is running on an Android device that is connected to a computer running Pure Data through local wireless network (using OSC protocol). The space for interaction and control can be extended to the limits of the wireless network coverage and the maximum distance of the radio frequency microphone and its receiver. The system had been tested and used in a factory building up to a 250-300 square meters sized area.
The client code is based on an excellent collection of Processing/Android examples by ~popcode that is available to grab from Gitorious.
The source is available from here (note: the server Pd patch is not included hence its ad-hoc spaghettiness)