This video developed from my initial “Synchresis” project (see link below) and is a clear, audible example of how our perception works. The sounds can be immediately associated with the squares, as their source, even if they are in fact separate elements. Watching the video, the squares, through their corresponding sounds, gradually occupy a space in our memory. It is a semi virtual space, created by the physical nature of the sounds. This relationship between sound and position is a natural analogy with the three dimensional space we inhabit and to which we are well accustomed.

The occurrence of events can be interpreted as separate and traced only up to a point as the video progresses. Then it all becomes an indistinguishable, almost annoying, mass of events, probably even causing anxiety, or maybe boredom. 

More details about how it was made following the links:

- spatialisation with the ITD plugin made in Pure Data

- PD sampler

- initial Synchresis video

11 months ago 

The human auditory system uses several cues to determine the spatial localisation of sounds. The most relevant ones for my project are the Interaural Time and Level differences, which represent the tiny differences in the arrival time of a sound and its amplitude at our two ears.

At a speed of sound of 345 meters per second it takes a sound approximately 580 microseconds to travel 20 centimetres, and that is almost half of 1 thousandth of a second. Try rubbing your hands in front of you while moving them left-right with your eyes closed in quiet environment and listen. You can hear how accurate our brain is with numbers smaller than half of 1 thousandth of a second.

Each of the squares in the Synchresis 3D video is linked to a group of sounds with the same pitch. Implementing ITD firms up the correlation between these, because it takes advantage of the high precision with which the brain reads the location of sounds.

It made sense to spread the sounds from left and right on each row in the PD sampler and use 13 positions, 6 on each side and one in the middle. This fits perfectly with the number of squares on each row in the After Effects project.

I limited the distance to 6 meters because further away the sounds became too quiet compared to the close ones. In real life the spectrum (frequency content) of a sound changes with distance, and only in an anechoic room can a sound be heard on its own, without reverberation.

In this project the lighter coloured the squares are, the higher pitched and quieter their sonic representation. However, the image presents all squares on the same plane. The colours give the image its unity; neighbouring squares share similar spectrum. Sounds have to be different while transducing some attributes, a bit like groups of friends. In nature, sounds which are further away sound softer; less sharp; edges seem polished, rounded. The compromise in the current interpretation is to keep high frequencies as they are, not filtered out in conformity with distance, with the intention to apply some of the 2 dimensionality of  the squares to their appearance. (simultaneously far and close)

The model I chose for ITD is based on Joel D. Miller’s paper “Modeling interaural time difference assuming a spherical head” (2001),  where the author presents this method as a viable alternative to HRTF (head related transfer function) by comparing the results of the Spherical Head derived equation with actual measurements of the HRTF.

The images above represent my implementation of his equations in Pure Data.

1 year ago  1 note

Here can be seen a snapshot of the semi-automatic sound triggering mechanism I created exclusively for this this project. The concept is simple but implementation was not and I refined it over the course of 2 months (while also working on other projects).

My starting point was that I needed 91 audio files, to animate 13x7 squares in the After Effects project. I also wanted to use unique sounds made for this project.

I could have made a normal composition and used sounds from it to control the squares, but there are several reasons for not doing that.

It would have been harder for the sound to be perceived as part of the image if it had the qualities of a pre-composed piece of music.

Avoiding that issue required a certain level of control and determinism over the sounds, but also a great amount of chance.

I enforced my will over the shape of each sound, but allowed a certain degree of freedom in the development of events. Here are a few examples of good looking sounds.

In the current implementation each of the 7 rows represents an octave with 13 pitches, from C to C one octave above. For each pitch (note on the octave, or frequency) I made several sounds and there are equal opportunities for each of them to be played back. This offers some diversity while keeping the overall sound uniform.

The images at the top of the page are examples of rows.

I will also refer to sounds as “samples” - that is not to be confused with samples that represent a digital audio signal (as in 44100 samples per second).

There is a global control mechanism that changes the duration of events for each row, but also the speed with which the samples are triggered, the gaps between the notes on a row or the focus on a specific group of sounds on each row.

An important piece of the Synchresis 3D video is the interaural time and intensity (ITD) plugin. This places all the sounds in an exact point in space, which will be the same for every other occurrence of the same sound. Hopefully this will sustain the illusion of morphing sound and image.

1 year ago 

This is a technique I learned from Andrew Kramer’s website, a simple but powerful concept, and something I could understand and do myself. I had to extract keyframe information for each of the 91 audio tracks and zoom into each waveform to see where exactly the most change in intensity takes place. Then I rescaled those values to make the squares dim and not just turn on and off. The rows in the After Effects project represent 7 octaves, starting from 31 Hz (C1) On each row  there are 13 squares (and tones - pitches) because I wanted to fit 7 rows in the picture. The actual reason for using this horizontal distribution (13 instead of 12) is because it best suited the spacing between the squares, but it works really well as it makes each octave span between C and the C an octave above.  The audio files I used in this project are recorded with the PD (Pure Data) sampler. However, the sound design of the individual sounds used with the sampler was made with the native ProTools plugins.

This is a technique I learned from Andrew Kramer’s website, a simple but powerful concept, and something I could understand and do myself.

I had to extract keyframe information for each of the 91 audio tracks and zoom into each waveform to see where exactly the most change in intensity takes place. Then I rescaled those values to make the squares dim and not just turn on and off.

The rows in the After Effects project represent 7 octaves, starting from 31 Hz (C1) On each row  there are 13 squares (and tones - pitches) because I wanted to fit 7 rows in the picture.

The actual reason for using this horizontal distribution (13 instead of 12) is because it best suited the spacing between the squares, but it works really well as it makes each octave span between C and the C an octave above. 

The audio files I used in this project are recorded with the PD (Pure Data) sampler. However, the sound design of the individual sounds used with the sampler was made with the native ProTools plugins.

1 year ago  View high resolution

Since I had no previous experience with video or image editing software, I chose to make the project in After Effects, where I could use the amplitude of the audio signal to animate the squares. I aimed to create a panoramic picture, a submersive experience, and as much as possible to avoid any external distraction when watched in full screen mode on a laptop. The only slightly brain engaging operation was to calculate the square size, and the coordinates for each square. It took about two evenings and a lot of patience to make all the squares, choose the right colours and place them in the right spot. After the upper row and left-hand column it became easier because I had learned the coordinates for the vertical position, and the horizontal one was the same for the entire column. Here is more on how I animated the squares with audio keyframe data.

Since I had no previous experience with video or image editing software, I chose to make the project in After Effects, where I could use the amplitude of the audio signal to animate the squares.

I aimed to create a panoramic picture, a submersive experience, and as much as possible to avoid any external distraction when watched in full screen mode on a laptop.

The only slightly brain engaging operation was to calculate the square size, and the coordinates for each square. It took about two evenings and a lot of patience to make all the squares, choose the right colours and place them in the right spot.

After the upper row and left-hand column it became easier because I had learned the coordinates for the vertical position, and the horizontal one was the same for the entire column.

Here is more on how I animated the squares with audio keyframe data.

1 year ago  View high resolution

This is an exploration of Synchresis, a concept developed by Michel Chion (Audio-vision: sound on screen, 1994) regarding our perception of simultaneously occurring sounds and images, and how the brain glues these together.

To test the theory I have chosen to contrast smooth, subtle colours and raw, unrefined sound quality. The effect seems to work quite well.


For this first video I chose about 200 sounds from a sample library created by myself and my fellow students. The main selection criterion was for the sounds to have a slightly perceivable difference in pitch. Then I arranged them as notes within octaves, in a “tiny” PD (Pure Data) sampler. The video is made in the Adobe After Effects project, where I also linked the sounds with  the squares, associating lower pitches with dark colours and higher pitches with the lighter ones.

In order to emphasize the synchresis effect I am developing the system further. In the next video each square will have a distinct audible position in space. Based on the interaural time and intensity differences, I made a plugin (patch) in Pure Data that accurately places a sound 90 degrees left and right, and at a distance between 1 and 6 meters.

I have also worked on refining the sounds to make them more compatible with the squares and to better define their pitches. Have a look at some of the new waveforms.

Link to Synchresis 3D video.

1 year ago 

Tiny selection of waveforms with an appealing visual character, resulting from the intensive sound transformation process that followed my first video. These are the samples included in the final version of the Pure Data sample player.

1 year ago  9 notes

This video developed from my initial “Synchresis” project (see link below) and is a clear, audible example of how our perception works. The sounds can be immediately associated with the squares, as their source, even if they are in fact separate elements. Watching the video, the squares, through their corresponding sounds, gradually occupy a space in our memory. It is a semi virtual space, created by the physical nature of the sounds. This relationship between sound and position is a natural analogy with the three dimensional space we inhabit and to which we are well accustomed.

The occurrence of events can be interpreted as separate and traced only up to a point as the video progresses. Then it all becomes an indistinguishable, almost annoying, mass of events, probably even causing anxiety, or maybe boredom. 

More details about how it was made following the links:

- spatialisation with the ITD plugin made in Pure Data

- PD sampler

- initial Synchresis video

The human auditory system uses several cues to determine the spatial localisation of sounds. The most relevant ones for my project are the Interaural Time and Level differences, which represent the tiny differences in the arrival time of a sound and its amplitude at our two ears.

At a speed of sound of 345 meters per second it takes a sound approximately 580 microseconds to travel 20 centimetres, and that is almost half of 1 thousandth of a second. Try rubbing your hands in front of you while moving them left-right with your eyes closed in quiet environment and listen. You can hear how accurate our brain is with numbers smaller than half of 1 thousandth of a second.

Each of the squares in the Synchresis 3D video is linked to a group of sounds with the same pitch. Implementing ITD firms up the correlation between these, because it takes advantage of the high precision with which the brain reads the location of sounds.

It made sense to spread the sounds from left and right on each row in the PD sampler and use 13 positions, 6 on each side and one in the middle. This fits perfectly with the number of squares on each row in the After Effects project.

I limited the distance to 6 meters because further away the sounds became too quiet compared to the close ones. In real life the spectrum (frequency content) of a sound changes with distance, and only in an anechoic room can a sound be heard on its own, without reverberation.

In this project the lighter coloured the squares are, the higher pitched and quieter their sonic representation. However, the image presents all squares on the same plane. The colours give the image its unity; neighbouring squares share similar spectrum. Sounds have to be different while transducing some attributes, a bit like groups of friends. In nature, sounds which are further away sound softer; less sharp; edges seem polished, rounded. The compromise in the current interpretation is to keep high frequencies as they are, not filtered out in conformity with distance, with the intention to apply some of the 2 dimensionality of  the squares to their appearance. (simultaneously far and close)

The model I chose for ITD is based on Joel D. Miller’s paper “Modeling interaural time difference assuming a spherical head” (2001),  where the author presents this method as a viable alternative to HRTF (head related transfer function) by comparing the results of the Spherical Head derived equation with actual measurements of the HRTF.

The images above represent my implementation of his equations in Pure Data.

Here can be seen a snapshot of the semi-automatic sound triggering mechanism I created exclusively for this this project. The concept is simple but implementation was not and I refined it over the course of 2 months (while also working on other projects).

My starting point was that I needed 91 audio files, to animate 13x7 squares in the After Effects project. I also wanted to use unique sounds made for this project.

I could have made a normal composition and used sounds from it to control the squares, but there are several reasons for not doing that.

It would have been harder for the sound to be perceived as part of the image if it had the qualities of a pre-composed piece of music.

Avoiding that issue required a certain level of control and determinism over the sounds, but also a great amount of chance.

I enforced my will over the shape of each sound, but allowed a certain degree of freedom in the development of events. Here are a few examples of good looking sounds.

In the current implementation each of the 7 rows represents an octave with 13 pitches, from C to C one octave above. For each pitch (note on the octave, or frequency) I made several sounds and there are equal opportunities for each of them to be played back. This offers some diversity while keeping the overall sound uniform.

The images at the top of the page are examples of rows.

I will also refer to sounds as “samples” - that is not to be confused with samples that represent a digital audio signal (as in 44100 samples per second).

There is a global control mechanism that changes the duration of events for each row, but also the speed with which the samples are triggered, the gaps between the notes on a row or the focus on a specific group of sounds on each row.

An important piece of the Synchresis 3D video is the interaural time and intensity (ITD) plugin. This places all the sounds in an exact point in space, which will be the same for every other occurrence of the same sound. Hopefully this will sustain the illusion of morphing sound and image.

This is a technique I learned from Andrew Kramer’s website, a simple but powerful concept, and something I could understand and do myself. I had to extract keyframe information for each of the 91 audio tracks and zoom into each waveform to see where exactly the most change in intensity takes place. Then I rescaled those values to make the squares dim and not just turn on and off. The rows in the After Effects project represent 7 octaves, starting from 31 Hz (C1) On each row  there are 13 squares (and tones - pitches) because I wanted to fit 7 rows in the picture. The actual reason for using this horizontal distribution (13 instead of 12) is because it best suited the spacing between the squares, but it works really well as it makes each octave span between C and the C an octave above.  The audio files I used in this project are recorded with the PD (Pure Data) sampler. However, the sound design of the individual sounds used with the sampler was made with the native ProTools plugins.

This is a technique I learned from Andrew Kramer’s website, a simple but powerful concept, and something I could understand and do myself.

I had to extract keyframe information for each of the 91 audio tracks and zoom into each waveform to see where exactly the most change in intensity takes place. Then I rescaled those values to make the squares dim and not just turn on and off.

The rows in the After Effects project represent 7 octaves, starting from 31 Hz (C1) On each row  there are 13 squares (and tones - pitches) because I wanted to fit 7 rows in the picture.

The actual reason for using this horizontal distribution (13 instead of 12) is because it best suited the spacing between the squares, but it works really well as it makes each octave span between C and the C an octave above. 

The audio files I used in this project are recorded with the PD (Pure Data) sampler. However, the sound design of the individual sounds used with the sampler was made with the native ProTools plugins.

Since I had no previous experience with video or image editing software, I chose to make the project in After Effects, where I could use the amplitude of the audio signal to animate the squares. I aimed to create a panoramic picture, a submersive experience, and as much as possible to avoid any external distraction when watched in full screen mode on a laptop. The only slightly brain engaging operation was to calculate the square size, and the coordinates for each square. It took about two evenings and a lot of patience to make all the squares, choose the right colours and place them in the right spot. After the upper row and left-hand column it became easier because I had learned the coordinates for the vertical position, and the horizontal one was the same for the entire column. Here is more on how I animated the squares with audio keyframe data.

Since I had no previous experience with video or image editing software, I chose to make the project in After Effects, where I could use the amplitude of the audio signal to animate the squares.

I aimed to create a panoramic picture, a submersive experience, and as much as possible to avoid any external distraction when watched in full screen mode on a laptop.

The only slightly brain engaging operation was to calculate the square size, and the coordinates for each square. It took about two evenings and a lot of patience to make all the squares, choose the right colours and place them in the right spot.

After the upper row and left-hand column it became easier because I had learned the coordinates for the vertical position, and the horizontal one was the same for the entire column.

Here is more on how I animated the squares with audio keyframe data.

This is an exploration of Synchresis, a concept developed by Michel Chion (Audio-vision: sound on screen, 1994) regarding our perception of simultaneously occurring sounds and images, and how the brain glues these together.

To test the theory I have chosen to contrast smooth, subtle colours and raw, unrefined sound quality. The effect seems to work quite well.


For this first video I chose about 200 sounds from a sample library created by myself and my fellow students. The main selection criterion was for the sounds to have a slightly perceivable difference in pitch. Then I arranged them as notes within octaves, in a “tiny” PD (Pure Data) sampler. The video is made in the Adobe After Effects project, where I also linked the sounds with  the squares, associating lower pitches with dark colours and higher pitches with the lighter ones.

In order to emphasize the synchresis effect I am developing the system further. In the next video each square will have a distinct audible position in space. Based on the interaural time and intensity differences, I made a plugin (patch) in Pure Data that accurately places a sound 90 degrees left and right, and at a distance between 1 and 6 meters.

I have also worked on refining the sounds to make them more compatible with the squares and to better define their pitches. Have a look at some of the new waveforms.

Link to Synchresis 3D video.

Tiny selection of waveforms with an appealing visual character, resulting from the intensive sound transformation process that followed my first video. These are the samples included in the final version of the Pure Data sample player.

About:

audio visual work

- currently studying Creative Music Technology @ Bath Spa University

- contact: andrei.branea09@bathspa.ac.uk

Following:

Alexandru Adam