I’ve always been interested in synesthesia and steganography. The two aren’t really related, but when I was tasked with a project for an art class, I decided to make it interesting by trying to marry the two.
You can “listen” to a picture pretty easily. By simply copying and pasting the contents of a bitmap into a .wav file, you can trick a program like Audacity into “playing” an image file. Here’s what the Mona Lisa sounds like using this method:
While interesting, I wanted something that was a little more musical. I wanted something that sounded nice. For a while I played with the idea of trying to make an image that, when converted into a .wav in this way, would sound interesting. I quickly discovered that the color content of the image was much less important than the its spatial frequency content. That is, while a single vertical red bar and a green bar might sound identical (a single tone), two bars stacked next to each other would sound very different (a doubling in frequency). This is probably better illustrated by the image below:
Our eyes produce these ‘emergent’ colors by both being sensitive to intensity and by treating combinations of colors differently. To try to imitate this, I wrote a java program using JFugue to translate the pixels in an image into a midi sequence of three-note piano chords. For each pixel, the three values for red, green, and blue get mapped onto corresponding sections of the piano.
It’s not perfect, but the results were actually pretty interesting. I had to downsample the images heavily so they could be played in a timely manner (640×480 pixels at 2 pixels per second would take about 42 hours to play). Here are some sample images and their corresponding ‘songs’:
The rhythm you hear in the Mona Lisa midi is actually due to the padding bytes used in the bitmap format the image was saved in. These padding bytes occur at regular intervals ever few pixels, and always have the same value. This causes the rhythm effect your hear. It had never occurred to me that a file format might have a rhythm, but by all accounts it makes sense. The padding bytes were removed in the blue flower midi.
At any rate, it was a really interesting project and I hope to get more time someday to keep playing with it.