Archive for September, 2008

Frequency Mandelbrot

Sunday, September 21st, 2008

Frequency Mandelbrot 1Where I had been experimenting with just the outsides of the mandelbrot before, I’ve started having a look at modifying the inside of the fractal as well. I’ve continued to use the frequency spectrum for the effects simply because it goes with the frequency spectrum used in the background.

import ddf.minim.*;
import ddf.minim.analysis.*;

FFT fft;

AudioPlayer song;
BeatDetect beat;

float fSize;

void setup()
{
size(300, 300);

// always start Minim first!
Minim.start(this);

// specify 512 for the length of the sample buffers
// the default buffer size is 1024
song = Minim.loadFile("freedom.mp3", 512);
song.play();
beat = new BeatDetect();

float fSize = 20;
// an FFT needs to know how
// long the audio buffers it will be analyzing are
// and also needs to know
// the sample rate of the audio it is analyzing is
fft = new FFT(song.bufferSize(), song.sampleRate());
}

void draw()
{
background(0);
// first perform a forward fft on one of song's buffers
// I'm using the mix buffer
//  but you can use any one you like
fft.forward(song.mix);
fill(255);

loadPixels();

// Maximum number of iterations for each point on the complex plane
float maxiterations = 200;
float xmin = -2.5;
float ymin = -2;
float wh = 2;

beat.detect(song.mix);
if ( beat.isOnset() ) fSize += 0.1;

fSize = constrain(fSize, -10, 10);

ymin = -1;
xmin = -1.5;

maxiterations = maxiterations + fSize;

// x goes from xmin to xmax
float xmax = xmin + wh;
// y goes from ymin to ymax
float ymax = ymin + wh;

// Calculate amount we increment x,y for each pixel
float dx = (xmax - xmin) / (width);
float dy = (ymax - ymin) / (height);

// Start y
float y = ymin;
for(int j = 0; j < height; j++) {
// Start x
float x = xmin;
for(int i = 0;  i < width; i++) {

// Now we test, as we iterate z = z^2 + cm does z tend towards infinity?
float a = x;
float b = y;
int n = 0;
while (n < maxiterations) {
float aa = a * a;
float bb = b * b;
float twoab = 2.0 * a * b;
a = aa - bb + x;
b = twoab + y;
// Infinty in our finite world is simple, let's just consider it 16
if(aa + bb > 16.0f) {
break;         }
n++;
}

// We color each pixel based on how long it takes to get to infinity
// If we never got there, let's pick the color black
if (n == maxiterations) pixels[i+j*width] = 0;
else pixels[i+j*width] = color(n*(song.right.get(i)*50) % 255, n*(song.left.get(i)*50) % 200, n*(song.right.get(i)*30));
x += dx;
}
y += dy;
}
updatePixels();
}

void stop()
{
song.close();
super.stop();
}

Frequency Mandelbrot 2In this version I have used the BeatDetect class and another variable, fSize with a constraint on it and then added to the maximum iterations. This allows the frequency spectrum to spread through the mandelbrot fractal, or appear to anyway. It is actually just using the frequency spectrum to colour the fractal, and this makes it appear like spectrum is forming the fractal.

You can have a look at it in action here. The song used in this piece is “Freedom (Waking Mix)” by vo1k1 2008 – Licensed under Creative Commons Attribution Noncommercial (3.0).

This is fairly CPU intensive, so many computers will have trouble running this, it is also about 13mb to load so you probably don’t want to try it on a low speed connection.

Minim and Beat Detection

Wednesday, September 17th, 2008

While I was looking through the Minim documentation, I came across the built in BeatDetect class that analyses the audio input for beats. My understanding is that it looks for patterns in the rhythm to determine individual beats of a particular type, be it the bass drum, snare, cymbals etc. It has a couple of modes for detecting beats, one is sound energy mode and the other is frequency energy mode.

Sound energy mode just looks for spikes in the audio mix levels and returns a boolean response of true when a spike occurs.

Frequency energy mode allows you to look at particular frequency bands and use their output the same way you would the output of sound energy mode.

I decided to have a play with the sample code using frequency energy mode that is provided in the documentation to figure out how to control it and so on.

The sample code checks the frequency bands for the snare, kick/bass drum and high hats and then has text for each one that responds to the audio input for their particular band.

I didn’t do anything overly complex with this as of yet, just changed the objects that are manipulated to boxes rather than text so I could get an idea of of what does what.

Here is the code that I ended up with after my changes:

import ddf.minim.*;
import ddf.minim.analysis.*;

class BeatListener implements AudioListener
{
private BeatDetect beat;
private AudioPlayer source;

BeatListener(BeatDetect beat, AudioPlayer source)
{
this.source = source;
this.source.addListener(this);
this.beat = beat;
}

void samples(float[] samps)
{
beat.detect(source.mix);
}

void samples(float[] sampsL, float[] sampsR)
{
beat.detect(source.mix);
}
}

AudioPlayer song;
BeatDetect beat;
BeatListener bl;

float kickSize, snareSize, hatSize;

void setup()
{
size(512, 200);
smooth();
Minim.start(this);
song = Minim.loadFile("song.mp3", 1024);
song.play();
// a beat detection object that is FREQ_ENERGY mode that
// expects buffers the length of song's buffer size
// and samples captured at songs's sample rate
beat = new BeatDetect(song.bufferSize(), song.sampleRate());
beat.setSensitivity(300);
kickSize = snareSize = hatSize = 16;
// make a new beat listener, so that we won't miss any buffers for the analysis
bl = new BeatListener(beat, song);
}

void draw()
{
background(0);
fill(255);
if ( beat.isKick() ) kickSize = 32;
if ( beat.isSnare() ) snareSize = 32;
if ( beat.isHat() ) hatSize = 32;
stroke(255);
strokeWeight(kickSize);
rect(width/4, height/2, 20, 20);
stroke(255);
strokeWeight(snareSize);
rect(width/2, height/2, 20, 20);
stroke(255);
strokeWeight(hatSize);
rect(3*width/4, height/2, 20, 20);
kickSize = constrain(kickSize * 0.95, 16, 32);
snareSize = constrain(snareSize * 0.95, 16, 32);
hatSize = constrain(hatSize * 0.95, 16, 32);
}

void stop()
{
// always close Minim audio classes when you are finished with them
song.close();
// always stop Minim before exiting
Minim.stop();

// this closes the sketch
super.stop();
}

For this to work, make sure you have an MP3 or Wav file in a data folder within the sketch directory for this sketch and change song.mp3 to the filename of your song.

Using The Output of Minim in Processing

Wednesday, September 17th, 2008

After following the quick start guide on the Minim website, I put together a little waveform generator that produces a waveform for the left and right channels based on the input of an audio file along with a frequency spectrum running over the top of it. The instructions are pretty easy to follow, and the additional functionality of Minim seems fairly simply to harness to do something useful with.

Minim Output 4I’ve since taken the code from that guide and combined it with a mandelbrot fractal from the Processing tutorials and references. At the moment, the mandelbrot doesn’t really do anything near like what I want it to do, but I am making some progress in figuring out what I can do with Minim and how I can use it for what I want to do.

Initially I took the frequency spectrum and used it to colour parts of the mandelbrot. To start with I only manipulated the red colouring of the fractal to see what would happen which gave me this (you may need to scroll to the right in some places sorry):

import ddf.minim.*;
import ddf.minim.analysis.*;

AudioPlayer song;
FFT fft;

float xmin = -2.5;
float ymin = -2;
float wh = 4;

void setup()
{
size(300, 300);

// always start Minim first!
Minim.start(this);

// specify 512 for the length of the sample buffers
// the default buffer size is 1024
song = Minim.loadFile("song.mp3", 512);
song.play();

// an FFT needs to know how
// long the audio buffers it will be analyzing are
// and also needs to know
// the sample rate of the audio it is analyzing is
fft = new FFT(song.bufferSize(), song.sampleRate());
}

void draw()
{
background(0);
// first perform a forward fft on one of song's buffers
// I'm using the mix buffer
//  but you can use any one you like
fft.forward(song.mix);

loadPixels();

// Maximum number of iterations for each point on the complex plane
int maxiterations = 200;

// x goes from xmin to xmax
float xmax = xmin + wh;
// y goes from ymin to ymax
float ymax = ymin + wh;

// Calculate amount we increment x,y for each pixel
float dx = (xmax - xmin) / (width);
float dy = (ymax - ymin) / (height);

// Start y
float y = ymin;
for(int j = 0; j < height; j++) {
// Start x
float x = xmin;
for(int i = 0;  i < width; i++) {

// Now we test, as we iterate z = z^2 + cm does z tend towards infinity?
float a = x;
float b = y;
int n = 0;
while (n < maxiterations) {
float aa = a * a;
float bb = b * b;
float twoab = 2.0 * a * b;
a = aa - bb + x;
b = twoab + y;
// Infinty in our finite world is simple, let's just consider it 16
if(aa + bb > 16.0f) {
break;  // Bail
}
n++;
}

// We color each pixel based on how long it takes to get to infinity
// If we never got there, let's pick the color black
if (n == maxiterations) pixels[i+j*width] = 0;
else pixels[i+j*width] =
color(n*(song.right.get(i)*50) % 255, n*16 % 200, n*16 % 100);
x += dx;
}
y += dy;
}
updatePixels();
}

void stop()
{
song.close();
super.stop();
}

To make the above work, you’ll need to put an MP3 or Wav audio file in a “data” directory within the sketch folder for this sketch and change song.mp3 to the name of your audio file. You can see it in action here. The song used in this piece is “Freedom (Waking Mix)” by vo1k1 2008 – Licensed under Creative Commons Attribution Noncommercial (3.0).

Minim Output 5aIt will run a series of lines of various lengths and thicknesses at varying speeds based on the input of the audio file.

Considering how cool this was looking, I decided to add the left audio channel for the green colouring to see what would happened, so I modified the colouring to the following:

if (n == maxiterations) pixels[i+j*width] = 0;
else pixels[i+j*width] = color(n*(song.right.get(i)*50) % 255, n*(song.left.get(i)*50) % 200, n*16 % 100);

Minim Output 5cThis is actually really cool, if you run an audio track through it that has a distinctively different left and right channel you get much more variation in the colouring and it appears to be synced more with the music.

If you try the above, you’ll get reds coming through for the right channel, greens coming through for the left and yellows where they overlap. Since you have two different sets of lines, you can see them change with each channel, rather than only with one channel, this is why it seems to be synced up better, and the colour changes help with this since the colours match the left and right mix. You can see it in action here. The song used in this piece is “Freedom (Waking Mix)” by vo1k1 2008 – Licensed under Creative Commons Attribution Noncommercial (3.0).

Adding the second colouring control in also makes colour variations immediately around the main parts of the mandelbrot more prominent, and as they change it makes the fractal appear to be behaving like it is encircled with flames.

Minim Output 7aThis does make the background blue though rather than grey which I don’t like too much since the fractal doesn’t stand out as much with the blue background as it did grey.

I decided to change the colouring again slightly, this time using the audio mix to control the blue colouring. I used the right audio channel again, no particular reason really. So this is what my colouring looks like now:

if (n == maxiterations) pixels[i+j*width] = 0;
else pixels[i+j*width] = color(n*(song.right.get(i)*50) % 255, n*(song.left.get(i)*50) % 200, n*(song.right.get(i)*30));

Minim Output 7e
Minim Output 7c

You can see it in action here. The song used in this piece is “Freedom (Waking Mix)” by vo1k1 2008 – Licensed under Creative Commons Attribution Noncommercial (3.0).

This has the same left and right channel variation and effects as the previous version, but it changes the colours slightly, the lines are now variations or predominately green and purple with a black background. This makes the mandelbrot stand out much more than before, but it also makes the appearance of flames around the fractal much more prominent as they are now blue closest to the fractal and shades of purple the further out you go. This looks really cool and makes the flames even more realistic (except for the colouring of course) and the flames now appear to reach out further making them stand and their variations stand out more.

Playing Audio in Processing

Friday, September 5th, 2008

I’ve been looking at how to import an audio source into Processing so that it can be used to manipulate a visual output. Using the Minim sound library I should be able to manipulate things based on either an audio file or an audio input through the line-in or mic-in.

I have been going through the Quick Start Guide which is very useful and having finished it, I have a Processing sketch that will playback a file and display the audio spectrum and waveforms which is pretty cool. I can see how I should be able to use the audio controls to manipulate an image, so I think I’m going to start having an experiment with applying the similar concepts to other images.