An audio file can be played, paused, rewound, and fast-forwarded. The left and right audio channels as well as beats and the overall volume are taken and used to colour a mandelbrot fractal. The left and right audio channels generate a series of lines that change speed and direction based on the audio peaks. The centre of the fractal has a constraint on it based on the beats, so as the amount of beats increase, it will freeze and stay still while the outer area of the fractal, which appears as a background continues to animate around it. And yes, it is meant to have the lines through it.

The keyboard controls for playback are:

Play: P
Pause: S
Fast-Forward: F
Rewind: R
Restart: H
Stop: Q

A note on these playback options, at the moment, ‘Q‘ will actually close the link to the song file, so this should not be used unless it is needed. Attempting to start the song again with ‘P‘ after ‘Q‘ has been used will cause Processing to error. I haven’t gotten around this as of yet unfortunately.

To replay a song from the start, press ‘H‘ and this will return the play head to the start of the file.

Rewind ‘R‘ and fast forward ‘F‘ skip the track back and forward by 500 milliseconds, so ‘R‘ should be used in place of ‘H‘ if you don’t want to return to the start of the song.

The source code is a little bit different to previous versions as I have added a class for keyPressed to determine what to do if the control keys are pressed:

// Create the controls.
void keyPressed()
{
if ( key == 'p' ) song.play(); // Press P to play the song.
if ( key == 's' ) song.pause(); // Press S to pause playing.
if ( key == 'h' ) song.rewind(); // Press H to reset to the start.
if ( key == 'f' ) song.skip(500); // Press F to fast forward.
if ( key == 'r' ) song.skip(-500); // Press R to rewind.
if ( key == 'q' ) song.close(); // Press Q to quit playing.
}

It’s all fairly straight forward, the Minim library is very simple to use and there are some great tools in it that you can play with. I highly recommend it if you are working with audio in Processing.

You can have a look at it in action here. The song used in this piece is “Freedom (Waking Mix)” by vo1k1 2008 – Licensed under Creative Commons Attribution Noncommercial (3.0).

This is fairly CPU intensive, so many computers will have trouble running this, it is also about 9mb to load so you probably don’t want to try it on a low speed connection.

The full source code is available from the same page as the applet, or you can see it after the jump:

import ddf.minim.*;
import ddf.minim.analysis.*;

FFT fft;

AudioPlayer song;
BeatDetect beat;

float fSize;

void setup()
{
size(300, 300);

// Start Minim
Minim.start(this);

// Set the length of the sample buffers.
song = Minim.loadFile("freedom.mp3", 512);
// Uncomment song.play() to play the song as soon as the environment loads
//  song.play();
beat = new BeatDetect();

float fSize = 20;
// Tell the FFT the buffer size and sample rate.
fft = new FFT(song.bufferSize(), song.sampleRate());
}

// Create the controls.
void keyPressed()
{
if ( key == 'p' ) song.play(); // Press P to play the song.
if ( key == 's' ) song.pause(); // Press S to pause playing.
if ( key == 'h' ) song.rewind(); // Press H to reset to the start.
if ( key == 'f' ) song.skip(500); // Press F to fast forward.
if ( key == 'r' ) song.skip(-500); // Press R to rewind.
if ( key == 'q' ) song.close(); // Press Q to quit playing.
}

void draw()
{
background(0);
// Run a mix FFT
fft.forward(song.mix);
fill(255);

loadPixels();

// Maximum number of iterations for each point.
float maxiterations = 200; // Lower renders quicker
float xmin = -2.5;
float ymin = -2;
float wh = 2;

// Detect the beats in the song mix
beat.detect(song.mix);
// Iterate fSize based on beats.
if ( beat.isOnset() ) fSize += 0.1;

// Restrict fSize to create variation when the beat count gets too high.
fSize = constrain(fSize, -10, 10);

ymin = -1;
xmin = -1.5;

maxiterations = maxiterations + fSize;

// x goes from xmin to xmax
float xmax = xmin + wh;
// y goes from ymin to ymax
float ymax = ymin + wh;

// Calculate amount we increment x,y for each pixel
float dx = (xmax - xmin) / (width);
float dy = (ymax - ymin) / (height);

// Start y
float y = ymin;
for(int j = 0; j < height; j++) {
// Start x
float x = xmin;
for(int i = 0;  i < width; i++) {

float a = x;
float b = y;
int n = 0;

while (n < maxiterations) {
float aa = a * a;
float bb = b * b;
float twoab = 2.0 * a * b;
a = aa - bb + x;
b = twoab + y;
if(aa + bb > 16.0f) {
break;
}
n++;
}

// Colouring pixels using the left and right audio channels - song.right.get(i) etc.
if (n == maxiterations) pixels[i+j*width] = 0;
else pixels[i+j*width] = color(n*(song.right.get(i)*50) % 255, n*(song.left.get(i)*50) % 200, n*(song.right.get(i)*30));
x += dx;
}
y += dy;
}
updatePixels();
}

// End Minim after the song finishes playing.
void stop()
{
song.close();
super.stop();
}

import ddf.minim.*;
import ddf.minim.analysis.*;

FFT fft;

AudioPlayer song;
BeatDetect beat;

float fSize;

void setup()
{
size(300, 300);

// always start Minim first!
Minim.start(this);

// specify 512 for the length of the sample buffers
// the default buffer size is 1024
song = Minim.loadFile("freedom.mp3", 512);
song.play();
beat = new BeatDetect();

float fSize = 20;
// an FFT needs to know how
// long the audio buffers it will be analyzing are
// and also needs to know
// the sample rate of the audio it is analyzing is
fft = new FFT(song.bufferSize(), song.sampleRate());
}

void draw()
{
background(0);
// first perform a forward fft on one of song's buffers
// I'm using the mix buffer
//  but you can use any one you like
fft.forward(song.mix);
fill(255);

loadPixels();

// Maximum number of iterations for each point on the complex plane
float maxiterations = 200;
float xmin = -2.5;
float ymin = -2;
float wh = 2;

beat.detect(song.mix);
if ( beat.isOnset() ) fSize += 0.1;

fSize = constrain(fSize, -10, 10);

ymin = -1;
xmin = -1.5;

maxiterations = maxiterations + fSize;

// x goes from xmin to xmax
float xmax = xmin + wh;
// y goes from ymin to ymax
float ymax = ymin + wh;

// Calculate amount we increment x,y for each pixel
float dx = (xmax - xmin) / (width);
float dy = (ymax - ymin) / (height);

// Start y
float y = ymin;
for(int j = 0; j < height; j++) {
// Start x
float x = xmin;
for(int i = 0;  i < width; i++) {

// Now we test, as we iterate z = z^2 + cm does z tend towards infinity?
float a = x;
float b = y;
int n = 0;
while (n < maxiterations) {
float aa = a * a;
float bb = b * b;
float twoab = 2.0 * a * b;
a = aa - bb + x;
b = twoab + y;
// Infinty in our finite world is simple, let's just consider it 16
if(aa + bb > 16.0f) {
break;         }
n++;
}

// We color each pixel based on how long it takes to get to infinity
// If we never got there, let's pick the color black
if (n == maxiterations) pixels[i+j*width] = 0;
else pixels[i+j*width] = color(n*(song.right.get(i)*50) % 255, n*(song.left.get(i)*50) % 200, n*(song.right.get(i)*30));
x += dx;
}
y += dy;
}
updatePixels();
}

void stop()
{
song.close();
super.stop();
}

In this version I have used the BeatDetect class and another variable, fSize with a constraint on it and then added to the maximum iterations. This allows the frequency spectrum to spread through the mandelbrot fractal, or appear to anyway. It is actually just using the frequency spectrum to colour the fractal, and this makes it appear like spectrum is forming the fractal.

You can have a look at it in action here. The song used in this piece is “Freedom (Waking Mix)” by vo1k1 2008 – Licensed under Creative Commons Attribution Noncommercial (3.0).

This is fairly CPU intensive, so many computers will have trouble running this, it is also about 13mb to load so you probably don’t want to try it on a low speed connection.

I’ve since taken the code from that guide and combined it with a mandelbrot fractal from the Processing tutorials and references. At the moment, the mandelbrot doesn’t really do anything near like what I want it to do, but I am making some progress in figuring out what I can do with Minim and how I can use it for what I want to do.

Initially I took the frequency spectrum and used it to colour parts of the mandelbrot. To start with I only manipulated the red colouring of the fractal to see what would happen which gave me this (you may need to scroll to the right in some places sorry):

import ddf.minim.*;
import ddf.minim.analysis.*;

AudioPlayer song;
FFT fft;

float xmin = -2.5;
float ymin = -2;
float wh = 4;

void setup()
{
size(300, 300);

// always start Minim first!
Minim.start(this);

// specify 512 for the length of the sample buffers
// the default buffer size is 1024
song = Minim.loadFile("song.mp3", 512);
song.play();

// an FFT needs to know how
// long the audio buffers it will be analyzing are
// and also needs to know
// the sample rate of the audio it is analyzing is
fft = new FFT(song.bufferSize(), song.sampleRate());
}

void draw()
{
background(0);
// first perform a forward fft on one of song's buffers
// I'm using the mix buffer
//  but you can use any one you like
fft.forward(song.mix);

loadPixels();

// Maximum number of iterations for each point on the complex plane
int maxiterations = 200;

// x goes from xmin to xmax
float xmax = xmin + wh;
// y goes from ymin to ymax
float ymax = ymin + wh;

// Calculate amount we increment x,y for each pixel
float dx = (xmax - xmin) / (width);
float dy = (ymax - ymin) / (height);

// Start y
float y = ymin;
for(int j = 0; j < height; j++) {
// Start x
float x = xmin;
for(int i = 0;  i < width; i++) {

// Now we test, as we iterate z = z^2 + cm does z tend towards infinity?
float a = x;
float b = y;
int n = 0;
while (n < maxiterations) {
float aa = a * a;
float bb = b * b;
float twoab = 2.0 * a * b;
a = aa - bb + x;
b = twoab + y;
// Infinty in our finite world is simple, let's just consider it 16
if(aa + bb > 16.0f) {
break;  // Bail
}
n++;
}

// We color each pixel based on how long it takes to get to infinity
// If we never got there, let's pick the color black
if (n == maxiterations) pixels[i+j*width] = 0;
else pixels[i+j*width] =
color(n*(song.right.get(i)*50) % 255, n*16 % 200, n*16 % 100);
x += dx;
}
y += dy;
}
updatePixels();
}

void stop()
{
song.close();
super.stop();
}

To make the above work, you’ll need to put an MP3 or Wav audio file in a “data” directory within the sketch folder for this sketch and change song.mp3 to the name of your audio file. You can see it in action here. The song used in this piece is “Freedom (Waking Mix)” by vo1k1 2008 – Licensed under Creative Commons Attribution Noncommercial (3.0).

It will run a series of lines of various lengths and thicknesses at varying speeds based on the input of the audio file.

Considering how cool this was looking, I decided to add the left audio channel for the green colouring to see what would happened, so I modified the colouring to the following:

if (n == maxiterations) pixels[i+j*width] = 0;
else pixels[i+j*width] = color(n*(song.right.get(i)*50) % 255, n*(song.left.get(i)*50) % 200, n*16 % 100);

This is actually really cool, if you run an audio track through it that has a distinctively different left and right channel you get much more variation in the colouring and it appears to be synced more with the music.

If you try the above, you’ll get reds coming through for the right channel, greens coming through for the left and yellows where they overlap. Since you have two different sets of lines, you can see them change with each channel, rather than only with one channel, this is why it seems to be synced up better, and the colour changes help with this since the colours match the left and right mix. You can see it in action here. The song used in this piece is “Freedom (Waking Mix)” by vo1k1 2008 – Licensed under Creative Commons Attribution Noncommercial (3.0).

Adding the second colouring control in also makes colour variations immediately around the main parts of the mandelbrot more prominent, and as they change it makes the fractal appear to be behaving like it is encircled with flames.

This does make the background blue though rather than grey which I don’t like too much since the fractal doesn’t stand out as much with the blue background as it did grey.

I decided to change the colouring again slightly, this time using the audio mix to control the blue colouring. I used the right audio channel again, no particular reason really. So this is what my colouring looks like now:

if (n == maxiterations) pixels[i+j*width] = 0;
else pixels[i+j*width] = color(n*(song.right.get(i)*50) % 255, n*(song.left.get(i)*50) % 200, n*(song.right.get(i)*30));

You can see it in action here. The song used in this piece is “Freedom (Waking Mix)” by vo1k1 2008 – Licensed under Creative Commons Attribution Noncommercial (3.0).

This has the same left and right channel variation and effects as the previous version, but it changes the colours slightly, the lines are now variations or predominately green and purple with a black background. This makes the mandelbrot stand out much more than before, but it also makes the appearance of flames around the fractal much more prominent as they are now blue closest to the fractal and shades of purple the further out you go. This looks really cool and makes the flames even more realistic (except for the colouring of course) and the flames now appear to reach out further making them stand and their variations stand out more.

The feedback was part of a practice peer review, but he went into some detail on how i could fix up my current code so that the lines follow on from each other and how to reduce the load on the CPU per cycle with the same effect. The advice is greatly appreciated.

View the new version online here.

The changes and source code are after the jump.

Originally I had:

float x = random(width);
float y = random(height);
float x2 = random(width);
float y2 = random(height);void setup()
{
size(400, 400);
strokeWeight(10);
smooth();
background(100);
frameRate(12);
}

void draw()
{
x += int(random(width*2)-width);
y += int(random(height*2)-height);
x2 += int(random(width*2)-width);
y2 += int(random(height*2)-height);
stroke(random(255));
line(x, y, x2, y2);
}

Now I have:

float x = random(width);
float y = random(height);
float x2 = 0;
float y2 = 0;
float tempx = 0;
float tempy = 0;

void setup()
{
size(400, 400);
strokeWeight(10);
smooth();
background(100);
frameRate(12);
}

void draw()
{

x2 = random(-width, width);
y2 = random(-height, height);
stroke(random(255));
line(x, y, x2, y2);
x = x2;
y = y2;
}

The main changes are assigning the results of the random set of coordinates to the original coordinates with:

x = x2;
y = y2;

These variables (x, y) were assigned 0 values initially and this is then replaced by the new coordinates at the end of every draw() cycle.

The other main change is the reduced load on the CPU with:

x2 = random(-width, width);
y2 = random(-height, height);

This is in place of:

x2 += int(random(width*2)-width);
y2 += int(random(height*2)-height);

Thats about it, ideally I should fix the colour changes as well so that they aren’t sharp changes with each line, but a gradual fade from one to another. I’ll see how I go experimenting with that and throw it up here as well when I get a chance.

If you look at my Deviant Art account at all, you’ll notice that it is absolutely full of fractal stills. I also had some big A1 and A2 fractal prints on exhibition, as well as an animation, at a recent inter-university festival between QUT and UQ. So I can easily say that fractals are definitely where my love in art is at. Since I first began experimenting with fractals, there has been one particularly big influence in my work, and that has been the work of Scott Draves aka Spot.

Spot’s work has been of interest to me basically since I began working with fractals and it continues to do so.

He has done a lot of work in fractals and started the Flam3, Electric Sheep and Dreams in High Fidelity projects, all of which are quite influential on my work, both in fractals and in other computational arts.

Not all of his work has been focused on fractals, but many of his idea’s share similarities with them. One example of this are his Dub Visuals which take chaotic still images and morph them, to form other still images in a flow similar to that of the evolution of his fractal animations.

I have previously done fractal animations where they are synced with music, both music performed by another artist and music that I have composed myself. However, this hasn’t been done in real time and has been a long process to setup each portion of the animation and sync it up with the music. Ultimately, I would love to develop a way of rendering fractals, even low quality or low resolution fractals in real-time based on audio input. Basically this would be combining the work I have done previously with the basic concepts of what I did for KKB210 where I did a series of lines and boxes that responded to audio input. The lines one was probably getting close, though ultimately no where near as detailed as I would like it to be, and definitely nowhere near as chaotic and fractal-like as I want.

That is my aim though and hopefully I will get to it sooner or later.

This version should allow further development to make the lines even more random by separating all the variables, where the original was more limited since the numbers were largely defined within the line parameters.

This version is also technically more stochastic then the first one as well as the lines are added to and built on using the += command to add the new line position to the previous line position. Unfortunately this still draws a new line every frame that is not necessarily joined to the line from the previous frame.

float x = random(width);
float y = random(height);
float x2 = random(width);
float y2 = random(height);

void setup()
{
size(400, 400);
strokeWeight(10);
smooth();
background(100);
frameRate(12);
}

void draw()
{
x += int(random(width*2)-width);
y += int(random(height*2)-height);
x2 += int(random(width*2)-width);
y2 += int(random(height*2)-height);
stroke(random(255));
line(x, y, x2, y2);
}

Walking lines are effectively randomly generated paths that the lines follow to create a random image.

I have been using Processing for this particular focus. Thus far I have random lines, however they don’t follow any particular path as of yet. This is effectively still walking lines as the position of the lines is added to and built on. Unfortunately this actually draws a new line every frame that is not necessarily joined to the line from the previous frame.

Have a look at it online here.

The code used to do this is extremely simple and just involves a few random variables based on the width and height of the canvas. I don’t believe this is entirely stochastic because the randoms are tightly controlled, but it is random within its boundaries (unless you want to get into the nitty gritty of it, which I don’t).

The source code is available after the jump.

void setup()
{
size(400, 400);
strokeWeight(10);
smooth();
background(0);
}

void draw()
{
float r = random(0, height);
float x = random(0, width);
stroke(r, 100);
line(random(r), random(width), random(x*2), random(height));
}

I have mostly fixed this now, it does still occasionally happen but not too often and in general it responds to all audio better than it did before. It is more sensitive overall.

There are some changes to the rotations as well so now there is more variation in how the lines appear, including the illusion of a horizon, so there is a lot more randomness to it.

I have also gone through and added in notes to make it easier to figure out what does what later on, also so anyone looking at it can figure it out relatively easily.

The Quartz Composer file, responsive_lines_2.qtz is available here.

It requires input via the built in mic in Macbook’s or iMac’s. It can of course be changed to accept audio input from the microphone-in, or line-in on Mac Pro’s or Mac Mini’s. With the right plug-in, it can also accept input from iTunes.

The most ideal is via line-in or microphone in as this gives the most control over what exactly the graphics are being synchronised with. However, using the built-in microphone means that no extra accessories are required.

Note that this was done using  Quartz Composer in Leopard.

The lines turned out to be a bit harder to work with than the particles since there are quite a few more things to control, but they did produce an interesting result.

The Quartz Composer file, responsive_lines.qtz is available online here.

Note that this one involves a lot more flashing than the other’s, so if you are epileptic or just don’t handle rapid flashing colours very well, you are looking at it at your own risk.