Blendnik is a system Nick Porcaro and Ellen Levy have been developing for the purpose of making interactive digital pieces based on our free form jam sessions where she paints.
In the beginning, we were just getting together to do jam sessions where Nick would play the piano and Ellen would paint, usually on vellum paper with pastel, acrylic and ink. Here is what a typical session may have resulted in:
Improv piano tracks by Nick:
The first thing we tried doing was making a movie in AfterEffects. The result of that experiment was called “Buldgo”, a abstract two minute movie. The sound for Buldgo was done by improvising with some Max/MSP patches.
AfterEffects had some 3D capabilities (that was used at the end of “Buldgo”) but they were limited. Around this time we became aware of Blender and the Elephant’s Dream movie. So we decided Blender was worth giving a try.
Next we tried making an accurate 3D model of Buldgo, which illustrated how hard it was to convert 2D images into 3D models. We learned a bunch about Blender during this time, but were not too happy with this result.
This was lots of hard work, so we backed off and looked for a more enjoyable process. We wanted to try something more interactive. So we looked into using Max/MSP and Jitter (not open source) and Pure Data/GEM. These systems work very well but are oriented towards generating graphics and manipulating QuickTime movies. An extensive knowledge of OpenGL is required for complex operations, and they are not full featured animation systems like Blender.
We also spent about a month studying Maya, but decided it was too expensive.
This led us to experiment with the Blender Game Engine, which had the advantage of being integrated into Blender, but the disadvantage of being rather primitive compared to other game engines. Also, a “game engine” is not necessarily the best fit for interactive art, which has a different orientation. Nonetheless, I was impressed by the Blender community and I figured if worse came to worse I could modify the Blender Game Engine myself, since it is open source.
Soon after this the Yo Frankie! game and the Blender GameKit 2 came out, so I started having more confidence that the Blender Game engine had a future.
At first we had a simple Python script in Blender talking through OSC to Pure Data, then it developed into complex system that was oriented towards designing the scene in Blender and having Pd be a dumb back end. We wrote a paper that describes this and presented it at the Pd conference in Brasil.
While writing this paper, I used the SynthBuilder Electric Guitar Physical Model that I ported to the STK into the Blendnik Pd patch.
This worked relatively easily and it completely changed my mind about how this system should work – the semantics of the mappings should take place in Pd not Blender.
This has the advantage of allowing for arbitrary mappings and a far more flexible Pd patch, but the disadvantage is one had to be both an expert in Pd and Blender to use the system instead of just Pd. I decided that it was worth the trouble to get really good in Pd anyway, so I charged ahead with a major design change, which resulted in a far simpler Python script in Blender. I also wrote a custom C++ external for Pd to simplify the OSC communications.