Sadly by Your Side isn’t just an album–it’s an album, a book, and a smartphone app.

An Album You Can Remix With Your Smartphone’s Camera

Sadly by Your Side” isn’t just an album–it’s an album, a book, and a smartphone app. But those last two bits aren’t just thrown in as a bit of supplementary multimedia; they’re integral to the music itself. Open the app, point your phone at the book’s pages, and you’ll hear the tracks as they were originally composed. Point it anywhere else and you’ll hear your own unique remixes of those same songs, with the tracks reworked dynamically depending on your surroundings.

The app was developed by Italian interaction designer Angelo Semeraro as part of a broader exploration of digital music in today’s technological environment. “I started thinking how an album can be expressed through an app and how mobile devices can augment the listening experience,” Semeraro says. “Mobile devices allow people to listen to music in many different places, so I thought that maybe the music can adapt to these places and give a personal listening experience.”

Semeraro got Davide Cairo, the musician behind the album, to separate his woozy electronic tracks out into three layers: rhythm, melody, and harmony. From this data, the team’s graphic designers, Matteo Di Iorio and Claudio Fabbro, developed the visual language of the album–abstract smatterings of red, blue, and black forms derived from those three core sonic components. Semeraro expanded on the idea for the app, developing an algorithm that counts the red, black, and blue pixels in whatever image the camera happens to be looking at and then remixes the music’s layers accordingly, applying dollops of distortion and delay to further transform the sound.

Imagine a track that amps up its tempo based on how fast you’re physically moving–building in intensity with a jog.

It makes for a unique listening experience, to be sure–assuming you deviate from the book’s pages even a little bit, the album you’re hearing will be distinctly your own. But the project’s exploring some interesting territory in terms of music making, too. As the idea came together, the team worked together carefully to figure out a product that would allow for variation without pulling Cairo’s musical vision too far afield in one direction or another. The songs had to be built, in other words, with a very specific bit of wiggle room in mind. As Semeraro puts it, that challenge of “developing a system that allows the listener to participate in the production of the music” actually “challenges preconceptions about what it means to compose” in the first place.

But that takes us down an interesting path. You can think of a future where a “remix” isn’t a discrete reworking by a single artist but a more fluid type of sonic transformation that could happen depending on how you were listening to a song. Imagine a track that amps up its tempo based on how fast you’re physically moving–building in intensity, say, as you push yourself harder on a jog, or adjusting itself slightly so down beats always came with every plodding step you took as you walked home from work. In the same vein, though a notch or two less experimental, you could think of a playlist that adapted to the weather, cuing up something dramatic for a thunderstorm, or even a Kinect-connected jukebox that played Tom Waits when you were by yourself but put on Tom Tom Club when there were enough people in the room for a party.

Of course, in reality, it would be absurd to tie the sound of “Eleanor Rigby”–or the next “Eleanor Rigby”–to the intensity of a workout. But that doesn’t mean we can’t give it a shot with Flo-Rida.

[Hat tip: Creative Applications]