Transcribing drums into sheet music on the fly

As we were experimenting with our motion capture drumming system, we had the idea of testing it out as the controller for a rhythm game. Games like Rock Band and Guitar Hero show the notes to be played as coloured symbols on a vertically scrolling pane. Our idea was to instead use actual music notation to represent the notes. In other words, we wanted to recreate the experience of sight reading a drum transcription and hopefully make a didactic tool in the process.

The following video shows a rough prototype of our game where the top staff line shows what should be played, the bottom line shows what is being played and each hit is scored based on its accuracy. Notice how the music notation is generated on the fly and in realtime based on what the drummer is doing.

Playing around with this system led to a revelation, the idea of adapting live transcription into a tool that drummers could use to edit music notation in the simplest way imaginable. Unlike other editors, ours would work by letting the user place notes on a virtual timeline and, from that, the notation would be entirely generated automatically. After two years of development we will be releasing the result, Aered, on this website very soon. How it works and what sets it apart from existing software will be the topics of the next batch of posts.