Augumented Reality with VR Smartphone Headsets for Aerodrums
Posted: Sat Nov 21, 2015 4:24 pm
Hello, i have an idea today. This is not a product, its just a brainstorm. im not affliated with Aerodrums company.
Im starting to develop a Android and IOS app that put a Virtual 3D Drum in front of the Drummer using Google CardBoard or any other VR Headset for Smartphones with a ChromaKey Shader. The system will use the smartphone camera to not block the real enviroment, and the drummer will see a Virtual 3D Drum that will react with the animations of each piece synced with MIDI signal ( without Lag ).
I don't have the 'aerodrums kit' yet ($$$), but i make some tests with my ipad with a MIDI Pad app send the midi notes to PC, so simulating the Aerodrums MIDI Output. I get the default aerodrums Midi notes table from the website documentation.
The software will not be intrusive and will not run on Windows, its just for Smartphones. On Windows or Mac (side-by-syde with Aerodrums app) will run a small 'MIDI Sync to Android' service that will working by Wifi.
Here somes videos with a similar idea:
https://www.youtube.com/watch?v=mcc6Lb6ndm0
and other video: https://www.youtube.com/watch?v=P0hDq2uoYI4
If you like this project, you are good with Unity 3D , C++ and want to help, here i will make the detailed piece of software that im using to build:
Software List and assets to build:
- Unity 3D 5.2 32 bits version ( 32bits is important ) ( free ) -http://unity3d.com/ ( The Game Engine )
- KUDAN SDK for Unity ( download the demos too ) ( free ) - https://www.kudan.eu/ ( The AR SDK to track a position of 3D Virtual Drums )
- Keijiro Takahashi MIDI Receiver for Unity ( free and open ) - https://github.com/keijiro/unity-midi-receiver-test ( A Wrapper to get all MIDI Output on Windows or MacOS inside Unity 3D application )
- OVRVision SDK for Unity ( free ), just to get awesome Chromakey shader for Free --> http://dev.ovrvision.com/doc_en/index.php?downloads ( just for the shader as i said ).
- A simple 3D Model Drumkit from Trimbel Sketchup ( free ) - https://3dwarehouse.sketchup.com/model. ... 21be68f057 ( the 3d model )
- Network Game Lobby for Unity, or any other Network project template for Unity - https://www.assetstore.unity3d.com/en/#!/content/41836 (this item is necessary to sent trigged data (peace of drumkit) over network to Android or IOS Smartphone, The MIDI-receiver package from Keijiro doens't work with Android and IOS, so we need to send only the action (triggered by MIDI note) we wish to animate the drumkit on smartphone. We need to build two applications, one for Windows PC, and other to Smartphone, but this is simple to do with this templates.)
- UCLA Wireframe Shader (Free), this shader is nice to setup a wireframe 3D Drums, a nice style - https://www.assetstore.unity3d.com/en/#!/content/21897
Done.
Overview:
Wished features:
- You can see you the real enviroment and block the light circle with a 3D object in front.
- You see your hands and drumsticks in front of the 3D Drums;
- 3D Drumkit pieces are animated with realtime reaction ( no latency );
- Embedded Music player with a no-touch system to select instrumental musics with no-drums track moving only your head (?!);
So what do you think ? I appreciate any support.
I will publish the progress of development here.
21/11/2015 - First test: Chromakey Shader with KUDAN SDK, the objetive is change hue, saturation and other settings to get only the arms and drumsticks differs color from the ground of jam room.
21/11/2015 - Testing MIDI Notes from Aerodrums ( values from the official table, see MIDI Map section ). This Unity Scene have 30 spheres, each one have script with the respective MIDI NOTE, when i hit a note from my MIDI Controller (ipad 2) an action can be done, like make animations, transitions, enable or disable , etc.
Soon as possible i will publish to download this 'test notes latency application', to use with aerodrums.
But exists others SDK with SLAM option, but only one for free = Kudan SDK).
