Page 1 of 1

Augumented Reality with VR Smartphone Headsets for Aerodrums

Posted: Sat Nov 21, 2015 4:24 pm
by rebit
Image

Hello, i have an idea today. This is not a product, its just a brainstorm. im not affliated with Aerodrums company.

Im starting to develop a Android and IOS app that put a Virtual 3D Drum in front of the Drummer using Google CardBoard or any other VR Headset for Smartphones with a ChromaKey Shader. The system will use the smartphone camera to not block the real enviroment, and the drummer will see a Virtual 3D Drum that will react with the animations of each piece synced with MIDI signal ( without Lag ).

Image

I don't have the 'aerodrums kit' yet ($$$), but i make some tests with my ipad with a MIDI Pad app send the midi notes to PC, so simulating the Aerodrums MIDI Output. I get the default aerodrums Midi notes table from the website documentation.

The software will not be intrusive and will not run on Windows, its just for Smartphones. On Windows or Mac (side-by-syde with Aerodrums app) will run a small 'MIDI Sync to Android' service that will working by Wifi.

Image

Here somes videos with a similar idea:

https://www.youtube.com/watch?v=mcc6Lb6ndm0

and other video: https://www.youtube.com/watch?v=P0hDq2uoYI4

If you like this project, you are good with Unity 3D , C++ and want to help, here i will make the detailed piece of software that im using to build:

Software List and assets to build:

- Unity 3D 5.2 32 bits version ( 32bits is important ) ( free ) -http://unity3d.com/ ( The Game Engine )
- KUDAN SDK for Unity ( download the demos too ) ( free ) - https://www.kudan.eu/ ( The AR SDK to track a position of 3D Virtual Drums )
- Keijiro Takahashi MIDI Receiver for Unity ( free and open ) - https://github.com/keijiro/unity-midi-receiver-test ( A Wrapper to get all MIDI Output on Windows or MacOS inside Unity 3D application )
- OVRVision SDK for Unity ( free ), just to get awesome Chromakey shader for Free --> http://dev.ovrvision.com/doc_en/index.php?downloads ( just for the shader as i said ).
- A simple 3D Model Drumkit from Trimbel Sketchup ( free ) - https://3dwarehouse.sketchup.com/model. ... 21be68f057 ( the 3d model )
- Network Game Lobby for Unity, or any other Network project template for Unity - https://www.assetstore.unity3d.com/en/#!/content/41836 (this item is necessary to sent trigged data (peace of drumkit) over network to Android or IOS Smartphone, The MIDI-receiver package from Keijiro doens't work with Android and IOS, so we need to send only the action (triggered by MIDI note) we wish to animate the drumkit on smartphone. We need to build two applications, one for Windows PC, and other to Smartphone, but this is simple to do with this templates.)
- UCLA Wireframe Shader (Free), this shader is nice to setup a wireframe 3D Drums, a nice style - https://www.assetstore.unity3d.com/en/#!/content/21897

Done.

Overview:

Image

Wished features:

- You can see you the real enviroment and block the light circle with a 3D object in front.
- You see your hands and drumsticks in front of the 3D Drums;
- 3D Drumkit pieces are animated with realtime reaction ( no latency );
- Embedded Music player with a no-touch system to select instrumental musics with no-drums track moving only your head (?!);

So what do you think ? I appreciate any support.

I will publish the progress of development here.

21/11/2015 - First test: Chromakey Shader with KUDAN SDK, the objetive is change hue, saturation and other settings to get only the arms and drumsticks differs color from the ground of jam room.

Image

21/11/2015 - Testing MIDI Notes from Aerodrums ( values from the official table, see MIDI Map section ). This Unity Scene have 30 spheres, each one have script with the respective MIDI NOTE, when i hit a note from my MIDI Controller (ipad 2) an action can be done, like make animations, transitions, enable or disable , etc.

Soon as possible i will publish to download this 'test notes latency application', to use with aerodrums.

Image

Re: Augumented Reality with VR Cardboard for Aerodrums

Posted: Wed Nov 25, 2015 12:24 am
by Richard
You have some great ideas there. Thanks for sharing them. We're not ready just yet to discuss how our own ideas relate to these but we have sent you an email with some more info.

Re: Augumented Reality with VR Cardboard for Aerodrums

Posted: Wed Nov 25, 2015 10:22 pm
by rebit
Thanks Richard.

UPDATED !!!!!!!! " KUDAN SDK DONT NEED MARKS" , Ignore this post.

Im starting today to develop. Yesterday my LG G3 smartphone broken ( Android OS crash ), i sent to support and will return in 3 or 4 days... :/

The best sample to use Vuforia is 'Smart Terrain' demo. Its perfect to work for Virtual Drums.

Here a Video Tutorial.

https://www.youtube.com/watch?v=4SxkzsH0yFs

I will use this simple AR Code Marker to easy be visible by Vuforia Video Texture ARCamera component for Unity 3D. Will be printed into a 'A4' Paper.

Image

Printed:

Image

Preview of position on the floor.

Image

Re: Augumented Reality with VR Cardboard for Aerodrums

Posted: Thu Nov 26, 2015 2:58 am
by rebit
First Preview of the Project with Drums with Material Shader.

Image

Re: Augumented Reality with VR Cardboard for Aerodrums

Posted: Thu Nov 26, 2015 3:11 am
by rebit
Wireframe Option.... ( maybe mono color ? )

Powered by UCLA Wireframe Shader: https://www.assetstore.unity3d.com/en/#!/content/21897

Animated .gif

Image

Re: Augumented Reality with VR Cardboard for Aerodrums

Posted: Thu Nov 26, 2015 3:18 am
by rebit
Option 02 - Shaded Model ( non pbr textures ).

Animated .gif:

Image

soon more news...

Re: Augumented Reality with VR Smartphone Headsets for Aerod

Posted: Thu Nov 26, 2015 4:28 am
by rebit
Off-topic of development.

One system that could be awesome to develop the 3D Virtual drum for aerodrums is 'AR with SLAM' method.

SLAM don't require 'markers' and is amazing. It' s markless. Take a look:

https://www.youtube.com/watch?v=5I5pbSs-yrU
This demo is from 'Metaio SDK' no longer available for purchade... :( But exists others SDK with SLAM option, but only one for free = Kudan SDK).

What is SLAM method ? https://en.wikipedia.org/wiki/Simultane ... nd_mapping

Kudan AR with SLAM ( Free ) : https://www.kudan.eu/ ( new, recent development, have slam but no tuto and demo for Unity 3D for markless yet, so lets wait and develop the first beta with Vuforia SDK with MARKS)

Watch 0:45 sec. https://www.youtube.com/watch?time_cont ... Rt4L2MyHnM

Re: Augumented Reality with VR Smartphone Headsets for Aerod

Posted: Thu Nov 26, 2015 9:35 am
by rebit
26-11 - Development diary -- TEST NOTES APPLICATION RELEASED !

Today i finished a 'Test Notes' application.

This is a simple application that test the performance of the Unity 3D application with MIDI Receiver code from Keijiro to receive MIDI NOTES from AERODRUMS application.

I dont have Aerodrums yet, ($$$). If you have the Aerodrums software and want to test for me, please share if it works fines.

Download here: https://mega.nz/#!ZQgDhIoC!T-gjUTS48KVj ... Eap3ZEHW84 15,7 Mb ( zip file).

How to install: Decompress into a folder and run the 'TEST-NOTES.exe'.

How it works ?

When you hit the snare, cymbal, crash etc... the respective note will blink with a ball on the screen of application, and this should be done with no latency !

Screenshot 1:

Image

Screenshot 2:

Image

How i make my test without aerodrums ?

I used LoopMIDI and DSWifiMIDI to create a Virtual INPUT MIDI Device on my Windows and with ipad 2 i simulate the Aerodrums with S1MIDITriggerFree IOS app, i changed the notes of each pad on S1MIDITriggerFree to respective official MIDI notes table of DrumKit from Aerodrums. You dont need this apps to test the application, you just need the aerodrums software and my test application.

Image

S1MIDITriggerFree for IPAD2: https://itunes.apple.com/br/app/s1midit ... 22995?mt=8

Image

Re: Augumented Reality with VR Smartphone Headsets for Aerod

Posted: Wed Mar 09, 2016 6:50 pm
by rebit
Nce set of PBR 3D Drums for Virtual Reality application or Project Mapping

https://www.assetstore.unity3d.com/#!/content/20066

3D View of Drum kit on SketchFab

https://sketchfab.com/models/6459450cca ... 44b80041a6

Image