Leap Motion is cool and so is Android so in this post, I’ll explain how to put them together to control a very simple fighter plane in 3D space. The example 3D application was taken from Rajawali’s example and you can take a look at its source code here.
Background
Leap motion is a device that captures your hand and finger movement through its IR sensor and a camera. This translates the data into a format that can be read and processed by some of the common programming languages. Currently, the only way to read those data is through the standard USB port present in your laptops and desktop PCs and the leap motion SDK will handle all the data translation for you.
Unfortunately, in Nexus 7 and most of the Android tablet out there, we only have the micro USB to serve as an input and obviously, we cannot connect the Leap Motion device directly to the tablet (or is there a converter that can convert the standard USB to micro USB? but then we still a Leap Motion SDK that runs on Android).
Solution
Since there’s no easy way of integrating them, I thought of running a node.js server that performs:
- Receiving all the hand data coming from the leap motion device.
- Listens to port 1337 and accept incoming socket connection.
- Write the hand data to the socket connection.
Tehcnically, these are not difficult to implement specially in Node.js. Another part of the story is the Android application that:
- Connects to the Node.js server application on port 1337.
- Parse and smoothen out the data using a Low-pass filter.
- Translate that data into 3D movements.
With that, let’s begin with our implementation.
Requirements
- You need a Leap Motion device and the SDK installed
- Install LeapJS
- A copy of Rajawali 3D framework
- A copy of Rajawali Examples particularly the Rajawali ChaseCam Example
Writing our Node.js Server
Our Node.js server is very simple socket server that receives hand data from Leap Motion device and at the same time listen to any incoming socket connection on port 1337.
{%codeblock Node.js Server lang:javascript https://github.com/mharkus/LeapDroid/blob/master/leapandroidserver.js%} var Leap = require(’../lib’).Leap var net = require(‘net’);
var mSocket; var started = false; var previousFrame;
var server = net.createServer(function(socket){ mSocket = socket; started = true;
console.log("client connected!");
socket.on("disconnect", function(){
started = false;
})
}); Leap.loop({enableGestures: true}, function(frame, done) { if(started){ if(frame.hands.length > 0){ var hand = frame.hands[0]; mSocket.write(hand.rotation + “:” + hand.palmPosition.toString() + “\n”); } } done(); })
server.listen(1337, ‘192.168.0.25’); {%endcodeblock%}
Lines 1-2 imports the necessary libraries for leap motion and for setting up a socket server.
Line 9 creates a socket server.
Line 20 listens for all the frames coming from the leap motion device.
Line 22 checks if there are valid hands from the frame and process the first hand in the array.
Line 24 writes the hand rotation and position data to the socket connection.
Line 30 starts the socket server on port 1337.
Next, in the Android code, we just need to connect to the socket server through port 1337. The app waits for any incoming data from the server and translates that into a 3D movement such as rotation and translation in X, Y and Z axis.
In this case, we will use a Service to handle the server connectivity and we’ll pass the data from the Service to the 3D Renderer through Messenger class. For the implementation, I’ve used the Android Service Tutorial as a reference (well, I copied most of it :D).
{%codeblock Node.js Server lang:java https://github.com/mharkus/LeapDroid/blob/master/src/com/mlst/leapdroid/MessengerService.java%} s = new Socket(“192.168.0.25”, 1337);
BufferedReader input = new BufferedReader(new InputStreamReader(s.getInputStream())); String line = “";
while (keepMonitoring) { line = input.readLine(); if (line.contains(":")) { String[] rots = line.split(":")[0].split(”,"); String[] vals = line.split(":")[1].split(”,"); float[] inputPosition = { Float.parseFloat(vals[0]), Float.parseFloat(vals[1]), Float.parseFloat(vals[2]) }; float[] smoothVals = applyLPF(inputPosition, positionOutput);
inputPosition = new float[] {
Float.parseFloat(rots[0]),
Float.parseFloat(rots[1]),
Float.parseFloat(rots[2]) };
float[] smoothRots = applyLPF(inputPosition,
rotationOutput);
float x = smoothVals[0];
float y = smoothVals[1];
if (y < 200) {
y = 200 - y;
} else {
y = y - 200;
}
float z = smoothVals[2];
float[] obj = { x, y, z, smoothRots[0] };
mMessenger.send(Message.obtain(null, MSG_SET_VALUE, obj));
}
} {%endcodeblock%}
{%codeblock Low Pass Filter lang:java https://github.com/mharkus/LeapDroid/blob/master/src/com/mlst/leapdroid/MessengerService.java#L179%} private float[] applyLPF(float[] input, float[] output) { if (output == null) return input;
for (int i = 0; i < input.length; i++) {
output[i] = output[i] + alpha * (input[i] - output[i]);
}
return output;
}
{%endcodeblock%}
Line 3 reads any incoming data and blocks/wait if there’s none.
Lines 14 and 22 applies a low-pass filter to the data to smoothen out any noise.
Line 27 converts the Y value so that 200 will be equivalent to Y=0 in the 3D space. Anything below that will be -Y and above will be +Y. I did these so that our Jet plane can go up or down the origin.
Line 36 sends the data to the handler, which in turn, sends it to the listener in our Activity.
{%codeblock Incoming Handler lang:java https://github.com/mharkus/LeapDroid/blob/master/src/com/mlst/leapdroid/RajawaliChaseCamActivity.java#L77%} class IncomingHandler extends Handler { @Override public void handleMessage(Message msg) { switch (msg.what) { case MessengerService.MSG_SET_VALUE: float[] reading = (float[]) msg.obj; mRenderer.setValues(reading[0], reading[1], reading[2], reading[3]); break; default: super.handleMessage(msg); } } } {%endcodeblock%}
Line 7 passes the values from our service to the 3D renderer and finally, our renderer will move the 3D object accordingly.
Get the Source
You can download the source code for the Android app and the node.js server here.
Conclusion
It’s really great to see new ways to interact with our computers and I admire the companies behind Leap Motion and Myo who are brave enough to dip their toe and test the water in this field.