, ,

Shifumi gesture recognition

Minority report: from fiction to (almost) reality!  

Gesture-based device control can bring many benefits, providing either a better user experience or supporting touchless applications for sanitary reasons. For demonstration purposes, we have created 3 classes to distinguish several hand poses, but the model can be trained with any gestures providing a wide range of new features to the final user.  

NanoEdge AI Studio supports the Time-of-Flight sensor, but this application can be addressed with other sensors, such as radar and more. 

Approach

– We used a Time-of-Flight sensor rather than a camera for smaller signals, simpler information. 
– We set the detection distance to 20 cm to reduce the influence of the background. Optional: binarizing the distance measured.
– We took 10 measures (frequency: 15Hz) and for each measure, we predicted a class. We then chose the class that appeared the most often.
– (Concatenating measures to create a longer signal is performed to study the evolution of a movement. Here, our goal was to classify a sign. No temporality is needed).
– We created a dataset with 3,000 records per class (rock, paper, scissors), avoiding empty measurement (no motion). 
– Finally, we created an ‘N-Class classification’ model (3 classes) in NanoEdge AI Studio and tested it live on a NUCLEO_F401RE

Sensor

Time-of-Flight sensor: VL53L5 

Data

3 classes of data: Rock, paper, scissors
Signal length: 64, successive 8×8 matrixes
Data rate: 15 Hz

Results

3 classes classification:
99.37% accuracy, 0.6 KB RAM, 192.2 KB Flash

Green points represent well classified signals. Red points represent misclassified signals. The classes are on the abscissa and the confidence of the prediction is on the ordinate