Gesture recognition by pattern matching using sensor fusion on an internet of things device

Details

Supervisors

Van Roy, Peter

Faculty

Ecole polytechnique de Louvain

Degree label

Master [120] en sciences informatiques, à finalité spécialisée

Abstract

Internet of Things devices are small computers with embedded hardware and minimal software. They are increasingly present in everyday life (home automation, household) and can be equipped with many sensors to collect data or have a small motor to perform a physical movement. But as they are present in the human environment, interaction with them should be simple and reliable. Our goal is to recognize a human gesture, using only an IoT device attached to a wrist. No cloud or other dependencies because of the many disadvantages. This gesture can then be used to control another device. The IoT we will use is a GRiSP, it is a prototype board for IoT that supports Erlang and Elixir on bare hardware. It has sockets for attaching up to 5 PMOD sensors and actuators. PMOD devices include navigation, barometer, humidity, range finding, motor control, etc. The approach is to program an IoT capable of learning and recognizing gestures. A gesture is a movement of a human arm in the environment. We will combine data from sensors passed through a filter. Then recognize the different gestures. It is the fusion of 3 sensors from the PMOD-NAV: an accelerometer, a gyroscope, and a magnetometer. Sent to a Kalman Filter, it estimates the state of a system from measurements. But it is robust against incomplete data and noise. Return an acceleration in real-time, and sent it to the machine learning algorithm to recognize the gesture. A gesture is recognized with a classifier by pattern matching. Each gesture is a different category. There is an initial list of learned gestures, and more can be added to diversify possibilities. Several gestures can be detected in succession, but the user must stop moving between 2 gestures.