Adaptive applications with multimodal user interfaces

Details

Supervisors

Mens, Kim ; Duhoux, Benoît ; Dumas, Bruno

Faculty

Ecole polytechnique de Louvain

Degree label

Master [120] en sciences informatiques, à finalité spécialisée

Abstract

Context-oriented programming is a programming paradigm that allows an application to adapt its behaviour depending on the surrounding context rather than conventional conditions. This new mechanism allows developers to create a flexible adaptable application without having to be bottle-necked by the architecture. But how powerful would it be to mix this paradigm to the multimodal user interface, as nowadays, the most popular applications are not even running on a computer but on a small devices. This is what has been explored throughout this academic year. This master thesis shows that it is possible to integrate multimodalities in a FBCOP language. Our solution and implementation are based on the programming framework RubyCOP, written by B.Duhoux. Within this programming framework, we succeeded to integrate different new possible interactions such as vocal interactions and gesture recognition through a flexible architecture that would allow anyone who would like to add another modalities to do so smoothly. First, the voice input interaction has been implemented using the google-speech API, the gesture recognition has been possible thanks to a Leap Motion, and the voice output has been achieved through the rubyflite library. Then, after integrating the interactions within RubyCOP, we needed to challenge the solution via a study case following the FBCOP methodology. The main goal was to create this application using our solution such as it allows multimodal interaction depending on the current context. We implemented a study case which is a smart meeting application where a user can create meetings, send messages or add participants. This application adapts its interaction mode, depending on the noise level around, user condition and the availability of input or output devices. Eventually, this application has been tested using a test suite generated by an external tool written by P. Martou. This work proves that it is possible to create an implementation of FBCOP that can change vary its interactions depending on the current environment, user’s state or device’s availability.