Feasibility study to investigate eHealth tools that facilitate communication between practitioners and clients via chat and video calling, enriched with emotion recognition algorithms.

Outcome-driven Learning Infrastructure for Multimodal Affect Recognition

Apr 2018 – Feb 2019

Background

Mood status is an important predictor for various mental health problems. In depression, for example, people experience long periods with a negative mood and state of mind, and in bipolar disorders, there are strong changes between positive and negative states. It is often very difficult for the client to objectively determine how his or her state of mind changes over time. The practitioner sees this better in theory, but in practice often sees the client in snapshots, which can further distort the image. Recent developments in the field of eHealth make it possible to communicate virtually, but this eliminates certain non-verbal elements, making it more difficult for the practitioner to estimate how a client is really doing. By making more frequent measurements and using additional user-generated data, we can provide a better picture of a person’s state of mind and how it changes over time.

Objective

OLIMAR is a feasibility study in order to investigate modular systems and tools that facilitate communication between practitioner and client via chat and video calling, enriched with technical algorithms that determine the client’s state of mind. In this way a practitioner also gains more insight into how his client is doing with digital communication. In this project three specific modules for emotion recognition are examined, namely emotion recognition from speech, text, and facial expression.

Results

A software prototype was implemented in order to evaluate various open source emotion recognition technologies for different media streams (text, audio and video) and adapt them to content available during eHealth interventions.  In conclusion, the benefits of bringing sentiment analysis on for mental healthcare can have major improvements on the detection of high risk situations along with the other circumstances such as relapse prevention, and user mental state improvement or decline.