Assessment and rehabilitation of cognitive skills using virtual reality

Welcome to the homepage of the masters thesis project of Sacha Panic. The thesis focuses on the assessment and rehabilitation of cognitive skills, by using an augmented reality and affect adaptive game based training environment.

Project collaborators

This thesis includes project work done at ETH Zurich [1]

Prof. Riener's lab [2], head of the Cognimat [3] project

The IDEA league [4] provided financial support for my stay abroad

Thesis position

The thesis position can be summarized as follows:

1. Neurocognitive rehabilitation is possible to a higher extent than previously assumed, which can be explained by constructs such as brain plasticity and cognitive reserve.
2. Therapeutic exercises can be made more motivating for the patient, by using game design criteria from affective gaming.
3. Due to the aging population, in the near future more people will need cognitive (re)habilitation than have access to, or can be treated in the existing clinics. Telerehabilitation can be used to provide therapy while the patients are situated in their home environments. In this way a larger part of the (patient) population can be rehabilitated, starting prior to or during the preclinical stages of their disease.
4. Human factors play a significant and undervalued role when designing technology. Technology and the related interaction mechanisms should be designed to allow natural and embodied interaction as much as possible, to reduce the chance that it will be rejected by that part of the targeted population that is not very technologically savvy.

Short introduction & experimental design

Although not widely used yet, virtual reality (VR) has been shown to have useful applications in health care. For example, the Delft University of Technology and the University of Amsterdam have successfully used VR to treat phobias as part of their joint VRET project. The assessment and rehabilitation of cognitive skills is another area where VR shows potential. Some cognitive impairments appear to be shared among conditions such as Alzheimer’s and Parkinson’s disease, or dementia.I have developed a thesis and research proposal which would investigate this use of VR. The research would be partially conducted at the Sensory Motor Systems Lab at the ETH in Zurich in coordination with the head of the lab, Prof. Riener. The Sensory Motor Systems Lab has experience in using VR in clinical rehabilitation, and this cooperation would allow me to conduct a clinical trial as part of my master's thesis work.

The research is centered around the creation of a virtual environment which can be used in the clinical assessment of an Alzheimer's patient, as well as in support of the (tele)rehabilitation of the patient's cognitive skills.

The experiment is designed to find answers to the following research questions:

1. Does the use of augmented reality based training influence the rate of learning and motivation on a commonly used instrument of cognitive performance, when compared to a standard computer based training?
2. Does the use of affect-adaptive training influence the rate of learning and motivation on a commonly used instrument of cognitive performance, when compared to a standard computer based training and/or augmented reality based training?
3. Does the use of human computer interaction devices which support more natural and embodied interaction lead to a higher rate of acceptance by the targeted population? More specifically, will the input and output devices that are used be acceptable to and usable by the entire targeted population?

The cognitive performance of the participants will be assessed using a standard instrument such as the mental rotation task, a visuo-spatial skill of which can degrade due to the typical aging process, neurodegenerative diseases and traumatic brain injury. The participants will be divided into three groups. All participants will be trained and assessed on their performance on the mental rotation task. The modes of training differ between the three groups: the control group will train using a standard computer based training of the mental rotation task, the second group will receive an augmented reality based version of the standard computer based training. The third and last group will be trained on the task using an affect-adaptive game. From all participants motivational feedback on the training round, as well as task performance during an assessment round, is recorded and statistically analyzed.

Project status

A draft version of the literature research is available for download from the next section on this page. From December 7th 2009 until March 31st 2010 I will be at the ETH in Zurich working on the project.

As of February 1st 2010, with the help of members of the SMS lab I have written and submitted a proposal for two studies to the ethics committe: one study with healthy participants to calibrate the software, and a followup study with participants diagnosed with mild cognitive impairments due to Alzheimer's Disease. See the next section for more info about the proposal.

In the following sections you can find information about the current state of the rehabilitation technology that I have developed for this project. It is well underway and I expect to finish the development at the planned date (mid March 2010), in time for a pilot study at the ETH.

Downloads

November 10th, 2009 - Literature research (draft): [5] A draft version of the literature research into the topics 'cognition and neurocognitive rehabilitation', 'virtual reality based neurocognitive rehabilitation' and 'affect, learning and games'

December 7th, 2009 - Intermediate project presentation: [6] These powerpoint slides were used during an intermediate presentation of the project and its progress so far. The presentation will be given at the ETH on December 7th, and has also been given in a lecture Man Machine Systems at the DUT (3mE). The powerpoint contains clickable links to introductory papers that were published on related topics, as well as additional (requested) background information (e.g. presentations on TED and Discovery channel about brain plasticity and cognitive reserve)

February 1st, 2010 - Study proposal for the ETH ethics committee: [7] In Switzerland it is prohibited by law to conduct experiments using human participants without an approval by an ethics committee. This is regardless of the medical status of the participants (healthy or diagnosed with a medical condition). In the Netherlands there is something similar: the 'wet medisch wetenschappelijk onderzoek' [8], see also the website of 'Centrale Commissie Mensgebonden Onderzoek' [9].

This proposal details the study goals, design and protocol to be carried out by the project collaborators from the SMS lab. In this study the protocol, questionnaires and software used will be verified and calibrated using healthy participants (age 45-70). The study is expected to take place from approximately March 22nd to June 7th 2010. However this depends on approval of the proposal by the ETH ethics committee. This study will then be repeated in a followup (clinical) study with participants diagnosed with mild cognitive impairments due to Alzheimer's Disease. However since the followup study targets the clinical population, this will require approval by another (kantonal or regional) ethics committee.

February 5th, 2010 - Presentation of designed study and software development work in progress: [10] In this second progress presentation the focus is on showing the software as it has been developed so far, and obtain feedback on it.

CogniPotent prototype technology

This project will be developed using open source software which helps with optimizing the workflow of creating VR and AR based applications. The virtual training and assessment environment will be presented to the user using a consumer grade computer attached to a TV screen, while input and tactile output is also possible using a WiiMote. The orientation and position of a user's head will also be tracked (first demonstrated in the infamous video of a student at Carnegie Mellon University [11]). This creates a novel setup in which any TV is transformed into a holographic surface, and can act as a catalyst for user motivation and engagement with any virtual environment that is presented.

Please note that all the text and iconography displayed in the screenshots below, are meant as placeholders only and subjected to change in the final version of the software. The main features of the software which still need some lovin' (ie: are not finished yet) are the affective gaming based training modes. These will be completed by mid March 2010.

The mental rotation task as an assessment and rehabilitation method of short term memory as well as visuo-spatial abilities. Stimuli of varying complexity can be automatically generated by a parametrized L-System based algorithm	With this application you can edit the Mental Rotation Task stimuli which are used during a task training or assessment session.	Loading screen.	Main menu screen. From here you can select to engage in task training in one of the available modes, or engage in a task assessment.
Each of the training and assessment modes is introduced with visual and auditory text.	The way in which the stimuli are presented in the assessment mode, is similar to Shepard and Metzler's original method.	In the affective training mode, the simple task itself is embedded in a game which can be played with different strategies. It is expected that this makes the training more engaging to the patient.	The affective training mode provides more visual feedback as clues to the spatial structure of the stimuli. Here you get an idea of how stimuli are animated.
By physically moving in front of the TV or projection screen, the patient can have a look at the stimulus from different viewpoints.	This screen lists an overview of the patients accomplishments. It will show the highscores that were achieved for the particular training mode, as well as medals earned by playing the game.	This screen allows the presentation software to be configured, mainly the Nintendo Wii remote controllers used as a gamepad and for headtracking, and the sound volumes used during the training and assessment modes. For those unable to use a gamepad, this software can be operated using a mouse and/or keyboard.	This is me, hard at work ;-) at the Sensory Motor Systems lab at the ETH.

The software also records performance related data, and outputs this in several file formats (XML, CSV) which are understandable by statistical analysis software. See for instance this example [12].

Here is a video which shows the technology in development: <anyweb>http://www.youtube.com/v/_JeNFnKCa1s</anyweb>

Involved Members

• Sacha Panic