VRET Archi

The goal of the system architecture is to provide an infrastructure for VRET related projects so that we

• provide the therapist with basic feedback like video, sound, heart rate
• provide programmers with a common approach and tools to handle remote therapy
• enable re-use of commonly required components
• easy switching between various therapies

The figure below shows the first proposal we arrived at after significant discussions with Willem-Paul, Niels, Christian, Guntur, and Daniel.

In the below sections we discuss the VRET hardware and network mechanism, and the modules shown in the figure. The colors in the picture match components in Therapist and Patient system to parts in the Database.

Hardware

Check VRET_Hardware for details on the VR machine

Check Video_Hardware for details on the video streaming hardware

Check VRET_physiological for details on physiological measurement hardware

Network Mechanism

To handle communication between the patient monitoring system and the patient computers, we currently use plain sockets. While fast and simple, this requires all firewalls on the route between server and client to be open. Usually these ports will not be open except inside a single domain. Opening them will require negotiations with the network security staff and seems better avoided if possible.

I discussed this issue with Ruud, Tim Verwaart, and a number of people in the computer graphics group. Our safest bet seems to be using the HTTP ports instead of just arbitrary sockets.

We have the following requirements for the connection

• server (the side handling the call) needs to be notified when the network connection gets disabled.
• some people in the project asked if it's possible to call functions directly instead of building large if(received="commandX") then .... loops. RPC would enable somethign like that. That will mostly replace the if-then constructs with "def commandX()" constructs so I do not belief that it will save much space.
• Support for "channel coding": putting objects into the stream and re-building an object out of it at the other end of the channel. In Java this is trivial by extending "serializable" but in Python apparently it is not. An XML library is just one of many solutions for that. XML blows up data easily by a factor 6 and requires extra parsing causing an order of magnitude slowdown [ File:Xmlrpc-perf.pdf ]). For small transactions there is not much difference and in our case XML seems acceptable as transfer format.
• Many python RPC servers on a single computer have to be able to use the same port at the same time. So the addressing seems to require more than just the computer name and port number.

One possibility is xml-rpc. A number of people in the graphics group working often with this do recommend xml-rpc. Maybe it's possible to replace the serve_forever() call with something else?

Update 12aug. Discussed further with Ruud. There is an issue with using HTTP ports as well. Ruud now distinguishes "incoming" and "outgoing" ports. Outgoing (the 'client') usually is not restricted in choice of port number but incoming (the 'server') is. According to Ruud (if I understand this right, he is a bit unsure it seems) ALL outgoing ports have to be opened explicitly, even HTTP ports. That means we can not avoid security discussions anyway.

Therapist System

This section discusses the modules inside the system used by the therapist (see Figure).

The therapist system contains a number of "Monitors" and a Telephone.

Telephone

The telephone is a plain old telephone. It should be completely independent from the computer network. This telephone is needed in case the computer system (possibly its network system) breaks down and the therapist needs to call a stand-by nurse on location.

Monitor

Each Monitor system gives the therapist a view on the actual status of the patient and the progress on his therapy. It may physically consist of multiple computer monitors and also other apparatus to support the therapist.

In the Monitor system we can distinguish the General and the Customized interface.

General interface

The general interface is shown as the "TV" (therapist view) in the picture. It shows a number of panels/widgets common to all therapies:

• The room view widget: this is a panel showing a live video stream giving an overview of the room. The patient should be well visible so that the therapist can watch his physical well being

• The VR view widget: this is a panel showing a live video stream of what the patient sees and hears inside the VR helmet. To ensure accuracy this stream directly taps from the video and audio feeds to the helmet.

• The Physical Status widget: this widget shows the physical status of the patient, possibly in a time=line style matching the output of the scenario widget.

• Scenario widget: this widget shows the progress of the scenario along a time line.

Customized interface

The therapist is provided with a customized interface to control the current therapy. This is because every therapy requires varying specific parameters in worlds selected especially for that therapy.

Widgets

Widgets are ready-to-use panels that can be placed inside a GUI to add functionality.

Scenario Widget

This widget shows the current state of the scenario. The plan is to show a timeline with markers representing steps in the scenarioetc. The physical status widget should combine neatly with it

Physical Status widget

This widget shows the current scaredness and maybe details like heart rate, sweat, SUD reports etc.

Patient System

This section discusses the modules inside the Patient system (Figure).

The main part is the Virtual Therapist. This is an automatic computer therapist "physically" located at the patients location. "physically located" means that the therapist can keep contact with the patient even in the case of a network failure.

The most important job of the Virtual Therapist is to take over control in case the physical therapist appears not to respond properly to changing patient's behaviour, particularly when the patient gets too scared. If possible the virtual therapist will contact the real therapist to ask permission to take over. The details of this take-over process are adjustable via the "VT scaredness settings" in the Database. The real therapist can edit these settings via the "VT script editor" widget.

Scaredness Analyser

The job of this module is to estimate the patient's current scaredness. It can base this estimation on the patient's heart rate, sweat rate, and estimated SUD scores.

speech analysis SUD score

SUD score is a measure for the scaredness of the patient. This module tries to recognise the patient's own oral reports on his SUD score. The therapist can help this module via the SUD scoring widget.

Database

This section discusses the parts inside the database. The database should not be confused with the database objects of the dialog manager. Actually that database objects are part of the database.