Difference between revisions of "VRET Archi"
(→Sony 15-CC25DNV 1/3" IR sensitive PAL color camera) |
(→Network Mechanism) |
||
| (66 intermediate revisions by 2 users not shown) | |||
| Line 5: | Line 5: | ||
* easy switching between various therapies | * easy switching between various therapies | ||
| − | The figure below shows the first proposal we arrived at after | + | The figure below shows the first proposal we arrived at after significant discussions with Willem-Paul, Niels, Christian, Guntur, and Daniel. |
| + | |||
| + | In the below sections we discuss the VRET hardware and network mechanism, and the modules shown in the figure. The colors in the picture match components in Therapist and Patient system to parts in the Database. | ||
[[Image:Vret_archi.png|thumb|500px]] | [[Image:Vret_archi.png|thumb|500px]] | ||
| + | == Download == | ||
| + | Resources are on SVN: [https://mmi.tudelft.nl/cgi-bin/vret_oud.cgi/browser/Tools Svn tools]. | ||
| + | |||
| + | == Therapist Interface == | ||
| + | [[Therapist Interface discussion]] | ||
== Hardware == | == Hardware == | ||
| + | Check [[VRET_Hardware]] for details on the VR machine | ||
| + | |||
| + | Check [[Video_Hardware]] for details on the video streaming hardware | ||
| + | |||
| + | Check [[VRET_physiological]] for details on physiological measurement hardware | ||
| + | |||
| + | == Network Mechanism == | ||
| + | |||
| + | To handle communication between the patient monitoring system and the patient computers, we currently use [http://www.xmlrpc.com XML-RPC] which runs on sockets. While reasonably fast and simple, this requires all firewalls on the route between server and client to be open. Usually these ports will not be open except inside a single domain. Opening them will require negotiations with the network security staff and seems better avoided if possible. | ||
| + | |||
| + | I discussed this issue with Ruud, Tim Verwaart, and a number of people in the computer graphics group. Our safest bet seems to be using the HTTP ports instead of just arbitrary sockets. | ||
| + | |||
| + | We have the following requirements for the connection | ||
| + | * server (the side handling the call) needs to be notified when the network connection gets disabled. | ||
| + | * some people in the project asked if it's possible to call functions directly instead of building large <code>if(received="commandX") then ....</code> loops. RPC enables this. This way we avoid if-then constructs with "def commandX()" constructs. | ||
| + | * Support for "channel coding": putting objects into the stream and re-building an object out of it at the other end of the channel. In Java this is trivial by extending "serializable" but in Python apparently it is not. XML-RPC handles all this for us. | ||
| + | * Many python RPC servers on a single computer have to be able to use the same port at the same time. So the addressing seems to require more than just the computer name and port number. | ||
| + | * Being notified of disconnection seems possible: [[http://xmlrpc-c.sourceforge.net/doc/libxmlrpc_server_pstream++.html | usermanual XML-RPC]] | ||
| + | |||
| + | Minus points: | ||
| + | * XML blows up data easily by a factor 6 and requires extra parsing causing an order of magnitude slowdown [ [[Image:Xmlrpc-perf.pdf]] ]). For small transactions there is not much difference and in our case XML seems acceptable as transfer format. | ||
| + | |||
| + | A number of people in the graphics group working often with this do recommend xml-rpc. | ||
| + | |||
| + | |||
| + | Update 12aug. Discussed further with Ruud. There is an issue with using HTTP ports as well. Ruud now distinguishes "incoming" and "outgoing" ports. Outgoing (the 'client') usually is not restricted in choice of port number but incoming (the 'server') is. According to Ruud (if I understand this right, he is a bit unsure it seems) ALL outgoing ports have to be opened explicitly, even HTTP ports. That means we can not avoid security discussions anyway. | ||
| + | |||
| + | == Therapist System == | ||
| + | This section discusses the modules inside the system used by the therapist (see Figure). | ||
| + | |||
| + | The therapist system contains a number of "Monitors" and a Telephone. | ||
| + | |||
| + | [[Image:Therapist_archi.png|600px]] | ||
| + | |||
| + | |||
| + | === Telephone === | ||
| + | The telephone is a plain old telephone. It should be completely independent from the computer network. This telephone is needed in case the computer system (possibly its network system) breaks down and the therapist needs to call a stand-by nurse on location. | ||
| + | |||
| + | === Monitor === | ||
| + | |||
| + | Each Monitor system gives the therapist a view on the actual status of the patient and the progress on his therapy. It may physically consist of multiple computer monitors and also other apparatus to support the therapist. | ||
| + | |||
| + | In the Monitor system we can distinguish the General and the Customized interface. | ||
| + | |||
| + | |||
| + | ==== General interface ==== | ||
| + | The general interface is shown as the "TV" (therapist view) in the picture. It shows a number of panels/widgets common to all therapies: | ||
| + | |||
| + | * The room view widget: this is a panel showing a live video stream giving an overview of the room. The patient should be well visible so that the therapist can watch his physical well being | ||
| + | |||
| + | * The VR view widget: this is a panel showing a live video stream of what the patient sees and hears inside the VR helmet. To ensure accuracy this stream directly taps from the video and audio feeds to the helmet. | ||
| + | |||
| + | * The Physical Status widget: this widget shows the physical status of the patient, possibly in a time=line style matching the output of the scenario widget. | ||
| + | |||
| + | * Scenario widget: this widget shows the progress of the scenario along a time line. | ||
| + | |||
| + | ==== Customized interface ==== | ||
| + | The therapist is provided with a customized interface to control the current therapy. This is because every therapy requires varying specific parameters in worlds selected especially for that therapy. | ||
| + | |||
| + | === Widgets === | ||
| + | |||
| + | Widgets are ready-to-use panels that can be placed inside a GUI to add functionality. | ||
| + | |||
| + | ==== Scenario Widget ==== | ||
| + | This widget shows the current state of the scenario. The plan is to show a timeline with markers representing steps in the scenarioetc. The physical status widget should combine neatly with it | ||
| + | |||
| + | ==== Physical Status widget ==== | ||
| + | This widget shows the current scaredness and maybe details like heart rate, sweat, SUD reports etc. | ||
| + | |||
| + | == Patient System == | ||
| + | This section discusses the modules inside the Patient system (Figure). | ||
| − | + | The main part is the Virtual Therapist. This is an automatic computer therapist "physically" located at the patients location. "physically located" means that the therapist can keep contact with the patient even in the case of a network failure. | |
| − | + | The most important job of the Virtual Therapist is to take over control in case the physical therapist appears not to respond properly to changing patient's behaviour, particularly when the patient gets too scared. If possible the virtual therapist will contact the real therapist to ask permission to take over. The details of this take-over process are adjustable via the "VT scaredness settings" in the Database. The real therapist can edit these settings via the "VT script editor" widget. | |
| − | |||
| − | [[Image: | + | [[Image:Patient archi.png|600px]] |
| − | + | === Scaredness Analyser === | |
| − | |||
| − | |||
| − | |||
| − | + | The job of this module is to estimate the patient's current scaredness. It can base this estimation on the patient's heart rate, sweat rate, and estimated SUD scores. | |
| − | + | ==== speech analysis SUD score ==== | |
| − | + | SUD score is a measure for the scaredness of the patient. This module tries to recognise the patient's own oral reports on his SUD score. The therapist can help this module via the SUD scoring widget. | |
| − | |||
| + | === Dialog Manager === | ||
| − | + | === VR System === | |
| − | |||
| − | The | + | == Database == |
| + | This section discusses the parts inside the database. The database should not be confused with the database objects of the dialog manager. Actually that database objects are part of the database. | ||
| − | + | [[Image:Database_archi.png]] | |
| − | + | The database is not the SVN server. That is because the SVN server is ment to store all project information. I suppose we do not want therapists to get access to that. | |
Latest revision as of 07:00, 2 August 2010
The goal of the system architecture is to provide an infrastructure for VRET related projects so that we
- provide the therapist with basic feedback like video, sound, heart rate
- provide programmers with a common approach and tools to handle remote therapy
- enable re-use of commonly required components
- easy switching between various therapies
The figure below shows the first proposal we arrived at after significant discussions with Willem-Paul, Niels, Christian, Guntur, and Daniel.
In the below sections we discuss the VRET hardware and network mechanism, and the modules shown in the figure. The colors in the picture match components in Therapist and Patient system to parts in the Database.
Contents
Download
Resources are on SVN: Svn tools.
Therapist Interface
Therapist Interface discussion
Hardware
Check VRET_Hardware for details on the VR machine
Check Video_Hardware for details on the video streaming hardware
Check VRET_physiological for details on physiological measurement hardware
Network Mechanism
To handle communication between the patient monitoring system and the patient computers, we currently use XML-RPC which runs on sockets. While reasonably fast and simple, this requires all firewalls on the route between server and client to be open. Usually these ports will not be open except inside a single domain. Opening them will require negotiations with the network security staff and seems better avoided if possible.
I discussed this issue with Ruud, Tim Verwaart, and a number of people in the computer graphics group. Our safest bet seems to be using the HTTP ports instead of just arbitrary sockets.
We have the following requirements for the connection
- server (the side handling the call) needs to be notified when the network connection gets disabled.
- some people in the project asked if it's possible to call functions directly instead of building large
if(received="commandX") then ....loops. RPC enables this. This way we avoid if-then constructs with "def commandX()" constructs. - Support for "channel coding": putting objects into the stream and re-building an object out of it at the other end of the channel. In Java this is trivial by extending "serializable" but in Python apparently it is not. XML-RPC handles all this for us.
- Many python RPC servers on a single computer have to be able to use the same port at the same time. So the addressing seems to require more than just the computer name and port number.
- Being notified of disconnection seems possible: [| usermanual XML-RPC]
Minus points:
- XML blows up data easily by a factor 6 and requires extra parsing causing an order of magnitude slowdown [ File:Xmlrpc-perf.pdf ]). For small transactions there is not much difference and in our case XML seems acceptable as transfer format.
A number of people in the graphics group working often with this do recommend xml-rpc.
Update 12aug. Discussed further with Ruud. There is an issue with using HTTP ports as well. Ruud now distinguishes "incoming" and "outgoing" ports. Outgoing (the 'client') usually is not restricted in choice of port number but incoming (the 'server') is. According to Ruud (if I understand this right, he is a bit unsure it seems) ALL outgoing ports have to be opened explicitly, even HTTP ports. That means we can not avoid security discussions anyway.
Therapist System
This section discusses the modules inside the system used by the therapist (see Figure).
The therapist system contains a number of "Monitors" and a Telephone.
Telephone
The telephone is a plain old telephone. It should be completely independent from the computer network. This telephone is needed in case the computer system (possibly its network system) breaks down and the therapist needs to call a stand-by nurse on location.
Monitor
Each Monitor system gives the therapist a view on the actual status of the patient and the progress on his therapy. It may physically consist of multiple computer monitors and also other apparatus to support the therapist.
In the Monitor system we can distinguish the General and the Customized interface.
General interface
The general interface is shown as the "TV" (therapist view) in the picture. It shows a number of panels/widgets common to all therapies:
- The room view widget: this is a panel showing a live video stream giving an overview of the room. The patient should be well visible so that the therapist can watch his physical well being
- The VR view widget: this is a panel showing a live video stream of what the patient sees and hears inside the VR helmet. To ensure accuracy this stream directly taps from the video and audio feeds to the helmet.
- The Physical Status widget: this widget shows the physical status of the patient, possibly in a time=line style matching the output of the scenario widget.
- Scenario widget: this widget shows the progress of the scenario along a time line.
Customized interface
The therapist is provided with a customized interface to control the current therapy. This is because every therapy requires varying specific parameters in worlds selected especially for that therapy.
Widgets
Widgets are ready-to-use panels that can be placed inside a GUI to add functionality.
Scenario Widget
This widget shows the current state of the scenario. The plan is to show a timeline with markers representing steps in the scenarioetc. The physical status widget should combine neatly with it
Physical Status widget
This widget shows the current scaredness and maybe details like heart rate, sweat, SUD reports etc.
Patient System
This section discusses the modules inside the Patient system (Figure).
The main part is the Virtual Therapist. This is an automatic computer therapist "physically" located at the patients location. "physically located" means that the therapist can keep contact with the patient even in the case of a network failure.
The most important job of the Virtual Therapist is to take over control in case the physical therapist appears not to respond properly to changing patient's behaviour, particularly when the patient gets too scared. If possible the virtual therapist will contact the real therapist to ask permission to take over. The details of this take-over process are adjustable via the "VT scaredness settings" in the Database. The real therapist can edit these settings via the "VT script editor" widget.
Scaredness Analyser
The job of this module is to estimate the patient's current scaredness. It can base this estimation on the patient's heart rate, sweat rate, and estimated SUD scores.
speech analysis SUD score
SUD score is a measure for the scaredness of the patient. This module tries to recognise the patient's own oral reports on his SUD score. The therapist can help this module via the SUD scoring widget.
Dialog Manager
VR System
Database
This section discusses the parts inside the database. The database should not be confused with the database objects of the dialog manager. Actually that database objects are part of the database.
The database is not the SVN server. That is because the SVN server is ment to store all project information. I suppose we do not want therapists to get access to that.
