Generic Vision Module
Registering to the VIM
The GVM sends a request via the broker to register the following VIM parameters:
- name (Id): If there is no other module already subscribed with this name, the name will be used as identifier. Otherwise, name will be extended with some additional characters (e.g. "_bis"). As this might be resulting from a programming mistake, only 8 instances of this module are allowed.
- resolution (size):
- VGA (640 * 480 )
- QVGA (320 * 240)
- QQVGA (160 * 120)
- color space:
- YUV422 (native format of the camera)
- YUV (24 bits)
- Y (8 bits)
- RGB (24 bits)
- BGR (24 bits)
- HSY (24 bits, see the appendix at the end of this document for further details)
- frames per second (fps): actually this parameter is not managed at the VIM level but at the applications level. It waits for the required time between two image requests.
At this stage, the VIM can look in the database to know what to provide to every GVM sending a request.
In the figure below, you can see that three different GVMs have registered to the VIM (shown in the blue section). For an explanation on how VIM works internally, let's consider that the two first GVMs registered are asking for the same image format. Suppose further that the third GVM needs the same image format as the one provided natively by the video source device.
In the VIM thread section (green part), you can see the ALImage containers that have been created by the VIM to manage future requests from the GVM.
The first set of containers to the left will just receive a pointer access to driver buffers. It is just an encapsulation to our image format without any memory allocation in order to set different attributes (width, height, resolution, color space, lockers, etc.) to buffers containing only raw data.
The second set of containers to the right have their own memory allocated because they will receive transformed images with the resolution and color space requested by GVMs.
| Note: |
Actually all the buffers from the right set allocate the maximum amount of memory that resolution and color space combinations allow. So changing resolution or color space will not need any memory reallocation, which is time consuming, but just parameters modification. |
|---|
Setting and asking parameters
- setResolution, setColorSpace, setFrameRate: at any moment, each GVM can have its resolution or color space set to a new value that will replace the one used when registering. The VIM will automatically checks if new settings are then necessary for the video source, and applies them if necessary.
- setParam: in the same way, new settings can be requested to the video source device, as enabling/disabling auto gain or white balance for NaoCam.
- getGVMResolution, getGVMColorSpace, getGVMFrameRate: return corresponding parameter for a GVM.
- getVIMResolution, getVIMColorSpace, getVIMFrameRate: Useful to know in which mode the video source device is running (it will work in the most suitable mode to answer the needs of every GVM).
- getParam: returns the value of a particular video source's parameter
Asking for an image
A GVM can request images in two ways: standard access that allows image conversion and direct raw access for accessing the driver's image buffers directly. Both ways can be used in local or remote mode.
- Standard access
> We recommend this mode. When using getImageLocal() function, the image is provided by the VIM with the format needed by the GVM.
(1) In the figure below, a GVM requests an image in local mode to the VIM.
(2) Suppose that this request is the first one, or that no request of the same kind was done recently. The VIM is going to ask the driver for the most recent buffer filled by the video source.
(3) Unlike "read" mode of the V4L driver, "streaming" mode doesn't require a copy of data from kernel space into user space, but just a remapping (action on pointers, not on data). So the driver is going to provide access to its latest updated raw image buffer by dequeuing it from the circular buffer, accessible via a ALImage buffer - left part of the VIM thread. In this example, the driver will therefore work temporarily with 4 buffers instead of 5 initially.
(4) The VIM is going to transform the up-to-date raw image buffer to the format required by the GVM that asked for an image. This image will be stored in the first obsolete and available buffer (not locked for writing) of the corresponding format. This appears in the right part of the VIM thread section on the diagram below. Once the conversion is done, the raw image buffer will be requeued automatically in the driver's circular buffer.
(5) As the request from the GVM is local, it will access to this converted ALImage buffer through a pointer. This ALImage buffer will be locked for writing as long as this GVM (and all others that might get an access to it) doesn't release it.
> Now suppose that another GVM wanting the same kind of image sends a request to the VIM.
(1) Let's say the request will be done in remote mode via getImageRemote(), to illustrate how remote calls differ. Local mode would have been equivalent, except for the way to access the buffer in the final stage.
(2) Here again the VIM will ask to the driver which is the latest updated buffer and will compare its timestamp with the one of the more recent ALImage with the requested format. Note that buffers are timestamped at the driver level when we start acquiring a new frame, so their accuracy is higher than a millisecond.
(3) If there is no newer raw image buffer, then the VIM will provide the same ALImage buffer as used by the first GVM. This way, we avoid redundant conversions by mutualizing the processes.
(4) As the request was done in remote mode, the ALImage buffer will be converted into an ALValue and sent via "soap" to the remote GVM. As the remote GVM receives a copy of the image via "soap", it doesn't need any access to the ALImage buffer anymore. Therefore, the remote GVM can release its hold on this buffer. This is done automatically at the end of the getImageRemote() function. Local GVM needs to release the ALImage buffer explicitly when they're done using it. So each getImageLocal() call should be paired with a releaseImage() when the local GVM task is done (not doing so blocks the GVM from obtaining new images with next calls to getImage). Calling releaseImage() from a remote GVM as we do for a local one can be done as this method will just do nothing.
Once every GVM has released a buffer, this one is available again for writing but can still be accessed for reading if needed.
| Note: |
It's obvious that GVMs are not supposed to modify the incoming image in order to let other GVMs obtain correct data. Therefore, it is strongly recommended to use an outcoming image if the result of the GVM process is an image. |
|---|
- Direct raw access
> In this mode, the user has a direct access to raw image buffers from the driver. This means that the GVM process must be done in order to work on the native video source format. In this case, instead of copying data from the unmapped driver buffer to an ALImage buffer with the correct format (arrow 4) and then providing a pointer (arrow 5), the GVM process will get a direct access through a pointer to the driver's latest updated buffer (arrow 4bis). It's faster and consumes less processing power. But there are restrictions for using this mode.
> In such a mode, a GVM keeps the unmapped driver's raw buffer as long as necessary, and so the driver's circular buffer continues working with less raw buffers. If as many GVMs as the number of driver's raw buffers keep a raw buffer unmapped for more than the needed time for the camera to write an image, then at some point no buffer will be available to write the video source's last frame. The device will then have to wait for the first released raw buffer to write an image again.
> This is why we recommend to ensure that there are always less GVMs needing raw buffers for more than ~25ms before calling the releaseDirectRawImage function than the number of raw buffers themselves. Actually, the driver usually sets 4 buffers on NAO, and NAO's camera provide a new frame every 33ms, but you need a safety margin for the time spent into VIM management and in other thread processes.
> Another point you have to be aware of when using this mode is that doing any action requesting modifications in the way camera is running (e.g. resolution, switch between camera) will cause troubles if any driver's raw buffer is still accessed.
> Like for the remote standard access mode, remote direct raw access mode releases automatically raw buffers as soon as the ALValue conversion is done.
> And as you will have probably noticed, the standard access mode uses the direct raw access mode internally.