Matrix inversion problems in VRJ VNC

patch for modules/vrjuggler/tools/vrjvnc/VNCDesktop.cpp

Fixing the compile problems (which are my fault) with the use of gmtl::invertFull() results in a working version of VNCDesktop. Can you explain item #3 in your description in more detail?

I committed item #2 in your patch to the 2.0 Alpha 2 branch and to the HEAD branch of the 'juggler' module.

This comment is from Chris Just (who posted the original bug report):

The main problem I saw when trying to drag the desktop window was that, as soon as it was grabbed, it would rapidly fly of into space. It looked to me like the code was calculating the new transformation and then, instead of replacing the old value with the new, it multiplied the old value by the new value. The matrix manipulation is rather hairy in that code, so it's possible I was misinterpretting the intent somehow.

If I recall correctly, I also saw incorrect results (most obvious when rotating the frame) when the user's navigation matrix was not an identity matrix. The frame would appear to rotate around a point offset somewhat from the user's wand. That's what the bits in the patch about the world xform were trying to resolve. At any rate, it worked for me.

I have hacked around this problem by using gmtl::invertFull_orig() instead of gmtl::invert() (see Revision 1.42 of VNCDesktop.cpp). Something is going wrong in vrjvnc::VNCDesktop that prevents gmtl::invert() from choosing the correct matrix inversion algorithm to use.