Image Overlay and Mixed Reality
Accurate Image Overlay with Vision/Inertial hybrid Tracking
Lesson Learned from WYSIWYF Display
The first prototype of WYSIWYF Display allowed the user to move the camera/display system only at the initial stage. During the simulation, the camera must be fixed due to the limitation of our hardware specification. To remove this limitation, we introduced a fast video tracker, the Tracking Vision by FUJITSU. The result was disappointing, resulting in a large misalignment between the real scene and the virtual scene. This was due to the end-to-end system delay, the total delay from the image capturing by the camera until the completion of image rendering.
At that time, we did not know about end-to-end system delay and dynamic registration error.
WYSIWYF Display
 |
 |
When the camera is fixed (left), there is no misalignment. When the camera is moved while a user is moving a virtual cube, a large misalignment occurs (right). This error is due to the end-to-end system delay.
Accurate Image Overlay with Hybrid Tracking
We propose a method for accurate image overlay on head-mounted displays (HMDs) using vision and accelerometers. The proposed method is suitable for video see-through HMDs in augmented reality applications but not limited to them.
Acceleration information is used for predicting the head motion to compensate the end-to-end system delay and to make the vision-based tracking robust. Experimental results showed that the proposed method can reduce the alignment errors within 6 pixels on average and 11 pixels at the maximum, even if the user moves his/her head quickly (with 10 [m/s2] and 49 [rad/s2] at the maximum). The viewing range was enlarged by placing the additional landmarks in the environment.
Prototyped video see-through HMD
System configuration
Experimental results when using accelerations
Experimental results when not using accelerations (vision only)
Related papers
- Y.Yokokohji, Y.Sugawara and T.Yoshikawa, "Accurate Image Overlay on Head-Mounted Displays Using Vision and Accelerometers", Transactions of the Virtual Reality Society of Japan, Vol.4, No.4, pp.589 - 598, 1999. (in Japanese)
- Y.Yokokohji, Y.Sugawara, and T.Yoshikawa, "Accurate Image Overlay on See-Through Head-Mounted Displays Using Vision and Accelerometers," Proc. IEEE VR2000, pp.247 - 254, 2000.
- Y.Yokokohji, D.Eto, and T.Yoshikawa, "It's Really Sticking! -Dynamically Accurate Image Overlay Through Hybrid Vision/Inertial Tracking,"Proceedings International Symposium on Mixed Reality (ISMR 2001), pp. 196-197, 2001.
Image Overlay for Vehicle Navigation
As an application of our image overlay technique, we proposed a method for image overlay on the front glass of a vehicle to navigate a driver to a desired destination.
By overlaying the navigation information on the front glass, the driver need not gaze at the console panel. Therefore, accidents caused by gazing at console panel could be reduced. To overlay the image accurately on the target object through the front glass, both the vehicle's position/orientation and the driver's position/orientation are estimated by vision-based tracking and measuring angular velocities of the vehicle's rear wheels. Experimental results show the validity of the proposed method.
Optical See-Through HMD
Head-up display can be regarded as a big optical see-through display
Conventional car navigation system (A driver must look at a small display)
On-the-Scene head-up display concept (J.Fukano, S.Okabayashi, M.Sakata, 1994)
Experimental setup
Overview of the prototyped wagon
Experimental results
Related paper
- T.Nakatsuru, Y.Yokokohji, Daisuke Eto, and T.Yoshikawa: "Image Overlay on Optical See-through Displays for Vehicle Navigation," Proc. the 2nd IEEE and ACM International Symposium on Mixed and Augmented Reality, pp.286-287, 2003.