Visual/Haptic Interface

WYSIWYF Display

To build a VR training system for visuo-motor skills, an image displayed by a visual interface should be correctly registered to a haptic interface so that the visual sensation and the haptic sensation are consistent spatially and temporally. In other words, it is desirable that what you see is what you feel (WYSIWYF).

We propose a method that can realize correct visual/haptic registration, namely WYSIWYF, by using a vision-based object tracking technique and a video keying technique. Combination of the encountered-type haptic device with the motion command type haptic rendering algorithm makes it possible to deal with two extreme cases (free motion and rigid constraint) and the system can provide realistic haptic sensations, such as free-to-touch and move-and-collide.

A first prototype was built and several demonstrations were presented. The user could encounter the haptic device exactly when her hand reaches the virtual object in the display. Although the first prototype remains some technical problems to be solved, it showed the validity of the proposed approach.

Go to the WYSIWYF Display Page in CMU.


Overview of WYSIWYF Display


Manipulating a virtual cube

Related papers

  1. Y.Yokokohji, R.L.Hollis and T.Kanade, "What You can See Is What You can Feel," Proc. IEEE Virtual Reality Annual International Symposium (VRAIS'96), pp.46 - 53, 1996.
  2. Y.Yokokohji, R.L.Hollis and T.Kanade, "Vision-Based Visual/Haptic Registration for WYSIWYF Display," Proc. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS'96), pp.1386 - 1393, 1996.
  3. Y.Yokokohji, R.L.Hollis, T.Kanade, K.Henmi and T.Yoshikawa, "Toward machine mediated Training of Motor Skills -Skill Transfer from Human to Human Via Virtual Environment-," Proc. IEEE International Workshop on Robotics Human Communication (RO-MAN'96), pp.32 - 37, 1996.
  4. Y.Yokokohji, R.L.Hollis and T.Kanade, "A Visual/haptic Interface to Virtual Environment: WYSIWYF Display", Transactions of the Virtual Reality Society of Japan, Vol.2, No.4, pp.17 - 26, 1997. (in Japanese)
  5. Y.Yokokohji, R.L.Hollis and T.Kanade, "WYSIWYF Display: A Visual/Haptic Interface to Virtual Environment," PRESENCE, Teleoperators and Virtual Environments, Vol.8, No.4, pp.412 - 434, 1999.

Virtual Control Panel

To render multiple virtual objects with a single encountered-type haptic device, the following two behaviors must be realized; (i) the device should move to the target object location in advance of the user's hand so that the user can encounter it; (ii) otherwise, the device should keep away from the user's hand, avoiding an unexpected collision. To realize such a consistent encounter, the user's hand motion must be tracked and the path planning of the device should be carefully considered.

We proposed a path-planning algorithm for encountered-type haptic devices that render multiple virtual objects in 3D space. The proposed algorithm determines the device location based on the minimum distance problem between a point (user's hand) and a convex polyhedron constructed from the reference points of multiple objects. A virtual control panel of automobiles was constructed using an encountered-type haptic device where the proposed path-planning algorithm was implemented. The validity of the proposed algorithm was confirmed and the required performances such as device speed and accuracy of hand tracking were evaluated.


Rendering multiple objects by a single haptic device


Virtual Control Panel

Related papers

  1. Y.Yokokohji, J.Kinoshita, and T.Yoshikawa, "Path Planning for Encountered-type Haptic Devices that Render Multiple Objects in 3D Space,"Proc. IEEE VR2001, pp.271 - 278, 2001.
  2. Y.Yokokohji, J.Kinoshita, and T.Yoshikawa: "Path Planning for Encountered-type Haptic Devices that Render Multiple Objects in 3D Space",Transactions of the Society of Instrument and Control Engineers, Vol.40, No.2, pp.139 - 147, 2004. (in Japanese)

Encountered-type haptic device for multiple-fingertip contacts

Unlike conventional haptic devices, an encountered-type device is not held by a user all the time. Instead, the device stays at the location of a virtual object and waits for the user to encounter it.
We extend this concept to fingertip contacts and design an encountered-type haptic display for multiple fingertip contacts to simulate tasks of grasping an object with any shape and size.

Before designing the device, we intensively observed human grasping behaviors. This observation was very helpful to determine the mechanism of the device. An encountered-type device for three-fingered grasping was actually prototyped based on our design.


Concept of encountered-type haptic device for multiple-fingertip contacts


First Prototype

A Real Tennis Ball
Simulating a Tennis Ball
A Real Bat
Simulating a Bat
A Real Paint Can
Simulating a Paint Can

Related papers

  1. Y.Yokokohji, N.Muramori, Y.Sato, and T.Yoshikawa: "Designing an Encountered-Type Haptic Display for Multiple Fingertip Contacts based on the Observation of Human Grasping Behavior," Preprints of the 10th International Symposium of Robotics Research (ISRR 2003), Oct.19-22, Siena, Italy, 2003.
  2. Y.Yokokohji, N.Muramori, Y.Sato, and T.Yoshikawa: "Designing an Encountered-Type Haptic Display for Multiple Fingertip Contacts based on the Observation of Human Grasping Behavior," Proc. the 12th Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, pp.66-73, 2004.
  3. Y.Yokokohji, N.Muramori, Y.Sato, T.Kikura, and T.Yoshikawa: "Design and Path Planning of an Encountered-type Haptic Display for Multiple Fingertip Contacts based on the Observation of Human Grasping Behavior," Proc. IEEE International Conference on Robotics and Automation, pp.1986-1991, 2004.

Go to top of page