MARIO: Mid-air Augmented Reality Interaction with Objects

I can’t wait to go to the Digital Content Expo in Japan this coming October. Last year, I have been mesmerized by the Aqua top display… I wonder which technology will surprise me this year !

Watch last year highlights: https://www.youtube.com/watch?v=Z608S5HmBPM
(also featured at the top of the post)

This is including MARIO – Mid-air Augmented Reality Interaction with Objects technology.

Interact with virtual characters in the real world This mixed reality interface places virtual characters in the real world. It was developed by the Naemura Lab at the University of Tokyo.

Users can have an animated character jump onto their hand, as well as guide the character onto blocks, creating a novel interactive experience.

Recently, devices have been developed that can form images in mid-air. We’ve utilized one of those, and combined it with sensors and a projector, to provide an intuitive display experience where a picture in the air is skillfully merged with the real world.

Data such as the height of the user’s hand and the shape of the block is obtained using Kinect, and the system is programmed to automatically move the character to the highest place. This data is used to display the character and its shadow correctly in mid-air, both geometrically and optically, creating a very realistic looking experience.

With the optical device used here, the image position is determined by the distance between the display and the device. So, we can change the image position by changing the display position. We’ve utilized this feature to make the character in mid-air look as if it’s moving backwards and forwards, by connecting the display to an actuator to enable physical movement.

With just a picture in mid-air, it’s hard for the human eye to recognize where the character is. So, we project a shadow separately. Projecting a shadow makes it easier to sense the picture’s depth.

Because this system can skillfully merge real objects with mid-air images, we think it could be applied to advertising showcases. We think cleverly combining real objects and video advertisements could provide a new experience.