Ukemochi
Eat real food without taking off your HMD
A 2019 MITOU Program project (IPA, Japan), certified as Super Creators
What is Ukemochi?
Ukemochi lets you eat real food while wearing an HMD (VR headset). It uses deep neural networks to find the food in the real-world camera image, then overlays it on the virtual world as a dish or a 3D model (Augmented Virtuality). You keep your sense of presence in VR while you eat.
Background
Cheap, capable HMDs and VR social platforms like VRChat now let us live inside the worlds of novels, games, and anime. There we spend our days as avatars of any gender, species, or shape. If you haven't tried it yet, go get yourself reincarnated in another world. Today, if possible. One thing is still hard in those worlds: eating. With an HMD on, you can't see the food in front of you. Just piping the real-world video into VR doesn't work either, because it breaks the feeling of actually being in the virtual world (presence). So we built an application that lets you eat without giving that up.
Toward Unexplored Frontiers
Two things we want to reach. First, a world where people who can't eat what they love, for whatever reason, can still get the feeling of eating it. Nakano and Horita study how changing what food looks like with StarGAN changes how it tastes. If that research makes it into Ukemochi, you might someday eat whatever you like, whenever you like. Second, dragon meat. Nobody we asked has ever eaten dragon meat (if you have, please tell us). VR experiences hit hard, so in a fantasy world like Monster Hunter, we suspect dragon meat would actually taste like something. That's why VR needs a real eating experience, and why we're building it.
Download
Hosted on BOOTH (free; the page is in Japanese)
News
- Our paper "Ukemochi: A Video See-through Food Overlay System for Eating Experience in the Metaverse" was presented at ACM CHI 2022 (Extended Abstracts).
- Nakano and Horita were certified as Super Creators of the 2019 MITOU Program (IPA, Japan).
- Ukemochi appeared on the Japanese TV show "Be Bop! High Heel" (ABC TV, March 26).
Team
Citation
If you use Ukemochi in your research, please cite:
@inproceedings{nakano2022ukemochi,
author = {Nakano, Kizashi and Horita, Daichi and Isoyama, Naoya and Uchiyama, Hideaki and Kiyokawa, Kiyoshi},
title = {Ukemochi: A Video See-through Food Overlay System for Eating Experience in the Metaverse},
booktitle = {CHI Conference on Human Factors in Computing Systems Extended Abstracts (CHI EA '22)},
publisher = {ACM},
year = {2022},
pages = {1--8},
doi = {10.1145/3491101.3519779}
}