Instant Avatar: Instant Generation of Photorealistic Human Models with Mobile Devices

Capturing humans in every detail and displaying them in a photo-realistic way is one of the hardest challenges of computer graphics and computer vision. The human body shows a huge variety in shape and appearance, and capturing every aspect of it is necessary to create a believable virtual representation of someone. Many technologies struggle with the so called uncanny valley effect: This term describes the fact that “humanoid objects which appear almost, but not fully, like real human beings elicit revulsion in observers”. State-of-the-art approaches in this area can generate digital human avatars, but do not reach photo-realism and fully believable results. Furthermore, current solutions require complex setups and powerful hardware to create a digital representation of a person. This limits the widespread use of the technology and therefore also impedes commercial success. Being able to overcome these problems would open up a large range of use cases. Applications in the fields of health care, education, e-commerce and entertainment, to only name a few, would benefit from such a technology. The activities of companies in this field emphasize the relevance of this topic. Almost every aspect of our digital communication will benefit from realistic avatars. To achieve this, the technical solution must be mobile and easy to use without the need for special hardware. Thus, the project partners will combine 4D avatar capturing and image-based rendering in a system incorporating principles of photogrammetry and surface light fields to capture fine details and view-dependent effects which are true to the original images of the subject. Facial expressions and body motion will be captured using novel tracking and classification algorithms and can then be applied to avatars to create a realistic and believably human impression. The Institute of Computer Graphics and Vision, lead of Prof. Dieter Schmalstieg, has decades of experience on the topics of 3D reconstruction, deep learning, augmented and virtual reality, visualisation and other topics relevant to this project. A team of experts focused on research and development of mobile AR solutions and has released commercial products in this field. Together with the funding of this project, all requirements to achieve both academic and commercial success are given.

Co-Investigator

Univ.-Prof. Dipl.-Ing. Dr.techn. Dieter Schmalstieg
Institute of Computer Graphics and Vision
Graz University of Technology