Team Overview

The research team of Prof. Dr. Dieter Schmalstieg focuses on interactive graphics, in particular augmented reality, visualization and GPU techniques.

Augmented reality (along with the related topics of mixed reality and virtual reality) has been the primary research field of the team for many years. The team has made many seminal contributions in this area, covering the entire scope of augmented reality topics, including visualization, rendering, spatial interaction, authoring, mobile tracking, software architectures and applications. 

Visualization is an essential element of visual computing. This is true for augmented and virtual reality, but even more for conventional interactive systems that work with big data and need to provide a comprehensive user interface to an information worker. Big and complex data sets are produced by many application areas, but particularly interesting source come from the life sciences.

GPU techniques are not only used for games and virtual reality, but increasingly also for simulation and visualization. The GPU architecture with thousands of computing cores lends itself to the development of highly parallel algorithms. We work on new parallel programming techniques for the GPU and on applications in graphics and procedural modeling

Augmented Reality


Situated visualization for augmented reality

Lead: Denis Kalkofen, Dieter Schmalstieg

Summary: When consuming information using augmented reality, the user’s perception is necessarily affected by the situated nature of the presentation, which is – per definition – related to the real environment. This form of visualization is frequently called situated visualization, and it is different from visualization on conventional (desktop) displays since it must try to strike a compromise between virtual and real elements in the user’s field of view. Two very important topics that we have investigated in this context are: first, the placement of virtual labels and other overlays in the user’s field of view such that clutter and undesired occlusions are minimized, and, second, X-ray visualization revealing occluded information by simulated transparency effects on occluding real objects.

Selected publications

Markus Tatzgern, Valeria Orso, Denis Kalkofen, Giulio Jacucci, Luciano Gamberini, and Dieter Schmalstieg. Adaptive Information Density for Augmented Reality Displays. In Proc. IEEE Virtual Reality, March 2016.

Jacob Madsen, Markus Tatzgern, Denis Kalkofen, Dieter Schmalstieg, and Claus Madsen. Temporal Coherence Strategies for Augmented Reality Labeling. IEEE Transactions on Visualization and Computer Graphics (Proc.

Raphael Grasset, Markus Tatzgern, Tobias Langlotz, Denis Kalkofen, and Dieter Schmalstieg. Image-Driven View Management for Augmented Reality Browsers. In Proc. IEEE International Symposium on Mixed and Augmented Reality (ISMAR) 2012, pages 177-186, Atlanta GA, USA, November 2012.

Eduardo Veas, Erick Mendez, Steven Feiner, and Dieter Schmalstieg. Directing Attention and Influencing Memory with Visual Saliency Modulation. In Proc. ACM Conference on Human Factors in Computing Systems (CHI'11), Vancouver, Canada, May 2011.

Denis Kalkofen, Markus Tatzgern, and Dieter Schmalstieg. Explosion Diagrams in Augmented Reality. In Proc. IEEE Virtual Reality, pages 71-78, Lafayette LA, USA, March 2009.

Denis Kalkofen, Erick Mendez, and Dieter Schmalstieg. Comprehensible Visualization for Augmented Reality. IEEE Transactions on Visualization and Computer Graphics, 15(2):193-204, 2009.

Gerhard Schall, Erick Mendez, Ernst Kruijff, Eduardo Veas, Sebastian Junghanns, Bernhard Reitinger, and Dieter Schmalstieg. Handheld Augmented Reality for Underground Infrastructure Visualization. Journal of Personal and Ubiquitous Computing, 13(4):281-291, May 2008.

Denis Kalkofen, Erick Mendez, and Dieter Schmalstieg. Interactive Focus and Context Visualization in Augmented Reality. In Proc. 6th IEEE International Symposium on Mixed and Augmented Reality (ISMAR'07), pages 191-200, Nara, Japan, October 2007. Best student paper award.

Photometric registration for augmented reality rendering

Lead: Denis Kalkofen, Dieter Schmalstieg

Summary: In application areas such as entertainment or immersive shopping, the capability to graphically render mixed reality scenes such that the virtual elements are indistinguishable from the real world is highly desirable. Such a consistent rendering is only possible if the photometric properties of the real world are known. In particular, we need information on the incident lighting and the material properties of the objects in the scene. This is challenging, as both are, in general, not known. Objects can move and light sources can change. We have worked on methods to recover photometric properties and compute approximate global illumination robustly, in real time and from a minimum of observations.

Selected publications

David Mandl, Kwang Moo Yi, Peter Mohr, Peter Roth, Pascal Fua, Vincent Lepetit, Dieter Schmalstieg, and Denis Kalkofen. Learning Lightprobes for Mixed Reality Illumination. In Proc. International Symposium on Mixed and Augmented Reality (ISMAR), Nantes, France, October 2017.

Thomas Richter-Trummer, Jinwoo Park, Denis Kalkofen, and Dieter Schmalstieg. Instant Mixed Reality Lighting from Casual Scanning. In Proc. IEEE International Symposium on Mixed and Augmented Reality (ISMAR'16), Merida, Mexico, September 2016.

Lukas Gruber, Thomas Richter-Trummer, and Dieter Schmalstieg. Real-time Photometric Registration from Arbitrary Geometry. In Proc. IEEE International Symposium on Mixed and Augmented Reality (ISMAR) 2012, Atlanta GA, USA, November 2012.

Spatial interaction with augmented reality

Lead: Dieter Schmalstieg

SummaryIn contrast to desktop computing, where interaction is typically constrained to keyboard and mouse, augmented reality interface afford a variety of form factors. Recently, affordable augmented reality head sets are becoming available. However, since the early 2000s, handheld devices such as smartphones were the only widely available form-factor for deploying mobile augmented reality, and will continue to be pervasive in the foreseeable future. While a lot of work on next generation user interfaces remains only on a conceptual (mock-up) level, we have a reputation for explored interface designs by providing operational technical implementations. This work has spanned such diverse subjects as transparent devices, conversational agents, navigation and multi-device interfaces.

Selected publications

Andreas Hartl, Clemens Arth, Jens Grubert, and Dieter Schmalstieg. Efficient Verification of Holograms Using Mobile Augmented Reality. IEEE Transactions on Visualization and Computer Graphics, 22(7):1843-1851, 2016.

Jens Grubert, Matthias Heinisch, Aaron Quigley, and Dieter Schmalstieg. MultiFi: Multi Fidelity Interaction with Displays On and Around the Body. In Proc. ACM Conference on Human-Computer Interaction (CHI'15), Seoul, South Korea, April 2015.

Alessandro Mulloni, Hartmut Seichter, and Dieter Schmalstieg. Handheld Augmented Reality Indoor Navigation with Activity-Based Instructions. In Proc. MobileHCI 2011, Stockholm, Sweden, August 2011.

Alessandro Mulloni, Andreas Duenser, and Dieter Schmalstieg. Zooming Interfaces for Augmented Reality Browsers. In Proc. 13th International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI 2010), pages 161-169, Lisboa, Portugal, September 2010.

Istvan Barakonyi and Dieter Schmalstieg. Augmented Reality Agents for User Interface Adaptation. Journal of Computer Animation and Virtual Worlds, 19:23-35, 2008.

Dieter Schmalstieg, L. Miguel Encarnacao, and Zsolt Szalavari. Using Transparent Props For Interaction With The Virtual Table. In Proceedings of SIGGRAPH Symposium on Interactive 3D Graphics (I3D'99), pages 147-154, Atlanta, GI, April 1999. Best paper award nominee 1999 in the competition of Fraunhofer Haus der Graphischen Datenverarbeitung, Darmstadt, Germany.

Authoring for augmented reality

Lead: Denis Kalkofen, Dieter Schmalstieg

Summary: Once mature platform become widespread, augmented reality will be perceived as a new type of medium, alongside movies, TV, mobile games and so on. While content creation for entertainment benefits from an economy of scale (a particular movie or games can amortize large production costs by reaching a large audience), content creation for professional applications, such as navigation or repair instructions requires a larger degree of automation to be feasible. Dr. Schmalstieg and his team have investigated approaches that enable and author to leverage existing data sources, such as CAD, product databases, or even printed manuals and Youtube videos, in the creation of augmented reality content.

Selected publications

Peter Mohr, David Mandl, Markus Tatzgern, Eduardo Veas, Dieter Schmalstieg, and Denis Kalkofen. Retargeting Video Tutorials Showing Tools With Surface Contact to Augmented Reality. In Proc. ACM Conference on Human Factors in Computing Systems (CHI), Denver, CO, USA, May 2017.

Peter Mohr, Bernhard Kerbl, Denis Kalkofen, and Dieter Schmalstieg. Retargeting Technical Documentation to Augmented Reality. In Proc. ACM Conference on Human-Computer Interaction (CHI'15), Seoul, South Korea, April 2015.

Bernhard Kerbl, Denis Kalkofen, and Dieter Schmalstieg. Interactive Disassembly Planning of Complex Objects. Computer Graphics Forum(Proc. EUROGRAPHICS 2015), April 2015.

Tobias Langlotz, Daniel Wagner, Alessandro Mulloni, and Dieter Schmalstieg. Online Creation of Panoramic Augmented Reality Annotations on Mobile Phones. IEEE Pervasive Computing, 11(2):56-63, April 2012.

Erick Mendez, Gerhard Schall, Sven Havemann, Sebastian Junghanns, Dieter Fellner, and Dieter Schmalstieg. Generating Semantic 3D Models of Underground Infrastructure. IEEE Computer Graphics and Applications, 28(3):48-57, May 2008.

Dieter Schmalstieg and Daniel Wagner. Experiences with Handheld Augmented Reality. In Proc. 6th IEEE International Symposium on Mixed and Augmented Reality (ISMAR'07), pages 3-18, Nara, Japan, October 2007.

Dieter Schmalstieg, Gerhard Reitmayr, Gerhard Schall, Joseph Newman, Daniel Wagner, Florian Ledermann, and Istvan Barakonyi. Managing Complex Augmented Reality Models. IEEE Computer Graphics and Applications, 27(4):48-57, 2007. 2006.

Florian Ledermann and Dieter Schmalstieg. APRIL - A High Level Framework for Creating Augmented Reality Presentations. In Proceedings of IEEE Virtual Reality 2005, pages 187-194, Bonn, Germany, March 2005.

Mobile tracking for augmented reality

Lead: Clemens Arth, Dieter Schmalstieg 

Summary: The one key enabling technology for mobile augmented reality is real-time 3D tracking. Without such tracking, proper registration of virtual and real is impossible, and augmented reality applications are limited to placing “floating labels” in the general area of a point of interest. The challenge of mobile tracking lies of course in finding efficient algorithms that determine the 3D pose of a mobile device fast and robustly. Dr. Schmalstieg and his team have delivered seminal work on mobile tracking, starting with the first fully autonomous implementations of marker tracking (2003) and natural feature tracking (2007) on the commercial devices of the time. The findings on the efficiency of optical tracking methods were widely adopted. Some of the methods were acquired by Qualcomm and used in the Vuforia software product, which is a market leader today. Later research work in the group of Dr. Schmalstieg investigate the even more challenging problem of outdoor tracking “in the wild”, where the high cost of data collection and rapid visual changes make conventional model-based approaches infeasible.

Selected publications

Clemens Arth, Christian Pirchheim, Jonathan Ventura, Dieter Schmalstieg, and Vincent Lepetit. Instant Outdoor Localization and SLAM Initialization from 2.5D Maps. IEEE Transactions on Visualization and Computer Graphics (Proc. ISMAR 2015), 21(11), November 2015. Best paper award.

Jonathan Ventura, Clemens Arth, Gerhard Reitmayr, and Dieter Schmalstieg. Global Localization from Monocular SLAM on a Mobile Phone. IEEE Transactions on Visualization and Computer Graphics, 20(4):531-539, March 2014.

Christian Pirchheim, Dieter Schmalstieg, and Gerhard Reitmayr. Handling Pure Camera Rotation in Keyframe-Based SLAM. In Proc. IEEE International Symposium on Mixed and Augmented Reality (ISMAR) 2013, Adelaide, Australia, October 2013.

Clemens Arth, Manfred Klopschitz, Gerhard Reitmayr, and Dieter Schmalstieg. Real-Time Self-Localization from Panoramic Images on Mobile Devices. In Proc. IEEE International Symposium on Mixed and Augmented Reality (ISMAR), Basel, Switzerland, October 2011.

Daniel Wagner, Dieter Schmalstieg, and Horst Bischof. Multiple Target Detection and Tracking with Guaranteed Framerates on Mobile Phones. In Proc. IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pages 57-64, Orlando FL, USA, October 2009.

Clemens Arth, Daniel Wagner, Manfred Klopschitz, Arnold Irschara, and Dieter Schmalstieg. Wide Area Localization on Mobile Phones. In Proc. IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pages 73-82, 2009.

Daniel Wagner, Gerhard Reitmayr, Alessandro Mulloni, and Dieter Schmalstieg. Real Time Detection and Tracking for Augmented Reality on Mobile Phones. IEEE Transactions on Visualization and Computer Graphics, 16(3):355-368, 2009.

Daniel Wagner, Gerhard Reitmayr, Alessandro Mulloni, Tom Drummond, and Dieter Schmalstieg. Pose Tracking from Natural Features on Mobile Phones. In Proc. 7th IEEE International Symposium on Mixed and Augmented Reality, pages 125-134, Cambridge, UK, September 2008. Best Paper Award.

Daniel Wagner and Dieter Schmalstieg. ARToolKitPlus for Pose Tracking on Mobile Devices. In Proc. 12th Computer Vision Winter Workshop (CVWW'07), Sankt Lambrecht, Austria, February 2007.

Daniel Wagner and Dieter Schmalstieg. First Steps Towards Handheld Augmented Reality. In Proceedings of the 7th International Conference on Wearable Computers, pages 127-135, White Plains NY, USA, October 2003.

Software architectures for collaborative augmented reality

Lead: Dieter Schmalstieg 

Summary: Since in the late 1990s, we have explored the potential of augmented reality for computer-supported collaborative work. The results of this research endeavor was the first augmented reality system for co-located collaborative, called STUDIERSTUBE. Before the arrival of commodity techniques such as WiFi or multiplayer game engines, designing and building a software architecture that combined diverse hardware (tracking, head-mounted display), real-time display and distributed multi-user simulation with low latency, was a major challenge. Consequently, the highly cited works that were published as a result of this project equally contribute to computer-supported collaborative work and to software engineering.

Selected publications

Thomas Pintaric, Daniel Wagner, Florian Ledermann, and Dieter Schmalstieg. Towards Massively Multi-User Augmented Reality on Handheld Devices. In International Conference on Pervasive Computing (PERVASIVE 2005), pages 208-219, Munich, Germany, May 2005.

Gerhard Reitmayr and Dieter Schmalstieg. OpenTracker - A Flexible Software Design for Three-Dimensional Interaction. Virtual Reality, 9(1):79-92, December 2005.

Dieter Schmalstieg, Gerhard Reitmayr, and Gerd Hesina. Distributed Applications for Collaborative Three-Dimensional Workspaces. PRESENCE - Teleoperators and Virtual Environments, 12(1):52-67, 2003.

Hannes Kaufmann and Dieter Schmalstieg. Mathematics And Geometry Education With Collaborative Augmented Reality. Computers & Graphics, 27(3):339-345, 2003.

Dieter Schmalstieg, Anton Fuhrmann, Gerd Hesina, Zsolt Szalavari, L. Miguel Encarnacao, Michael Gervautz, and Werner Purgathofer. The Studierstube Augmented Reality Project. PRESENCE - Teleoperators and Virtual Environments, 11(1):32-54, 2002.

Gerhard Reitmayr and Dieter Schmalstieg. Mobile Collaborative Augmented Reality. In Proceedings of the 2nd IEEE International Symposium on Augmented Reality (ISAR'01), pages 114-123, New York NY, USA, October 2001.

Gerd Hesina, Dieter Schmalstieg, Anton Fuhrmann, and Werner Purgathofer. Distributed Open Inventor: A Practical Approach to Distributed 3D Graphics. In Proceedings of ACM Symposium on Virtual Reality Software & Technology (VRST'99), pages 74-81, London, UK, December 1999.

Visualization


Visualization in medicine

Lead: Dieter Schmalstieg 

Summary: In medicine, visualization is most often used in combination with volumetric imaging sources. Medical visualization can support diagnosis, planning of surgeries or minimaly invasive treatment, and intra-operative navigation. Volumetric visualization is particularly challenging because it must handle huge datasets in real time. We have investigated visualization tools for a number of clinical applications, including liver surgery, tumor ablation, forensic analysis and cranial surgery.

Selected publications

Christoph Ebner, Dieter Schmalstieg, Roberto Blanco Sequeiros, Horst R. Portugaller, Juergen Fuetterer, Michael Moche, Markus Steinberger, and Dieter Schmalstieg. Visualization-Guided Evaluation of Simulated Minimally Invasive Cancer Treatment. In Proc. EUROGRAPHICS Workshop on Visual Computing in Biology and Medicine (VCBM), Bergen, Norway, September 2016.

Markus Gall, Xing Li, Xiaojun Chen, Dieter Schmalstieg, and Jan Egger. Computer-Aided Planning and Reconstruction of Cranial 3D Implants. In Proc. IEEE Engineering in Medicine and Biology Society (EMBC'16), Orlando, FL, USA, August 2016.

Martin Urschler, Alexander Bornik, Eva Scheurer, Kathrin Yen, Horst Bischof, and Dieter Schmalstieg. Forensic Case Analysis: From 3D Imaging to Interactive Visualization. IEEE Computer Graphics and Applications, 32(4):79-87, July 2012.

Rostislav Khlebnikov, Bernhard Kainz, Judith Muehl, and Dieter Schmalstieg. Crepuscular Rays for Tumor Accessibility Planning. IEEE Transactions on Visualization and Computer Graphics, 12(7):2163-2172, October 2011.

Bernhard Reitinger, Alexander Bornik, Reinhard Beichel, and Dieter Schmalstieg. Liver Surgery Planning Using Virtual Reality. IEEE Computer Graphics and Applications, 26(6):36-47, November-December 2006.

Visual links and large displays

Lead: Dieter Schmalstieg 

Summary: When visualization systems encompass large amounts of heterogeneous data, the most common approach is to present multiple views or windows. In particular when using very large displays, the user can have difficulties to identify corresponding pieces of information. Visual links provide an attractive visual aid to link and connect the pieces. Dr. Schmalstieg and his team have investigated various efficient and effective ways to add visual links as a feature to existing desktop systems in a minimally invasive way that lends itself to everyday use.

Selected publications

Thomas Geymayer, Markus Steinberger, Alexander Lex, Marc Streit, and Dieter Schmalstieg. Show me the Invisible: Guidance to Hidden Content. In ACM Conference on Human Factors in Computing Systems (CHI), Toronto, Canada, April 2014. Best of CHI Honorable Mention Award.

Markus Steinberger, Manuela Waldner, and Dieter Schmalstieg. Interactive Self-Organizing Windows. Computer Graphics Forum, 31(2):621-630, May 2012.

Markus Steinberger, Manuela Waldner, Marc Streit, Alexander Lex, and Dieter Schmalstieg. Context-Preserving Visual Links. IEEE Transactions on Visualization and Computer Graphics, 17(12):2249-2258, October 2011. Best paper award at IEEE InfoVis 2011.

Manuela Waldner, Markus Steinberger, Raphael Grasset, and Dieter Schmalstieg. Importance-Driven Compositing Window Management. In Proc. ACM Conference on Human Factors in Computing Systems (CHI'11), Vancouver, Canada, May 2011. Honorable Mention Award.

Manuela Waldner, Werner Puff, Marc Streit, Alexander Lex, and Dieter Schmalst ieg. Visual Links Across Applications. In Proc. Graphics Interface 2010, pages 129-136, Ottawa, Canada, May 2010. Best student paper award.

GPU Techniques


High performance graphics

Lead: Markus Steinberger, Dieter Schmalstieg

Summary: We have established a track record of advanced parallel programming techniques on the GPU. A particular area of interest lies on algorithms, which do not have parallel workloads immediately available up-front, but require dynamic scheduling. Examples for such algorithms are sparse matrix operations, deep tree traversals or adaptive rendering techniques. Dynamic scheduling also lends itself to enforcing priorities and carefully balancing producer-consumer problems.

Selected publications

Bernhard Kerbl, Michael Kenzel, Dieter Schmalstieg, Hans-Peter Seidel, and Markus Steinberger. Hierarchical Bucket Queuing for Fine-grained Priority Scheduling on the GPU. Computer Graphics Forum, 2016.

Rostislav Khlebnikov, Philip Voglreiter, Markus Steinberger, Bernhard Kainz, and Dieter Schmalstieg. Parallel Irradiance Caching for Interactive Monte-Carlo Direct Volume Rendering. Computer Graphics Forum, 33(3):61-70, June 2014.

Markus Steinberger, Michael Kenzel, Pedro Boechat, Bernhard Kerbl, Mark Dokter, and Dieter Schmalstieg. Whippletree: Task-based Scheduling of Dynamic Workloads on the GPU. ACM Transactions on Graphics (Proc. SIGGRAPH Asia 2014), December 2014.

Markus Steinberger, Bernhard Kainz, Bernhard Kerbl, Stefan Hauswiesner, Michael Kenzel, and Dieter Schmalstieg. Softshell: Dynamic Scheduling on GPUs. ACM Transactions on Graphics (Proc. SIGGRAPH Asia), 31(6), November 2012. Article 161.

Markus Steinberger, Bernhard Kainz, Stefan Hauswiesner, Rostislav Khlebnikov, Denis Kalkofen, and Dieter Schmalstieg. Ray Prioritization Using Stylization and Visual Saliency. Computers and Graphics, 36(6):673-684, October 2012.

Bernhard Kainz, Markus Grabner, Alexander Bornik, Stefan Hauswiesner, Judith Muehl, and Dieter Schmalstieg. Ray Casting of Multiple Volumetric Datasets with Polyhedral Boundaries on Manycore GPUs. Transactions on Graphics (Proc. SIGGRAPH Asia), 28(5), 2009.

Procedural modeling

Lead: Markus Steinberger, Dieter Schmalstieg 

Summary: Procedural modeling is essential in applications which require very large geometry sets at a level of detail which cannot be created by hand. For example, detailed city models can contain hundreds of thousands of buildings and billions of polygons. Such large models may not even fit into the available memory. We have developed the first approach to evaluate procedural models on the fly, thereby supporting literally infinitely detailed models.

Selected publications

Pedro Boechat, Mark Doktery, Michael Kenzel, Hans-Peter Seidely, Dieter Schmalstieg, and Markus Steinberger. Representing and Scheduling Procedural Generation using Operator Graphs. ACM Transactions on Graphics (Proc. SIGGRAPH Asia), December 2016.

Markus Steinberger, Michael Kenzel, Bernhard Kainz, Joerg Mueller, Peter Wonka, and Dieter Schmalstieg. Parallel Generation of Architecture on the GPU. Computer Graphics Forum, 33(2):73-82, April 2014. Best paper honorable mention award.

Markus Steinberger, Michael Kenzel, Bernhard Kainz, Peter Wonka, and Dieter Schmalstieg. On-the-fly Generation and Rendering of Infinite Cities on the GPU. Computer Graphics Forum, 33(2):105-114, April 2014.