Mixed Reality in BIM

Mixed reality (MR) is a subset of a broader concept known as extended reality (XR). Extended reality refers to all combinations of real and virtual environments generated by computer technology where humans and machines can interact [1]. The term also refers to various hardware, software, and techniques used to provide the experience. These include head-mounted displays, smartphones with positional tracking features, and fixed displays with head tracking features [2].

Extended reality technologies have been put to use in many industries. The entertainment industry is using these technologies to deliver immersive gaming and movie-watching experiences. Educational platforms use extended reality to provide an interactive and immersive learning experience. Extended reality technologies have even infiltrated the agriculture industry. Interesting use of the technology can be seen in figure 1, where extended reality technology is used to create a relaxing environment for cows to increase milk yield in dairy farms. Further benefits of the technology can be seen in the healthcare industry, the manufacturing industry, and in the architecture, engineering, and construction (AEC) industry. This blog is focused on the AEC industry. It will discuss some important concepts in extended reality and their possible role in digitizing the AEC industry.

Figure 1: Virtual reality headset is being used to create a relaxing environment for cows to increase milk yield | Image: Ministry of Agriculture and Food of the Moscow Region

Even though the term ‘extended reality’ and other related terms were coined in the past few decades, the concept of visual manipulation has been around for much longer. For instance, Sala delle prospettive (see figure 2) is a 16th-century interior design in Rome where a view through a balcony is painted on the walls. When people stand at the right position in the room and look at the walls, they will have the illusion of watching outside of a window. Other efforts to manipulate the way people perceive the real world had been devised throughout history up until today. Novels (Pygmalion's Spectacles, Neuromancer) and movies (Tron, Ready Player One) that featured some form of manipulated reality have played some role in popularizing the concept. 

Figure 2: Sala delle prospettive illustrating the concept of visual manipulation | Image: Web Gallery of Art

Reality-Virtuality Continuum

Extended reality technologies can be represented using a continuum as shown in figure 3. This continuum is known as Reality-Virtuality Continuum. At the two opposite ends of the continuum are the real world and the complete virtual world. Between these opposing ends, we can find different technologies including augmented reality, augmented virtuality, and mixed reality. We will discuss each of them briefly.

At the left end of the continuum is the real world without any visual manipulation. In this category, we find technologies referred to as ubiquitous computing technologies. Ubiquitous computing is the concept of embedding computing capability to everyday devices and connecting them to a network so that they are available all the time [3]. Examples of such technologies include smart assistants (Amazon’s Alexa, Microsoft’s Cortana, Apple’s Siri, Samsung’s Bixby, and Google Assistant) and smart home devices (doorbells, speakers, light bulbs, thermostats, security cameras).

Figure 3: Reality-Virtuality Continuum | Image: CreatXR

Virtual reality (VR) is when everything in view is virtual and it is found on the right end of the continuum. Immersive-based simulations, mostly assisted by VR headsets, are representatives of virtual environments. The headsets could be mobile or tethered. Figure 4 presents a virtual space station that can be accessed on Oculus quest. The virtual environment could be a computer-generated world or a video capture of the real world. Through these technologies, one can take a computer-generated tour of cells or go beyond our world and visit other planets. Furthermore, many real-world places including the 7 wonders of the world can be virtually explored (and if you are curious about soldiers’ life in the trenches, there is a virtual simulation of that as well ).

Figure 4: A completely virtual environment seen through virtual reality (VR) headset | Image: Oculus Quest - Space station environment

Augmented reality (AR) refers to virtual objects that are overlaid on reality, and it is found in the middle of the reality-virtuality continuum, closer to the reality side. The technology can be implemented through smart glasses or phone and tablet displays (Apple ARKit, Google ARCore, Microsoft 3D Viewer). To understand its practical application, let us say we are about to purchase new furniture for our living room and we want to know how it will look before purchasing. By using augmented reality technology, we can add a digital copy of that furniture into our living room and visualize the result on our mobile or tablet (see figure 5).

Figure 5: Augmented reality (AR) being used to insert a digital copy of furniture into a real-world environment | Image: ADLOID furniture application

Augmented Virtuality (AV) are technologies situated between augmented reality and virtual reality. These are technologies that bring real-world objects into a virtual world. This is the opposite of augmented reality (AR) where virtual objects were added into the real world. As an example, let us consider that we are virtually exploring a space station and we wanted to see how our real-world chair would look like in a space station. Using augmented virtuality, we can bring our chair into the virtual space station. Figure 6 shows a user looking at their real hand in the virtual world.

Figure 6: Augmented Virtuality (AV) being used to bring the users hands (real-world object) into a virtual environment | Image: Gerd Bruder

Finally, we have Mixed Reality (MR) where the virtual world can sense and interact with the real world and it is placed right in the middle of the continuum (see figure 7). Unlike virtual reality, mixed reality operates in the real world with some amount of virtual content. As it can be seen in figure 8, mixed reality exploits the advantages of both augmented reality and augmented virtuality. It provides more interactable content than the other technologies.

Figure 7: Mixed reality (MR) provides an interactive virtual content overlaid on a real-world environment | Image: Innovae

Figure 8: Comparison of AR, AV, and MR based on their content and interactivity | Image: adapted from [4]

Extended reality in the AEC industry

While reading about the different types of extended technologies in the previous section, you might have already thought about some use cases in the AEC industry. A wide range of applications for the AEC industry have been developed and are in current use, while even more are being researched. We will look at some of these applications in this section.

A study by Al-Adhami et. al. [5] explored the application of extended reality in construction quality control. The study employed Microsoft HoloLens to visualize a BIM model in AR. In the study, a virtual model of a building was overlaid on its physical twin (the building in the real world) to inspect the quality of work and accuracy. A similar study was conducted by Raimbaud et al. [6] who explored the use of mixed reality technology in construction supervision. Mixed reality technology was used to provide a comparison of a BIM model with what actually was built.

Yilei [7] noted that mixed reality technologies support 3D formats widely used by many BIM applications such as Autodesk Revit, Trimble SketchUp, Trimble connect, and Autodesk Navisworks. The study used two Microsoft HoloLens mounted on Trimble hard hats. Then mixed reality was used for visualization and model interaction to facilitate decision-making in architectural design.

Another study by Elstner [8] explored some other use cases of extended reality in the AEC industry. The use cases were as follows:

• An evacuation simulation for a tunnel system in case of fire hazard. Extended reality was used to simulate evacuation and study human behavior during the process.
• Construction site safety. Extended reality was used for safety training by providing a life-like experience.
• Immersive design of residential buildings, hospitals, and airports. Extended reality was used to provide interactive visualization for the players involved in the design process.
• Planning of HVAC systems. Extended reality was used to visualize the complicated systems.
• Planning of road infrastructure. Extended reality was used to simulate traffic and test multiple design options.

In addition to the studies mentioned so far, there has been other research on the topic of the application of extended reality in the AEC industry. Alizadehsalehi [4] identified a total of 30 studies on that topic. Some of the use cases of extended reality explored by these studies include construction safety, prefabrication, site survey, environmental study, construction-related training, communication, and problem-solving. Figures 9 - 16 present some of the use cases discussed so far and some new ones including building design review (figure 9), landscape design review (figure 10), locating and communicating property boundaries (figure 11), construction inspection (figure 12), planning and managing earthwork (figure 13), plant design review (figure 14), visualizing underground assets (figure 15) and understanding visual impact of infrastructures (figure 16).

Figure 9: Building design review using extended reality| Image: Yeppar

Figure 10: Landscape design review using extended reality| Image: Pool & Spa

Figure 11: Locating property lines using extended reality (Homesnap app) | Image: Homesnap

Figure 12: Construction inspection using extended reality| Image: Pepper Construction

Figure 13: Trimble’s SiteVision using extended reality| Image: KHL Group

Figure 14: Plant design review using extended reality| Image: Extranet Evolution

Figure 15: Visualizing underground assets using extended reality| Image: Electrical Contractor

Figure 16: Studying visual impact using extended reality| Image: Inglobe Technologies

Safety Issues

Safety is an important issue when discussing the use of extended reality. Several videos on the internet show people getting into accidents while plugged into a virtual reality environment. The accidents include stumbling, tripping over obstacles, falling, crashing into walls or other people, and many damaged pieces of furniture. If such accidents occur at a construction site, they could lead to serious injuries and property damage. Similar accidents could also occur in an office environment.

Some VR products have a guardian system to provide safety for the user. Using this system, a user will be able to define a safe working area free of obstacles. Then, while working in the VR environment, if the user leaves the predefined safe working area or even comes close to the boundary, the guardian system will warn the user. However, even with such safety measures, accidents sometimes happen.

Virtual reality can also have negative physiological effects on users [9]. Users may feel disorientated, nauseated, or dizzy. These effects may continue even after the VR headset is removed. As a result, some VR devices warn against engaging in visually or physically demanding activities right after using the device (for instance read Oculus health and safety warnings). These safety concerns should be well considered when using fully immersive virtual reality headsets in offices and even more so when used at construction sites (A full report on the safety implications of virtual reality systems can be found here).

The mixed reality technologies commonly used at construction sites are not completely immersive (they don’t entirely replace the physical environment with a virtual one). However, they still can prevent users from fully comprehending their environment. Hence the safety of using mixed reality gadgets at a construction site as well as in offices should be considered.

Current and upcoming trends in Extended Reality

Open standards such as the industry foundation classes (IFC) are developed to facilitate data exchange between different building information modeling (BIM) software platforms (See our blog on openBIM standards). Similar efforts exist in extended reality development that is to develop a common standard [8]. OpenXR is part of such an effort. It is an open standard for virtual and augmented reality platforms, which will assist in unifying extended reality development. Figure 17 illustrates the advantages OpenXR can introduce to extended reality software development. Without cross-platform standards, extended reality applications must rely on proprietary standards [10]. This results in fragmentation where different tools went their separate way. OpenXR provides a solution to this fragmentation by providing cross-platform access into diverse extended reality devices and platforms. It is a royalty-free and open standard. It has been widely adopted by the industry where It is supported by large corporations such as Google, Microsoft, Intel, and Autodesk. A complete list of companies publicly supporting OpenXR can be found here.

Figure 17: The important role OpenXR plays in unifying extended reality development | Image: PC Perspective

Other standards have been developed to bring virtual reality to web browsers. At first, WebVR was proposed to provide a VR experience to everyone regardless of the device they use [11]. This was later deprecated and replaced by a WebXR, which is a web standard for extended reality.

In terms of technologies, the devices used to provide extended reality experiences are continuously evolving. The future is expected to bring extended reality devices that are smaller and yet more capable. Perhaps smart glasses that are the size of normal glasses with computer power (or a separate pocket-sized computer). This will relieve the strain of carrying around a relatively heavy head-mounted device. And it will be less intrusive as well. This could encourage their use at construction sites.

The use of extended reality technologies in the AEC industry, with sufficient consideration of safety, can play an important role in digitizing the industry. There are several scientific studies that show its potential and its different use cases, some of them mentioned in this blog. The technology itself is constantly evolving and, therefore, the research into its application in the AEC industry should also be continuous.

This blog post was based on a recent lecture given by Kim Nyberg during the Cambridge BIM School held in December 2020. Kim plays an active role in Trimble Buildings as a Technology Director.

Reference

[1] Fast-Berglund, Å., Gong, L., and Li, D., Testing and validating Extended Reality (XR) technologies in manufacturing, in Procedia Manufacturing, ( 2018), vol. 25, pp. 31–38, doi: 10.1016/j.promfg.2018.06.054.

[2] WebXR Device API. https://www.w3.org/TR/webxr/ (accessed Jun. 08, 2021).

[3] Shea, S., What is Ubiquitous Computing (Pervasive Computing)?, (Oct. 2019). https://internetofthingsagenda.techtarget.com/definition/pervasive-computing-ubiquitous-computing (accessed Jun. 10, 2021).

[4] Alizadehsalehi, Sepehr, Ahmad Hadavi, and Joseph Chuenhuei Huang. 2020. “From BIM to Extended Reality in AEC Industry.” Automation in Construction 116(December 2019).

[5] Al-Adhami, M., Wu, S. and Ma, L. (2019) ‘Extended Reality Approach for Construction Quality Control’, CIB World Building Congress, (June), pp. 17–21.

[6] Raimbaud, Pierre et al. 2019. “BIM-Based Mixed Reality Application for Supervision of Construction.” In 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), IEEE, 1903–7.

[7] Yilei, Huang. 2020. “Evaluating Mixed Reality Technology for Architectural Design and Construction Layout.” Journal of Civil Engineering and Construction Technology 11(1): 1–12.

[8] Elstner, M., Use cases of Extended Reality in the construction industry. Lappeenranta, (2020).

[9] Department for Business Energy & Industrial Strategy, The safety of domestic virtual reality systems: A literature review, (2020).

[10] The Khronos Group, OpenXR Overview - The Khronos Group Inc. https://www.khronos.org/openxr/ (accessed Aug. 04, 2021).

[11] WebVR - Bringing Virtual Reality to the Web. https://webvr.info/ (accessed Jun. 08, 2021).

Flavia de Andrade Pereira
BIM Enrichment for inclusion of Building Automation System information