What is Microsoft Mesh Mixed Reality: A Guide to Its Uses
Microsoft CEO Satya Nadella and technical fellow Alex Kipman unveiled Mesh at the company’s Ignite Conference in March 2021. Microsoft Mesh is a new mixed-reality platform built on Microsoft Azure, enabling people in different physical locations to join collaborative and shared holographic experiences using HoloLens 2, virtual reality (VR) headsets, smartphones, tablets,, or PCs.
Table of Contents
What is Microsoft Mesh Mixed Reality?
Microsoft Mesh is one of the latest innovations in Microsoft’s ecosystem of mixed reality solutions that aims to connect the physical and digital worlds. Microsoft experts believe that this will lead to better teamwork. Microsoft Mesh promises to give us the best of both worlds (virtual and real), where users can interact with different people via augmented reality, virtual reality, and artificial intelligence-powered devices for an immersive multiuser experience for better communication, collaboration, and cooperation.
Microsoft notes that the eventual vision is to offer holoportation technology where collaborating employees will appear as photo-realistic, lifelike virtual avatars to their colleagues. This would mean that users can appear as themselves in the Mesh space and “teleport” between settings. Mesh is like nothing else on the market, with spatial sound and latency-free rendering.
What are the Potential Benefits of Microsoft Mesh?
Microsoft notes the Mesh environment is suitable for connecting people around the world. It has a range of use cases – specialists can share perspectives and documents in real-time, teams can design projects together using all kinds of 3D maps and technology, etc. Whether physically present or available as a hologram, colleagues will have access to the content they can interact with in real-time.
Here are some of the biggest benefits of Microsoft Mesh:
- Multi-device support: Mesh supports immersive head-mounted displays like the HoloLens, to VR headsets. There’s also support for tablets and smartphones and a fully tethered experience for PCs and Mac devices too.
- Developer access: The comprehensive developer platform comes with extensive tooling functionality and access to crucial Microsoft apps. Users can unlock Azure Active Directory, Microsoft Graph, and Microsoft Accounts. There’s also a selection of AI-powered capabilities available for developers to access in an instant.
- Immersive presence: Mesh comes with out-of-the-box avatars you can customize, or you can import your own rigged avatars. Their movement is powered by AI motion models to accurately represent the user’s movement. You can also import volumetric captures for a more realistic visual of users. Mesh also enables 360-degree realistic holoportation with sensors. You can access sensors through a custom camera setup or use features like the Azure Kinect to access depth-enhanced images.
- Spatial maps: Spatial maps allow you to create digital maps of your world that are much more accurate than GPS and are persistent globally. This enables content to be anchored, users to share the same point-of-view, and collaboration on the same 3D model.
- Holographic rendering: Holographic rendering refers to Microsoft’s integration with Azure giving you the option to render locally using the power of the device you are on or rendering in the cloud.
Multiuser sync: The multiuser synchronization ensures that each user is seeing the same thing at the same time. All movements within the mixed reality space have 100 milliseconds of latency or less.
How to Use Microsoft Mesh Mixed Reality?
You can use Microsoft Mesh for the following:
- Virtual collaboration – Microsoft Mesh enables you to bring colleagues and teams together no matter the distance and time; it feels like you’re all in the same room. As Mesh integrates well with Microsoft 365, all your calendars, content, connections, and workflows are also present in the mixed reality world.
- Review designs and models – Microsoft 365 apps have advanced really well in the last few years, and any documents shared through OneDrive allow people to work on them together. This is called co-authoring. But one of the purposes of Microsoft Mesh is to bring that same feature and check and annotate 3D models in real-time, whether in person or through the hololens. Moreover, any work made during sessions will automatically be saved so you can start exactly where you left off.
- Remote guidance – Microsoft Mesh provides great comfort for employees when experts can teach others remotely in areas where they are needed and resolve issues faster.
- Training and learning – In many parts of the globe, employees can train and learn a skill or two (like in the medical field) using holoportation and holographic sharing and visualization through Microsoft Mesh. This approach could greatly reduce travel and logistical costs for the companies since all are done in digital reality.
- Virtual meet-ups – Meeting in mixed reality through virtual meetings with Microsoft Mesh can potentially provide a connection similar to a meeting in person as you can see who you’re talking to as well as the avatars of other people who are with you in the meeting.
Steps to Get Started with Microsoft Mesh Reality
- Download the Microsoft Mesh app (Preview version) from the Microsoft Store.
- On the HoloLens 2, from the pins panel, launch the Microsoft Mesh app.
- Sign into the app using either your Microsoft Azure AD enterprise account or your personal Microsoft Account.
- Read and accept the Microsoft Mesh (Preview) terms and conditions when creating an account (Also check Microsoft Privacy Statement.)
- Start browsing the Mesh app on your HoloLens 2.
Are There Hardware and Software Requirements to Use Mesh?
To try the Microsoft Mesh Preview version, you will need the following:
- HoloLens 2 (version 2004 or upper). If you need more information, check Update HoloLens and HoloLens 2 enterprise deployment and management documentation.
- One of the below accounts:
- An Azure Active Directory (Azure AD)
– OR –
- A personal Microsoft Account (MSA).
- (For more info about accounts, see Manage user identity and sign-in for HoloLens.)
Tips for Using Microsoft Mesh Effectively
Here are some great uses:
- Microsoft launched the Games for Work app on Microsoft Teams to let you play games with co-workers, as gaming at work can lead to greater workplace productivity, boost collaboration, break down barriers, and foster team building. The Games app is integrated directly into the flow of work and can even be added to the next Microsoft Teams meeting.
- The new schedule sent in Teams allows selecting the future date and time for a chat message to be delivered without disrupting a recipient’s off-hours. Simply select and hold (or right-click) the send button to schedule it.
- Sign language view is a brand-new meeting experience in Microsoft Teams that help people with hearing people, like interpreters, signers, and others who use sign language.
- Meeting facilitators using the following feature on Whiteboard can now lead to more engaging, inspirational, and visually guided discussions.
- New features in Adoption Score help IT administrators to maintain a strong, strong commitment to user-level privacy and aggregate valuable insights and data.
How Microsoft Mesh Mixed Reality Being Used in Various Industries?
Microsoft Mesh, the mixed-reality platform, is transforming the world of business. Mesh is used in various industries, and everyone who works with a Microsoft partner gets more value from this exciting innovation.
- Healthcare – Surgeons worldwide are already using Microsoft’s Hololens technology that “transports” users to a virtual world where they interact and share information via virtual avatars. For healthcare professionals, this virtual environment provides a secure space to practice surgical procedures and improve patient care.
- Interior design – Holoportation technology lets designers collaborate on projects in a 3D virtual world where they can view interior layouts from multiple angles as if they were all in the same physical location together. Designers can also use Mesh to present interior plans in 3D to clients.
- Education – Augmented reality, virtual reality, and artificial intelligence can provide better experiences at all educational levels and make learning more interactive. Teachers and tutors can demonstrate learning concepts, digitize tutorials and guides, and incorporate gaming elements into the curriculum.
- Vehicle manufacturing – With Mesh, manufacturers can collaborate virtually and replicate production methods such as testing and assembling car parts. Users can better visualize these methods for improved product builds.
- Culinary – Mesh enables users to simulate a kitchen environment and create recipes with others. Alternatively, it can serve as a training tool that teaches students advanced food preparation techniques.
- Engineering – Microsoft Mesh allows those who work with 3D machine models to collaborate on projects, regardless of their locations. Engineers can visualize how machine parts fit together and use spatial rendering to build new technologies.
Examples of Successful Implementations of Microsoft Mixed Reality
- Mixed reality startup 8i is responsible for creating a lifelike hologram of John Hamm for the Sundance Film Festival and a Buzz Aldrin hologram for SXSW. It’s expected that the company’s Holo app will find many more applications for musicians, brands, and celebrities. You can even download the Holo app to your phone to create your own 3D animations.
- The Microsoft HoloLens was used by Case Western Reserve University in Ohio to teach anatomy to medical students.
- BAE Systems used a mixed-realty system to improve its battery-building process by 40%. The system projects a holographic template that helps factory workers visualize all the steps in the process.
- Several years ago, PGA Tour fans were able to use an app with a 3D rendering of golf courses. The Tour is also looking to use HoloLens for tournament setup and golf course design while players can prepare their strategy using mixed reality well before hitting their first drive on the links.
- Japan Airlines uses a HoloLens to help train engineers without having to visit a hangar.
- NASA worked with Microsoft to create the OnSight software that helps scientists and engineers be virtually present on Mars while they are still very much on Earth.
Responsible and Ethical Uses
Many large tech companies and faculties are making Responsible, and Ethical AI usage a top priority and are establishing AI principles and working groups. Microsoft has created these working groups:
- Office of Responsible AI (ORA) – sets the rules and defines the governance processes together with teams across the company.
- AI, Ethics, and Effects in Engineering and Research (Aether) Committee – advises Microsoft’s leadership on the opportunities and challenges presented by innovative AI.
- Responsible AI Strategy in Engineering (RAISE) – a team for enabling engineering groups across the company with the implementation of Microsoft’s Responsible AI rules.
- Microsoft Fairness, Accountability, Transparency, and Ethics in AI Research team (FATE) – a cross-functional, diverse group from the fields of sociotechnical orientation, such as HCI, information science, sociology, anthropology, science and technology studies, media studies, political science, and law that studies the societal implications of artificial intelligence (AI), machine learning (ML), and natural language processing (NLP). The aim is to facilitate innovative and responsible computational techniques and prioritize fairness, transparency, accountability, and ethics.
What is the Technology Behind Microsoft Mesh Mixed Reality?
Mesh uses a technology called “holoportation” that demonstrates real-time holographic communications with the Microsoft Hololens. Holoportation is described as “a new type of 3D capture technology that allows high-quality 3D models of people to be reconstructed, compressed, and transmitted anywhere in the world in real-time. This allows users wearing virtual or augmented reality displays to see, hear and interact with remote participants in 3D, almost as if they were present in the same physical space. From an audio-visual perspective, communicating and interacting with remote users edges closer to face-to-face communication.”
