Week 5: Session Logistics & Basic Mesh Networking Demo
This week, we concluded the networking and logistical aspects of the implementation. As a result of this, we are now are able to create sessions, add players, send meshes from the Hololens to the server, and broadcast a mesh from the server to all Hololenses. We have built a basic demo on top of this implenentation which displays the same (coordinate system-aligned) mesh across multiple Hololens overlayed on the real world.
Networking
We’ve successfully implemented mesh transfer in both directions (to and from the Hololens). The Hololens can, on command, send a collection of submeshes of the scene to the server, as well as the appropriate timestamps. The Hololens can also request a mesh from the server.
Session Logistics & Coordinate System Alignment
We’ve extended the game loop handler to incorporate session handling (i.e. anchor point generation, player joining, sending anchor points to new players).
Mesh alignment & Analysis
Hamid did an experiment for mesh alignment in the server side. For simulating the multiplayer mesh alignment we capture the mesh from two different hololens devices and intentionally try to miss different parts of the scene in each of two hololens devices. Then, four point pairs between meshes are used for initial rough alignment between two meshes (this will be replaced with the anchor point handling and shared-coordinate system in multiplayer session). Once we have the initial alignment between two meshes, we run ICP for finer and more accurate alignment of two meshes. The merged result of final alignment is shown in the following 3D model.
Demo
Our demo currently allows the user to explore the scene, and as the Hololens’ internal model is updated, the mesh is periodically sent to the server. The demo application constantly displays the global mesh as it exists on the server. So, when multiple players are connected to the server, the mesh shown will include regions scanned by all of the Hololenses.
The current version of the demo doesn’t include any gameplay elements. We have also yet to find a way to easily delete the internal Hololens model of the scene, which causes problems, because the scene will appear to have been scanned by all Hololenses prior to receiving a new one from the server. We are currently clearing the mesh by resetting the device to factory settings. This is very time consuming and very inefficient. For our demo to have full effect, it will be important that we find at least a slightly easier way of doing this.
Plans
-
Game loop (placement of a sphere somewhere in the scene, move it when user clicks on it)
-
Piping the mesh from the server to our external mesh analysis application and back. This needs to be done quickly enough such that the model in each of the Hololenses are not inconsistent for too long.
-
Analyzing the aligned meshes to produce a list of candidate positions for the object to be placed
-
Animations, better 3D models, other game elements
PRD Discrepancy
By week 5, we estimated that we would be done with the gameplay system, the region selection, and a single-player demo. In reality, we have completed the gameplay system, which consists of coordinate system alignment, anchor point handling, session and player management, and mesh sending/receiving, but have yet to implement the region selection. The reason for this is that we have yet to connect the two components we’ve been working on (mesh analysis in Hamid’s code + mesh networking from our server). This should be an easy addition, and is one of the next steps.
While there’s currently a difference between our progress and the goals set in the PRD, this doesn’t mean we’ve fallen behind on per-week goals, and instead have just reordered a number of tasks in order to pipeline the remainder of development. Below is a summary of modifications we’ve made to our goals:
Things we planned to have done by now, but will be doing later:
- A single player demo. This is because working on the technical components of mesh alignment and hole/inconsistent region detection requires having multiple aligned Hololens meshes of the same regions, so we implemented the anchor point and shared-coordinate system handling first, to make sure we have the proper data. As a result of this, once we have the basic game loop in place, running a single player demo will be the same as running a multiplayer one.
Things we planned to do later, but have done already:
- Integrated game using mesh alignment from multiple clients. See above..
Things we didn’t have on the PRD, but realized we needed to do:
- Multiplayer session handling and anchor point management. This is crucial to get an initial estiamte for the mesh alignment, and to ensure the coordinate systems are aligned in the internal Hololens model.
TL;DR: we’re not as far from reaching our goals as it might seem from the fact that we don’t have a gameplay demo.