How to show camera view on remoting computer screen

[laultman]

The Microsoft Holographic Remoting sample app shows the HoloLens Camera video on the remoting computer's screen. I need to enable that capability in the StereoKit version.
Use case: Remote Assist via Teams. The Dynamics Remote Assist is a great app but does not allow a person wearing the HoloLens 2 device to also run a second application (Remote Assist is immersive). I need to use Teams with screen sharing from the remoting computer to another person on Teams. With the SK remoting app running AND Teams screen sharing the person remotely located could see what a person with the HoloLens is seeing. This is effectively the reverse of Dynamics Remote Assist.
How can I enable the feature?

[Edralos]

For now you can always use the "Connect" Windows feature which enables Miracast streaming. This is natively present in the HoloLens and can be activated directly from the bottom of the home menu. On your computer, "Connect" is a Windows feature you need to activate through the Optionnal features settings menu. Its name is called Wireless Display.

Warning

Since you wish see the users is experiencing AND run the app as a Holographic Remoting one, you should be aware that
both these features highly rely on the Wi-Fi network and that the Miracast will also run on your device. This could lead to a very strong lag on the HoloLens displayed image.
But something called frame mirroring is available within the Holographic Remoting libraries as extensions to OpenXR. Those features are not implemented yet in this project.

[laultman]

I appreciate the comments. I am looking for someone to help me with a custom version of the Holographic Remoting to continue it in StereoKit. This is one of the features I would like to implement for a client. Right now Microsoft Remote Assist will do this job with a little configuration of the Remote Assist session. My end game is to have a listener on a Teams account that will invoke the Remote Assist while in the StereoKit environment. At present, I have to connect in Remote Assist and then drop the call. I can then open my app and the Teams meeting members can see it live, latency is negligible with a half-decent connection. This only works for a pre-planned meeting in HL2 via Teams on Remote Assist.

[Edralos]

well, first of all, you can always set your StereoKit application to be an overlay application by setting to true the SkSettings.overlayApp though I must say I haven't tried that option yet on HL2 so I'm not sure how or if it even works.

Also, I might not have understood it well but you seem to want to run a StereoKit application on Holographic remoting through the Internet by having the server running the app being on a different network than the HoloLens acting as the display device. If so I do not recommend following that line, Holographic remoting works best on high quality and speed networks with little traffic to begin with. Though it can work (tried it over a VPN connection), the risk of having lag or worse a session cut short is quite high, so it is strongly advised to have the holographic app running on the same network as the client.

To me your best bet would be to either have your app running natively on the HL2 or on Holographic Remoting but in a network shared by both host and client and having the app running as overlayApp.

[laultman]

Thanks for the tips! I was not aware of the overlayApp setting. I’ll give it a try. I am configured in a “sort of” edge compute mode with a “backpack” edge compute device. It and the HoloLens are on a dedicated secured connection. The edge device uses another network for backend connectivity that is latency tolerant and has fault recovery. Larry

[Edralos]

I am not familiar with edge computing, but if it allows you to have a both stable and fast connection, you could always give it a go ! It would be interesting to test the limits of the Holographic Remoting system. 😄

As for the frame mirroring extension, I'm currently working on it though it does not natively provide any other abilities than giving the texture that'll be rendered on HL2. I also can't say for sure if this texture contains only the displayed meshes or both the meshes and an image of the captured environment