Amazon Chime SDK virtual background with Powerpoint’s slides.
This article is also available here.(Japanese)
https://cloud.flect.co.jp/entry/2020/08/24/123607
In our last article, we showed you an experiment in Amazon Chime SDK to exceed the video session limit (16 sessions) for video delivery.
(2020/09/15th added) The easiest way to make mmhmm-like presentations with virtual background in Zoom, Teams, Chime
I’d like to introduce the Amazon Chime SDK again. This time, I’ll show you how to achieve a feature that has recently been talked about in other video conferencing tools.
What I’m going to show you is how to use slides of Powerpoint or other as a virtual background. “mmhmm” was the first hot application these days. Later, a similar feature was released for Zoom. I think it’s a good way to add a sense of realism to a remote presentation.
I’ll show you how to achieve this using the Amazon Chime SDK. It uses Amazon Chime SDK (js), so you don’t need to install any software to use it. In addition to slides of Power point, you can also make all of your screen-shareable windows the background. So, for example, you can do live coding with speakers on an editor.
Here’s an image of the actual operation.
How to send the camera footage with Amazon Chime SDK
As we discussed in a previous article on virtual backgrounds, the Amazon Chime SDK shows you how to register a camera device’s ID as a common method for sending camera footage in tutorials and samples. However, it is actually possible to register a MediaStream using the same method.
So, in fact, the Amazon Chime SDK is designed to be flexible enough to share anything as long as you can get a MediaStream.
For example, you can get a MediaStream from HTMLCanvasElement. Because of this, you can draw something on HTMLCanvasElement and deliver it.
In the above article, we used this to achieve a virtual background.
In this article, I’ll use it again to create a virtual background for the slide.
Image acquisition of slides
In Virtual Backgrounds, BodyPix was used to discriminate between a person’s area and the background area of the camera image, and then the background area was replaced with the image of the specified background image and drawn in HTMLCanvasElement.
This time, we will retrieve the background image from an application window, such as PowerPoint. We can retrieve the application’s video as a MediaStream by calling the following. After setting this as the source of the HTMLVideoElement, we draw a frame on an HTMLCanvasElement of the same size as the camera footage. Then, for each frame of camera footage, this HTMLCanvasElement image is captured and used as a background image.
navigator.mediaDevices.getDisplayMedia().then(media => {
...
})The process flow can be summarized as follows. The upper part of the image is the normal process flow of the virtual background as described above. The lower part of the diagram is the current process. The changes are in red.
Virtual Foreground
From what we’ve seen so far, you can see that Amazon Chime SDK can deliver a variety of processed videos.
For example, the figure below is a line drawing of a person, or Ascii art.
It’s not a virtual background, but a virtual foreground.
As you can see, the Amazon Chime SDK allows for a variety of extensions by accepting MediaStream for the camera’s video delivery. It’s fun.
Demo
We’ve prepared a site that incorporates the same logic as the one we’ve presented here.
NPM
I also made an npm package for easy use.
FLECT Amazon Chime Meeting
We have also created a video conferencing room system as a testbed for new video conferencing features and have made the repository available.
The features I’ve described are also implemented here, so you can access them if you’re interested.
Finally
In this article, I’ve shown you how to set a slide as a virtual background in Amazon Chime SDK.
This method can also be used to set any application that can share a screen as a background.
For this reason, I’m sure you can think of many ways to use it, for example, to deliver live coding, so please try it out.
