How to create virtual background with the new feature of Amazon Chime SDK “video background replacement”
Note:
This article is also available here.(Japanese)
https://cloud.flect.co.jp/entry/2021/12/07/073749
Introduction
Jan. 25, 2022, Amazon Chime SDK for Javascript offically released the long-awaited new feature “background replacement”.
In the past, Amazon Chime SDK app developers had to create virtual backgrounds in their own way (e.g. google meet), but with the addition of this official AWS feature, they can now create virtual backgrounds more simply.
Also, since this feature is achieved using the previously provided Video Processing API in the Amazon Chime SDK for Javascript, developers who have been using this feature (e.g. Background Blur) can now use it with little additional learning.
Now, let’s try creating a virtual background function to see how it works. The behavior of the virtual background function you created will look like this.
Premise
If you are an engineer with experience developing applications using the Amazon Chim SDK for Javascript, the background replacement feature is fairly easy to use. I don’t think you need any special prerequisite knowledge. For more information about the Video Processing API, please refer to this article. Knowing the Video Processing API will help you understand it better. I have introduced the Video Processing API in the past, please refer to this article.
How to use
In this article, I am going to add the virtual background feature to the background blur demo introduced in the previous article.
In this section, we will only introduce the implementation of the virtual background function. For the whole picture, please refer to useChimeClient.ts in the repository described below.
The way to create a virtual background with the Replace Background feature is described in the official documentation. The following code is excerpted from the official page with some comments. It creates a virtual device (transformDevice) to replace the background of the video input from the camera device.
const processors = [];
if (await BackgroundReplacementVideoFrameProcessor.isSupported()) {
const image = await fetch("https://pathtoimage.jpeg"); // (1) fetch image for virtual background.
const imageBlob = await image.blob();
const options = { imageBlob };
const replacementProcessor =
await BackgroundReplacementVideoFrameProcessor.create(null, options); // (2) create processor for background replacement.
processors.push(replacementProcessor);
}
let transformDevice = new DefaultVideoTransformDevice(
logger,
device,
processors
); // (3) create virtual device.The image is fetched at (1). And then the backgroundReplacementProcessor is created with it as input at (2). This is the component that performs the background replacement process. At (3), we create a transformDevice with this backgroundReplacementProcessor and the camera device as input.
Finally, you can specify this in the “chooseVideoInputDevice” of the MeetingSession to create the virtual background function. For example, it might look like this.
meetingSession.audioVideo.chooseVideoInputDevice(transformDevice);It’s so easy.
You can follow this official document, but if you follow the source code, you will create BackgroundReplacementProcessor every time you change the background image, which is not efficient. By default, the backgroundReplacementProcessor seems to be launched by the Web Worker, so if you don’t handle the termination properly, the Web Worker process will keep growing. If you use Chrome’s Developper Tools, you’ll see something like this.
If you have a use case where you want to switch the background image, you may want to cache the reference to the backgroundReplacementProcessor you created once and call the setImageBlob method to set the background image.
replacementProcessor.setImageBlob(imageBlob);Demo
We have placed a demo of the contents of this article in the following repository, whose Branch is aws-demo02 .
Please clone the following. And then follow the readme to run it. As described in the readme, you will need to set up your AWS credential in advance.
$ git clone https://github.com/w-okada/amazon-chime-bg-blur-demo.git -b aws-demo02After entering the conference room, you will be able to set the virtual background from the button at the top of the screen.
This is what it looks like when you run it.
It’s a little rough, but it looks pretty good.
CPU Usage
I used Task Manager to see how much load the CPU was under. I used a ThinkPad (corei5 8350U 1.7GH) for this evaluation.
The part of the CPU usage graph commented in blue is the CPU load when the background replacement is not used. The area commented in red is the CPU load when the background replacement is used. It is a rough estimate, but it seems that the usage rate increases by about 6% at most by replacing the background. I think this is a fairly low load.
Under this load, the above fineness of background replacement is quite good, isn’t it?
And more
Although I did not introduce it this time, the background replacement function seems to have a function to skip processing depending on CPU usage. Since mid-range smartphones still have limited processing power, I think this is a great feature for those users. I’d like to try it out when I get a chance.
Finally
This time, I used the new background replacement feature of Amazon Chime SDK for Javascript to create a virtual background. It’s an official implementation that I’ve been really waiting for, so I’ll be using it a lot!
Acknowledgements
The video in the text is from this site.
https://pixabay.com/ja/videos/
Disclaimer
In no event shall we be liable for any direct, indirect, consequential, or special damages arising out of the use or inability to use the software on this blog.
