Virtual Background with Amazon Chime SDK & BodyPix

Amazon Chime SDK for Javascript and MediaStream

Amazon Chime SDK for Javascript provides methods to specify the video (including audio) to be shared with the other party, for each of the following video types.

  • Movie captured by a camera
  • Shared Desktop
  • Movie file

BodyPix

By the way, in order to achieve a virtual background, we need to identify the person and the background.

Implements Virtual Background

So, we can think of a way to achieve this with described above.

  • First, we draw the image from the camera in HTML CanvasElement.
  • Then, this HTMLCanvasElement is input to BodyPix to create a mask image that identifies the person and background.
  • The background of the image acquired from the camera is replaced by the data of another image and rendered in HTML CanvasElement.
  • We’ll get the MediaStream from this HTMLCanvasElement and set it to chooseVideoInputDevice in the Amazon Chime SDK.

Merge VideoImage and Background

That’s the end of my explanation of the whole picture, but how do you actually merge the background and the image captured from the camera?

for (let rowIndex = 0; rowIndex < maskedImage.height; rowIndex++) {
for (let colIndex = 0; colIndex < maskedImage.width; colIndex++) {
const pix_offset = ((rowIndex * maskedImage.width) + colIndex) * 4
if (maskedImage.data[pix_offset] === 255 &&
maskedImage.data[pix_offset + 1] === 255 &&
maskedImage.data[pix_offset + 2] === 255 &&
maskedImage.data[pix_offset + 3] === 255
) {
pixelData[pix_offset] = bgImageData.data[pix_offset]
pixelData[pix_offset + 1] = bgImageData.data[pix_offset + 1]
pixelData[pix_offset + 2] = bgImageData.data[pix_offset + 2]
pixelData[pix_offset + 3] = bgImageData.data[pix_offset + 3]
} else {
pixelData[pix_offset] = maskedImage.data[pix_offset]
pixelData[pix_offset + 1] = maskedImage.data[pix_offset + 1]
pixelData[pix_offset + 2] = maskedImage.data[pix_offset + 2]
pixelData[pix_offset + 3] = maskedImage.data[pix_offset + 3]
}
}
}

Demo

So, I’ve posted a site that incorporates this logic on Heroku.Try to see how it works.

Amazon Chime Meeting Testbed

We are also creating a conference room system as a test bed for new features using video conferencing.
It is published in the following repositories.
Here, in addition to video conferencing with virtual backgrounds, you can also text chat and send stamps.
We also use AWS’s Lamda and DynamoDB to achieve a serverless environment. Go ahead and access it if you’re interested.

Finally

In this article, I’ve shown you how to use Amazon Chime SDK to create a virtual background on a video conferencing system.

I am very thirsty!!

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store