Evaluate Background Blur Functionality of Amazon Chime SDK for JavaScript
Note:
This article is also available here.(Japanese)
https://cloud.flect.co.jp/entry/2021/11/05/144410
Introduction
The long-awaited background blur feature was added to Amazon Chime SDK for JavaScript![ref] As I mentioned in my previous article, it has been possible to implement virtual backgrounds or background blur using the Video Processing API. However, in this case, the developer had to prepare bodypix or other functions to separate the person from the background. This is a relatively high hurdle to implement background blur. With the new feature, Amazon Chime SDK for JavaScript takes care of the whole process of separating the person and background and then blurring the background. So we can implement such a functionality with very ease.
I introduced the usage and demo of this feature as guest blog in AWS officail site. Please see it.
In this blog, I would like to introduce the results of the performance evaluation of the simplified version that I conducted along with the writing of the guest blog above.
About Evaluation
The background blurring feature of video conferencing systems, such as this new feature, is commonly realized using an AI technique called semantic segmentation. This semantic segmentation is often evaluated roughly in terms of both (1) accuracy and (2) response time (computational complexity). In this article, I would like to briefly evaluate the response time by preparing two environments. Since the accuracy is not affected by the environment, I think it would be better for you to try the above guest blog demo in your environment. You can also find sample images in the blog above.
Evalution Items
In this article, I would like to look at the following three points as an evaluation of response time (computational complexity).
- Processing time
- FPS/Interval
- CPU usage
The “Processing time” is the time it takes to perform the background blurring process. Regarding “FPS/Interval”, we check if the FPS is degraded by the background blurring process. The background blur feature in video conferencing systems is controlled so that the processing cycle is not shorter than necessary, since there is no benefit to processing more than the FPS of the video. If the background blurring process is completed within this cycle, there should be no FPS degradation. Regarding “CPU usage”, we will check how much the CPU usage increases with the background blurring process.
Since we can change the strength of the background blur, we will evaluate it in three levels: Low, Mid, and High.
The background blur function of Amazon Chime SDK for JavaScript seems to be implemented using WebAssembly and Tensorflow Lite models, as described later. Since it is probably running on the CPU in the same way as Google Meet’s background blur, GPU usage was excluded from the evaluation.
Environment
This time, we used the following two environments to see the performance. We will use chorme as the browser for both experiments.
- ASUS UX390U (Win10, Core i5 7200U(Kaby Lake) 2.5GHz/2core, RAM 8G)
A notebook PC with a CPU that is about five years old, and is classified as a low-spec PC.(ref)
-> Below, I call this environment as ASUS. - M1 MBA (M1-chip 8core, 16GB)
This is last year’s MacBook Air with the M1 chip. Pretty fast PC.(ref)
-> Below, I call this environment as MBA.
How to evaluate
The evaluation method for each of the evaluation items is as follows. The following assumes that you have some knowledge of Amazon Chime SDK for JavaScript, so you can skip this section if you don’t need it.
- Processing time
Due to the configuration of the SDK, it is difficult to measure the raw processing time. This time, in the Video Processing API pipeline also used for background blur, we added a filter to obtain timestamps before and after the background blur process, and calculated the difference in time to simulate the measurement. Although there is an overhead in passing frames in the pipeline, we can confirm that the process was done in less than the measured time. The image of the measurement program is as follows.
- FPS/Interval
This is also measured in the same way, by adding a filter in the pipeline that records the timestamp of each cycle. The image of the measurement program is as follows.
- CPU usage
For ASUS, visually check the Windows System Monitor; for MBA, visually check the Activity Monitor. Neither is an exact measurement, but it will give you a general idea.
Result
We will review each assessment item for ASUS and MBA respectively.
ASUS
The image below shows ASUS without background blur applied yet, running at 14FPS. The CPU usage is hovering around the second scale, which is about 20%.
The background blur is applied as shown in the figure below. The strength of the blur is set to Low. The FPS remains at 14. Since the processing time is 36msec, I think it is reasonable that the FPS does not change since it can handle about 27FPS by simple calculation. The CPU usage has gone up a little bit, and it seems to go back and forth between two and three ticks, which means that the CPU load has increased by less than 10%.
I changed the blur strength to Mid, which is almost the same as Low.
I changed the blur strength to High. The result is almost the same as with Low. The strength of the blur does not seem to be a dominant factor.
MBA
The figure below shows the situation on the MBA without background blur applied yet, running at 16FPS. The CPU usage is about 20% for both system and user.
The background blur is applied as shown in the figure below. The strength of the blur is set to Low. The FPS remains at 16. The processing time is 8msec, which is incredibly fast. I think it is reasonable that the FPS does not change, since it can handle more than 100FPS by simple calculation. CPU usage is also almost unchanged. A few percent increase? It’s about the same.
I changed the blur strength to Mid, which is almost the same as Low.
I changed the blur strength to High. The result is almost the same as with Low. The strength of the blur does not seem to be a dominant factor. The trend is the same as with ASUS.
Summary and Discussion
The results are summarized in the table below. If you have an old PC like ASUS, the CPU usage will go up a little, but it was still around 10%. It seems to be a level where it’s okay to work on other things while the videoconference is going on (let’s focus on the videoconference). In addition, if you have a modern PC such as an MBA, it really doesn’t seem to have any impact. Looking at the ASUS evaluation, it seems that even older PCs can be used without any problem, so this is a feature that can be used by a wide range of users.
A little more detail (This is just a guess.)
As mentioned above, the added background blur feature seems to be created with a fairly lightweight and fast machine learning model. I think it is comparable to the Google Meet virtual background model that I introduced before. I was a little curious about this, so I checked the communication log. It looks like they are using WebAssembly to run the TFLite model, just like Google Meet(I couldn’t even tell if they were using the same model.).
Finally
We evaluated the performance of the new background blur feature in Amazon Chime SDK for JavaScript. It is quite lightweight and fast, so I think it can be implemented in a wide range of user environments, even on some older PCs. This time, the function is to blur the background, but in the future it would be nice if the function could be extended to replace any image.
