Note:
This article is also available here.(Japanese)
https://cloud.flect.co.jp/entry/2021/09/14/105249

Introduction

In my previous article, I introduced how to reduce the traffic by combining multiple users’ video streams into a single stream in a video conferencing system or game distribution system (Among us auto mute) using Amazon Chime SDK JS. In this article, I would like to introduce another way to reduce the traffic by reducing the resolution of the video image when it is sent, and use super-resolution technology after it is received.

The image below shows the result of applying this technique. The image on the far right is the result…


Note:
This article is also available here.(Japanese)
https://cloud.flect.co.jp/entry/2021/08/17/064854

Introduction

In the previous article, I showed you how to create Live streaming system for multiplayer games with Amazon Chime SDK for javascript (Amazon Chime SDK JS). This system has the ability of “Among us Auto Mute” and “Broadcast game screen”.

In this system, the game screens of up to 15 users are distributed in real time. For this reason, a certain amount of network bandwidth is assumed on the clients. In addition, the CPU load on the browser increases due to the need to decode the video as it is received.

In…


Note:
This article is also available here.(Japanese)
https://cloud.flect.co.jp/entry/2021/07/21/113845

Introduction

The Olympic Games are just around the corner, and e-sports is starting to get a lot of attention in the digital world. In a related vein, I’d like to use Amazon Chime SDK JS to create a live streaming system for multiplayer games. In particular, I’d like to focus on the subject of Among Us, where the official policy on fan creation is very clear.

Specifically, we will create something like the figure below. Each user’s game screen is displayed in the center. We need to make this screen visible to the…


Note:
This article is also available here.(Japanese)
https://cloud.flect.co.jp/entry/2021/05/19/125034

Introduction

In my previous blog, I introduced how to run AI models for virtual backgrounds of Google Meet with Tensorflow Lite (TFLite) in a browser. So, in this article, I would like to introduce one of the applications of this technology, which is a lightweight, high-speed barcode scanner.

This is how it works. Multiple barcodes facing different directions can be read at high speed.

Lightweight Semantic Segmentation Model

The AI model used in the virtual background is called the Semantic Segmentation model. This model classifies what is in each pixel of an image. For example, as shown…


Note:
This article is also available here.(Japanese)
https://cloud.flect.co.jp/entry/2021/03/29/131955

Introduction

I previously posted an article on running Google Meet’s virtual background model (Segmentation Model) with Tensorflowjs. In this article, I will try to run it with TFLite, which is built as a Wasm, in order to further improve performance.

As a result, I did quite well.

Review of the previous works and performance improvement

In my previous article, I found that we could expect roughly 1.6x speedup compared to MediaPipe’s BodyPix, a commonly used virtual background feature. However, it seemed to be difficult to achieve the amazing speed of the original Google Meet. According to the Google AI blog, the…


Note:
This article is also available here.(Japanese)
https://cloud.flect.co.jp/entry/2021/03/01/130602

Introduction

In the previous article, we showed how to use Voice Focus in Amazon Chime SDK for JS to remove noise. In this article, I would like to take it a step further and show how to add BGM or SE after denoising. I guess this is the audio version of the virtual background that I have introduced several times before(ref).

What I will create is something like the video below. Instead of the sound of rain, I added some space-like BGM (00:16-). …


Note:
This article is also available here.(Japanese)
https://cloud.flect.co.jp/entry/2021/02/16/113455

Introduction

In my previous post, I showed how to achieve virtual backgrounds using the Video Processing API of Amazon Chime SDK. In this post, I intrduce noise suppression feature provided in Amazon Chime SDK, as known as Voice Focus. In this article, I would like to show how to use and then take a look at this effect.

This video shows the actual application of Voice Focus (00:11-). You can see that the noise is suppressed quite nicely.

Voice Focus

Voice Focus is a feature that has been available in Amazon Chime (application) since…


Note:
This article is also available here.(Japanese)
https://cloud.flect.co.jp/entry/2021/01/19/130105

Note: Additional verification on the M1 Mac has been added. I also updated BodyPix to reflect the change in processing time.[28/Mar./2021]

>See also: Build TFLite Wasm/SIMD and run Google Meet Virtual Background

Introduction

In my previous post, I showed how to achieve virtual backgrounds using the Video Processing API of Amazon Chime SDK. In this post, we used Body Pix for segmentation of people and backgrounds. This time, I would like to implement and evaluate the virtual background function using the Google Meet model, which is more lightweight.

Here is what we have…


Note:
This article is also available here.(Japanese)
https://zenn.dev/wok/articles/0009_ffmpeg-wasm-screen-recorder

Introduction

Until a while ago, I was a Linux user and I used ffmpeg to make gif animations for my Blog. I recently moved to Windows and was wondering how I could make a gif animation. I found out that I could make them with ffmpeg for Windows. Well, that was the conclusion I came to, but I don’t like the fact that adding many software to Windows makes it unstable (this is the prejudice of someone who came back to Windows after a long time). So, I decided to use ffmpeg.wasm, which…


Note:
This article is also available here.(Japanese)
https://cloud.flect.co.jp/entry/2020/12/29/152543

Introduction

Amazon Chime SDK JS version 2.3.0 was released on December 21, 2020.
In this release, The new wonderful feature was added, the Video Processing API!

https://github.com/aws/amazon-chime-sdk-js/blob/master/CHANGELOG.md

The Video Processing APIs is an APIs that allows you to edit your own camera’s video (frames) in video conferencing before it is transmitted. It can be used, for example, to create a virtual background that blurs the background of the camera.

From reading the official documentation, the way to achieve this seems to be similar to the way I introduced before. It’s finally official! That’s…

dannadori

Software researcher and engineer

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store