Note:
This article is also available here.(Japanese)
https://cloud.flect.co.jp/entry/2021/07/21/113845

Introduction

The Olympic Games are just around the corner, and e-sports is starting to get a lot of attention in the digital world. In a related vein, I’d like to use Amazon Chime SDK JS to create a live streaming system for multiplayer games. In particular, I’d like to focus on the subject of Among Us, where the official policy on fan creation is very clear.

Specifically, we will create something like the figure below. Each user’s game screen is displayed in the center. We need to make this screen visible to the…


Note:
This article is also available here.(Japanese)
https://cloud.flect.co.jp/entry/2021/05/19/125034

Introduction

In my previous blog, I introduced how to run AI models for virtual backgrounds of Google Meet with Tensorflow Lite (TFLite) in a browser. So, in this article, I would like to introduce one of the applications of this technology, which is a lightweight, high-speed barcode scanner.

This is how it works. Multiple barcodes facing different directions can be read at high speed.

Lightweight Semantic Segmentation Model

The AI model used in the virtual background is called the Semantic Segmentation model. This model classifies what is in each pixel of an image. For example, as shown…


Note:
This article is also available here.(Japanese)
https://cloud.flect.co.jp/entry/2021/03/29/131955

Introduction

I previously posted an article on running Google Meet’s virtual background model (Segmentation Model) with Tensorflowjs. In this article, I will try to run it with TFLite, which is built as a Wasm, in order to further improve performance.

As a result, I did quite well.

Review of the previous works and performance improvement

In my previous article, I found that we could expect roughly 1.6x speedup compared to MediaPipe’s BodyPix, a commonly used virtual background feature. However, it seemed to be difficult to achieve the amazing speed of the original Google Meet. According to the Google AI blog, the…


Note:
This article is also available here.(Japanese)
https://cloud.flect.co.jp/entry/2021/03/01/130602

Introduction

In the previous article, we showed how to use Voice Focus in Amazon Chime SDK for JS to remove noise. In this article, I would like to take it a step further and show how to add BGM or SE after denoising. I guess this is the audio version of the virtual background that I have introduced several times before(ref).

What I will create is something like the video below. Instead of the sound of rain, I added some space-like BGM (00:16-). …


Note:
This article is also available here.(Japanese)
https://cloud.flect.co.jp/entry/2021/02/16/113455

Introduction

In my previous post, I showed how to achieve virtual backgrounds using the Video Processing API of Amazon Chime SDK. In this post, I intrduce noise suppression feature provided in Amazon Chime SDK, as known as Voice Focus. In this article, I would like to show how to use and then take a look at this effect.

This video shows the actual application of Voice Focus (00:11-). You can see that the noise is suppressed quite nicely.

Voice Focus

Voice Focus is a feature that has been available in Amazon Chime (application) since…


Note:
This article is also available here.(Japanese)
https://cloud.flect.co.jp/entry/2021/01/19/130105

Note: Additional verification on the M1 Mac has been added. I also updated BodyPix to reflect the change in processing time.[28/Mar./2021]

>See also: Build TFLite Wasm/SIMD and run Google Meet Virtual Background

Introduction

In my previous post, I showed how to achieve virtual backgrounds using the Video Processing API of Amazon Chime SDK. In this post, we used Body Pix for segmentation of people and backgrounds. This time, I would like to implement and evaluate the virtual background function using the Google Meet model, which is more lightweight.

Here is what we have…


Note:
This article is also available here.(Japanese)
https://zenn.dev/wok/articles/0009_ffmpeg-wasm-screen-recorder

Introduction

Until a while ago, I was a Linux user and I used ffmpeg to make gif animations for my Blog. I recently moved to Windows and was wondering how I could make a gif animation. I found out that I could make them with ffmpeg for Windows. Well, that was the conclusion I came to, but I don’t like the fact that adding many software to Windows makes it unstable (this is the prejudice of someone who came back to Windows after a long time). So, I decided to use ffmpeg.wasm, which…


Note:
This article is also available here.(Japanese)
https://cloud.flect.co.jp/entry/2020/12/29/152543

Introduction

Amazon Chime SDK JS version 2.3.0 was released on December 21, 2020.
In this release, The new wonderful feature was added, the Video Processing API!

https://github.com/aws/amazon-chime-sdk-js/blob/master/CHANGELOG.md

The Video Processing APIs is an APIs that allows you to edit your own camera’s video (frames) in video conferencing before it is transmitted. It can be used, for example, to create a virtual background that blurs the background of the camera.

From reading the official documentation, the way to achieve this seems to be similar to the way I introduced before. It’s finally official! That’s…


Note:
This article is also available here.(Japanese)
https://zenn.dev/wok/articles/0007_u2net-portrait-drawing

At First

I want to easily run machine learning models in a browser. So I’m working on a library to run various models in tensorflowjs(see this repository). In this article, I will try to run the function to generate a portrait from a photo in a browser.

For example, you can generate the picture on the right from the picture on the left, like this.( As usual, the image on the left was borrowed from Pakutaso. This is Kawamura-san. She’s always cute. )

A little about U²-Net

This time we will use U²-Net to generate a portrait. …


Note:
This article is also available here.(Japanese)
https://zenn.dev/wok/articles/0006_convert-from-pytorch-to-tfjs

At First

I want to easily run machine learning models in a browser. So I’m working on a library to run various models in tensorflowjs(see this repository). However, as you can see in the graph below, PyTorch has recently been used in 75% of the papers at major conferences, making it impossible to easily run edgy and interesting models with tensorflowjs.

State of AI Report 2020 (site)

In this situation, PINTO, an authority on model quantization, published a method for converting Pytorch to Tensorflow models at this year’s Advent Calender. Thanks for a very wonderful article.

When I tried to…

dannadori

Software researcher and engineer

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store