This article is also available here.(Japanese)
Zoom has begun offering an SDK. It’s a lot of excitement. I took the opportunity to experiment with this and will be posting several in a row, as it looked like it could do a lot of interesting things. Please see subsequent posts if you are interested.
Related articles are
(1) Time keeper bot with Zoom Meeting SDK
(2) Talking avatar with Zoom Meeting SDK
(3) Motion tracking avatar with Zoom Meeting SDK
(4) Voice changer with Zoom Meeting SDK
(5) Use avatar you like in Zoom ! ←This article
(6) Real-time transcription of Zoom meetings with Zoom Meeting SDK and Vosk browser
In terms of data flow to Zoom, there are input and output, input from the microphone and camera, and output from the speakers. In previous articles, we have experimented mainly how to handle the input. From next article I will to see if we can somehow tweak the output.
Before talking about the output side, however, I would like to introduce two minor features that have been added to the Zoom Meeting SDK.
The first is the motion capture record function. This allows you to reuse what you have recorded. The second is the ability to replace avatars. You can use any avatar you like.
motion capture record
It works like the following video, where once motion capture is done, it can be used again.
The operation is shown in the following figure. upper body capture is quite unstable, so you may want to try it out and disable it if you find it difficult.
Needs improvement, but I’m hoping for a mediapipe official.(ref)
Avatar replacement function
Open the settings dialog from the gear icon in the upper right corner. click on the avatar icon and you will see a button to load a vrm file. click on it and load the file of your choice. You can use avatars other than animals!
The operation is shown in the following figure.
Again, in addition to creating a Docker image, we are also deploying to Heroku. In the meantime, please try using it.
$ docker run -p 8888:8888 dannadori/zoom-meeting-plus:v05
$ docker run --rm -it -p '127.0.0.1:50021:50021' voicevox/voicevox_engine:cpu-ubuntu20.04-latest
$ docker run -it -p 5500:5500 synesthesiam/opentts:en
Heroku will stop if there is no access for a certain period of time, and it will take some time to restart. Please be patient until the application starts. (It will probably take less than a minute.)
The demo shown here is available in the following repositories
The latest version may not be stable as we are constantly updating it.
Please clone and use the following tag.
$ git clone https://github.com/w-okada/zoom-meeting-plus -b v05
In no event shall we be liable for any direct, indirect, consequential, or special damages arising out of the use or inability to use the software on this blog.