Motion tracking avatar with Zoom Meeting SDK
This article is also available here.(Japanese)
Zoom has begun offering an SDK. It’s a lot of excitement. I took the opportunity to experiment with this and will be posting several in a row, as it looked like it could do a lot of interesting things. Please see subsequent posts if you are interested.
Related articles are
(1) Time keeper bot with Zoom Meeting SDK
(2) Talking avatar with Zoom Meeting SDK
(3) Motion tracking avatar with Zoom Meeting SDK ←This article
(4) Voice changer with Zoom Meeting SDK
(5) Use avatar you like in Zoom !
(6) Real-time transcription of Zoom meetings with Zoom Meeting SDK and Vosk browser
Previously, the avatar participated in meetings as a bot, but this time we will capture the user’s facial expressions and synchronize them with the avatar ’s movements. So you can use the avatar to participate in the meeting.
The finished product looks like as below. This time, since no audio is used and there is no concern about feedback, I put the participants’ screens side by side. Here, I use video instead of camera video. (We support both camera video and video).
The AI recognizes facial expressions and poses, so there are some irregularities, but I think it works reasonably well. In particular, the reading of facial expressions is quite good, so you may want to try to capture only the face.
For now, we have made it into a Docker image, so please download and use it.
In addition, if you wish to continue using voice as in the previous article, please also activate VoiceVox and OpenTTS. For details, please refer to the previous article.
$ docker run -p 8888:8888 dannadori/zoom-meeting-plus:v03
$ docker run --rm -it -p '127.0.0.1:50021:50021' voicevox/voicevox_engine:cpu-ubuntu20.04-latest
$ docker run -it -p 5500:5500 synesthesiam/opentts:en
You can use it by accessing http://localhost:8888/.
Note that it is activated via http and must be used locally.
The operation is as follows
The demo shown here is available in the following repositories
The latest version may not be stable as we are constantly updating it.
Please clone and use the following tag.
$ git clone https://github.com/w-okada/zoom-meeting-plus -b v03
In no event shall we be liable for any direct, indirect, consequential, or special damages arising out of the use or inability to use the software on this blog.