Talking avatar with Zoom Meeting SDK

Note:
This article is also available here.(Japanese)
https://qiita.com/wok/items/0450c8620f11a371bd8b

Introduction

Zoom has begun offering an SDK. It’s a lot of excitement. I took the opportunity to experiment with this and will be posting several in a row, as it looked like it could do a lot of interesting things. Please see subsequent posts if you are interested.

Related articles are
(1) Time keeper bot with Zoom Meeting SDK
(2) Talking avatar with Zoom Meeting SDK ←This article
(3) Motion tracking avatar with Zoom Meeting SDK
(4) Voice changer with Zoom Meeting SDK
(5) Use avatar you like in Zoom !
(6) Real-time transcription of Zoom meetings with Zoom Meeting SDK and Vosk browser

I will try to make my avatar talk. The finished product looks like this.

Japanese version is here.

For now, we have it as a Docker image, so you can download and use it.

Note that we are using VoiceVox and OpenTTS for the voice. VoiceVox has a very cute and high quality voice, which is hard to believe that it is free. OpenTTS probably also has good English pronunciation, but I am not sure.

$ docker run -p 8888:8888 dannadori/zoom-meeting-plus:v02 
$ docker run --rm -it -p '127.0.0.1:50021:50021' voicevox/voicevox_engine:cpu-ubuntu20.04-latest 
$ docker run -it -p 5500:5500 synesthesiam/opentts:en

The operation is as follows.

Architecture

This is almost the same configuration as the timekeeper described in the previous blog. In the case of Timekeeper, the audio was fixed at the end time notification voice. This time, the text input by the user is converted to voice data and played in the stream. We use VoiceVox to convert text to voice data for Japanese. The fact that it is free of charge with such high quality is astounding. For other languages, we use OpenTTS.

Repository

The demo shown here is available in the following repositories

https://github.com/w-okada/zoom-meeting-plus

The latest version may not be stable as we are constantly updating it.
Please clone and use the following tag.

$ git clone https://github.com/w-okada/zoom-meeting-plus -b v02

I need coffee!!

Disclaimer

In no event shall we be liable for any direct, indirect, consequential, or special damages arising out of the use or inability to use the software on this blog.