Amazon Chime SDK Whiteboard with Data Messages for Real-time Signaling

This article is also available here.(Japanese)
https://cloud.flect.co.jp/entry/2020/06/01/115652

(NEW ARTICLE 20th/Oct./2020):
Faceswap and Virtual Background on your brower

In the last article, I explained how to create a virtual background for Amazon Chime SDK.

Virtual Background with Amazon Chime SDK & BodyPix

I’d like to continue talking about the Amazon Chime SDK in this post.

Well, did you know that Amazon announced a new feature addition to their Amazon Chime SDK the other day?

Amazon Chime SDK がリアルタイムシグナリング用のデータメッセージを追加

This feature allows participants in a conference to exchange data messages by using the data communication channel used by Amazon Chime. As mentioned in the announcement, this allows us to easily implement the whiteboards and emojis among participants in the conference room. And it can also be used to control the state of the conference room, such as forcing participants to mute.

So I’d like to show you how to make a whiteboard using this feature.

This is the behavior of the whiteboard I made this time.

Amazon Chime and Signaling

This additional features of the Amazon Chime SDK uses the signaling communication already existing in Amazon Chime. Amazon Chime’s video conferencing is achieved using a technology called WebRTC, and in WebRTC, signaling communication is used to control the session.

Specifically, WebRTC is used for P2P communication between browsers, and the signaling communication is used to identify the destination of the other party or to exchange keys for cryptographic communication in order to start this communication.

And, even though it is called P2P communication, it is necessary to go through a relay server called TURN when communicating over a firewall. The exchange of information about these routes is also done through signaling communication.

If you want to know more about WebRTC and its relationship with signaling, please refer to this page.

Amazon Chime provides managed relay servers and signaling channels to make it easy to start video conferencing in a variety of network environments. The new feature leverages the managed communication path for this signaling to allow arbitrary data messages to be exchanged. So developers can easily add things like shared whiteboards to their video conferencing systems without having to provide a server for messaging.

New API Overview

The three new methods offered are as follows

This function sends data messages with “Topic”.

First of all, each client registers a callback function that defines the process for each Topic. Then, when the sender sends a data message with Topic, the client receives the data message and calls the callback function corresponding to Topic. We don’t know the details of the internal processing, especially the data flow, but it’s probably running on a general publish/subscribe model.

After using it this time, I found it to be a very easy to use feature.

Note that this function may not be able to receive the data message even if the Publisher of the data message has subscribed to the topic of the data message, so you may need to be careful. I think the advantage of the publish/subscribe model is that it allows publishers and subscribers to completely ignore each other’s relationships, so the fact that publishers can’t receive data when they’re in the same software (session) was just a little off.

1. SendDataMessage
   https://aws.github.io/amazon-chime-sdk-js/interfaces/audiovideofacade.html#realtimesenddatamessage
2. Subscribe
   https://aws.github.io/amazon-chime-sdk-js/interfaces/audiovideofacade.html#realtimesubscribetoreceivedatamessage
3. Unsubscribe
   https://aws.github.io/amazon-chime-sdk-js/interfaces/audiovideofacade.html#realtimeunsubscribefromreceivedatamessage

Shared Whiteboard

Here’s the general process flow of the shared whiteboard we created.

1. Detects mouse events/touch events on the Publisher’s browser’s canvas (HTMLCanvasEelemnt) and identifies the coordinates.
2. Drawing on the Publisher’s canvas
3. Send coordinates to Broker(Chime) as a data message (real-timeSendDataMessage)
4. Sending coordinates from Broker to each Subscriber
5. Drawing on the canvas of each Subscribers

As mentioned earlier, Publisher cannot receive the data messages it sends, so it must draw on its own canvas before sending the data messages. When creating an application that wants to reflect user operations in the UI without delay, such as a whiteboard, it is better for the user experience to reflect them in the UI before sending data messages, so I think it will be similar regardless of whether Publisher can receive data messages or not.
Furthermore, since Publisher cannot receive the data messages it sends, it may be easier to implement because Publisher does not have to discard the received data.

Implementat

Subscribe

Here is an example of a wrapper function that registers a topic and a corresponding callback function to be subscribed by realtimeSubscribeToReceiveDataMessage. Here, we define a callback function that calls app.app.receivedDataMessage when we receive a data message and use it as an argument. Please note that app.app.receivedDataMessage itself can be defined elsewhere for arbitrary processing.

export const setRealtimeSubscribeToReceiveDataMessage = (app:App, audioVideo:AudioVideoFacade, topic:string) =>{
    const receiveDataMessageHandler = (dataMessage: DataMessage): void => {
        app.receivedDataMessage(dataMessage)
    }
    audioVideo.realtimeSubscribeToReceiveDataMessage(topic, receiveDataMessageHandler)
}

Send DataMessage

This is an example of how to send a data message using realtimeSendDataMessage.
In order to draw on the whiteboard, the coordinates of the starting and ending points, stroke information, line thickness, etc. are JOSNized and sent.

    sendDrawsingBySignal = (targetId: string, mode:string, startXR:number, startYR:number, endXR:number, endYR:number, stroke:string, lineWidth:number)=>{
        const gs = this.props as GlobalState
        const message={
            action: 'sendmessage',
            data: JSON.stringify({ 
                cmd         : MessageType.Drawing,
                targetId    : targetId, 
                startTime   : Date.now(),
                mode        : mode,
                startXR     : startXR,
                startYR     : startYR,
                endXR       : endXR,
                endYR       : endYR,
                stroke      : stroke,
                lineWidth   : lineWidth
            })
        }
        gs.meetingSession?.audioVideo.realtimeSendDataMessage(MessageType.Drawing.toString(), JSON.stringify(message))
    }

Demo

WhiteBoard

This is how the whiteboard function you created works. This demo will be a simulated classroom whiteboard. You can see that what you draw on the right side is reflected on the left side of the screen.

WhiteBoard with SharedDisplay

You can also create this whiteboard as an overlay so you can use it with the Amazon Chime SDK’s screen sharing feature to give a presentation.

Code

The features described in this article are built into a test bed of new features using video conferencing.
If you are interested in it, please visit the following repository.

FLECT-DEV-TEAM/FLECT_Amazon_Chime_Meeting

Finally

This time, I tried to create a whiteboard using the latest features of Amazon Chime SDK.

In Japan, it was recently announced that the state of emergency has been lifted. However, it still seems difficult to get many people in the classroom to teach a lesson. Also, face-to-face customer service can be risky and difficult to do in the same way. I think that video conferencing and shared whiteboards may be an option to address these issues.

I am very thirsty!!