Implement semantic segmentation with npm

Advance preparation

We’ll create a demo with React.
First, let’s set up the environment.

$ create-react-app demo --typescript

Install package

Install semantic segmentation package.

$ npm install scalable-semantic-segmentation-js
$ node node_modules/scalable-semantic-segmentation-js/bin/install_worker.js public
file is copied
$ ls public/WEB_MODEL/300x300_0.10/
group1-shard1of1.bin model.json

Source code

When you are ready, create the source code. Here, I will only explain the parts that I think are important. The entire source can be found in the following repositories.

scalableSS:ScalableSemanticSegmentation = new ScalableSemanticSegmentation()
componentDidMount() {
console.log('Initializing')

const initWorkerPromise = this.initWorker() // <-- (1)

if (navigator.mediaDevices && navigator.mediaDevices.getUserMedia) { // <-- (2)
const webCamPromise = navigator.mediaDevices
.getUserMedia({
audio: false,
video: DisplayConstraintOptions[this.state.videoResolution]
})
.then(stream => {
console.log(this.videoRef)
this.videoRef.current!.srcObject = stream; // <-- (3)
return new Promise((resolve, reject) => {
this.videoRef.current!.onloadedmetadata = () => {
resolve();
};
});
});

Promise.all([initWorkerPromise, webCamPromise])
.then((res) => {
console.log('Camera and model ready!')
})
.catch(error => {
console.error(error);
});
}
}
async initWorker() {
// SemanticSegmentation
this.scalableSS.addInitializedListener(()=>{ // <-- (1-1)
const props = this.props as any
this.setState({initialized:true})
this.requestScanBarcode() // <-- (1-2)
})
this.scalableSS.addMaskPredictedListeners((maskBitmap:ImageBitmap)=>{// <-- (2-1)
// 再キャプチャ
this.requestScanBarcode() // <-- (2-2)

})

this.scalableSS.init(
AIConfig.SS_MODEL_PATH,
AIConfig.SPLIT_WIDTH,
AIConfig.SPLIT_HEIGHT,
AIConfig.SPLIT_MARGIN) // <-- (3)
return
}
requestScanBarcode = async () => {
console.log('requestScanBarcode')
const video = this.videoRef.current!
const controller = this.controllerCanvasRef.current!
controller.width = this.overlayWidth
controller.height = this.overlayHeight

const captureCanvas = captureVideoImageToCanvas(video) <--(1)
if(captureCanvas.width === 0){
captureCanvas.remove()
window.requestAnimationFrame(this.requestScanBarcode);
return
}
this.scalableSS.predict(captureCanvas,
this.state.colnum,
this.state.rownum) <--(2)
captureCanvas.remove()
}
<Label basic size="tiny" color={this.state.showSS?"red":"grey"} onClick={()=>{
const newValue = !this.state.showSS
this.scalableSS.previewCanvas = newValue ? this.workerSSMaskMonitorCanvasRef.current! : null
this.setState({showSS:newValue})
}}>ss</Label>
<Label basic size="tiny" color={this.state.showGrid?"red":"grey"} onClick={()=>{
const newValue = !this.state.showGrid
this.scalableSS.girdDrawCanvas = newValue ? this.controllerCanvasRef.current! : null
this.setState({showGrid:!this.state.showGrid})
}}>grid</Label>

Demo

Github repository and npm package

This source code is stored in the following repositories.

I am very thirsty!!

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store