TensorFlow.js lets you build machine learning projects from zero. When necessary data is available, then the models can be trained and executed directly in the browser. TensorFlow.js uses the graphics card (GPU) of the computer via the browser API WebGL. You lose a bit of performance because WebGL can only be tricked into executing the matrix multiplication required by TensorFlow.js. But they are necessary because TensorFlow, as a machine learning strategy, mainly supports neural networks. These can be mapped very well by matrix multiplications during training as well as during prediction. Here is the first advantage of TensorFlow.js over TensorFlow: While TensorFlow currently only supports NVIDIA GPU over CUDA, TensorFlow.js works with any graphics card.
Here is the example code to use the High Level API to create a sequential neural network in the browser:
// create a sequential model
const model = tf.sequential();
// add a fully connected layer with 10 units (neurons)
// add a convolutional layer to work on a monochrome 28x28 pixel image with 8
// filter units
inputShape: [28, 28, 1],
// compile the model like you would do in Keras
// the API speaks for itself
Interact with all browser APIs
Addressing interfaces on different operating systems and devices can still be a painful experience. Not so when developing a browser-based application. Even access to such complex hardware as a camera or a microphone are anchored in the HTML standard and are supported by all current browsers. The nature of the browser, which is naturally designed for interaction as well, suits you here. Interactive applications with a share of machine learning are now easier than ever.
Machine learning without installation on any computer
TensorFlow.js allows you to deploy a model created in advance with TensorFlow. This model may already have been fully or partly trained on strong hardware. In the browser it comes then only to the application or is further trained.
In the next example you can play Pac-Man using images trained in your browser through your webcam.
The model is converted by a supplied program and can be loaded asynchronously after being loaded by a line similar to the following:
const model = await tf.loadModel('model.json');
After that, the model is no longer distinguishable from a model created directly in the browser. It is very easy to use for prediction, which in turn runs asynchronously on the GPU:
const example = tf.tensor([[150, 45, 10]]);
const prediction = model.predict(example);
const value = await prediction.data();
In addition to entertainment through games even more useful applications are conceivable here. Gesture navigation or interaction can be helpful to people with disabilities as well as people in special situations. And as I said: without any installation, by simply loading a website.
import * as posenet from '@tensorflow-models/posenet';
import * as tf from '@tensorflow/tfjs';
// load the posenet model
const model = await posenet.load();
// get the poses from a video element linked to the camera
const poses = await model.estimateMultiplePoses(video);
// poses contain
// - confidence score
// - x, y positions
User data does not have to leave the browser
Nowadays, where privacy due to GDPR regulation gets more and more of importance, some think twice, whether he wants to have a particular cookie on his computer or whether you want to send a statistic of his user data for improving the user experience to the manufacturer of a software. But what about the reversal? The manufacturer provides a general model of how to use software, and similar to the Pac-Man game, it is adapted to the individual user using a transfer-learning model.
Machine learning in the browser does not seem to make much sense at first for many developers. But if you take a closer look, there are applications that no other platform can offer:
- Education: You can interact directly with machine-learning concepts and learn by experimenting
- Development: If you already have or want to build a JS application, you can use machine learning models or practice it directly
- Games: Real-Time Position Estimation only via the camera (ie how the people in front of the camera are moving) or Image Recognition can be coupled directly with games. There are very cool examples of what you can do with it. However, the possibilities go well beyond games
- Deployment: You already have a machine learning model and wonder how to get it into production. The browser is a solution. Even finished models can be integrated into the own application without deep knowledge of machine learning
- Interactive visualizations: For interactive clustering or artistic projects
As we see, we still have some drawbacks to the same hardware in terms of performance over TensorFlow. The comparison runs on a 1080 GTX GPU and measures the time for a prediction with the MobileNet, as it is used for the examples here. In this case, TensorFlow runs three to four times faster than TensorFlow.js; however, both values are very low. The WebGPUS standard, which will allow more direct access to the GPU, gives hope for better performance.
Tutorials can be found on the TensorFlow.js website.