An AI lab that runs in your browser

Josep Castells, , 3 min read

Cover with the Lab figures: 93.2% for the Try-on, 97.1% for Voice and 0.428 mAP for the detector
Image: Togent

Machine learning demos with their real figures, running on your device without sending anything to a server.

When a company asks me whether artificial intelligence is useful to them, the honest answer rarely fits in a presentation. You need to see it working with real data, see where it gets things right and, above all, where it fails. That is why I built the Lab: a collection of models that run entirely in your browser, with no server in between.

What is inside

Each demo is a real model with its own model card: what data it was trained on, how accurate it is and under what conditions that was measured.

  • [Try-on](/laboratorio/probador). You draw a piece of clothing or show one to the camera, and three different architectures (a dense network, a convolutional network and a vision transformer) tell you what they see. The convolutional network gets 93.2% of the images in the Fashion-MNIST test set right, compared with 87.7% for the dense network. It also shows a heat map of where the model is looking and which neighbourhood of the latent space your drawing falls into.
  • [Voice](/laboratorio/voz). A model with 73,004 parameters recognises ten English words, silence and "another word". It scores 97.1% on the Speech Commands test set and 94.8% when background noise is added. You use it to steer a snake with your voice, and you can check how well it understands your accent.
  • [Detection](/laboratorio/deteccion). Eighty types of object from the camera, with tracking across frames, a line-crossing counter and a privacy blur. It uses D-FINE-N, a detector of about four million parameters under the Apache 2.0 licence. Its mAP on COCO is 0.428; that figure comes from the author's model card, I have not measured it myself, and the page says so.
  • [Self-driving](/laboratorio/conduccion-autonoma). A lidar point cloud with its 3D boxes, the segmentation and the six cameras from PandaSet, with faces and number plates blurred before publishing.
  • Ingredients. A detector of my own trained on food classes from Open Images. It is not fully trained yet, so its page says "in preparation" and is not indexed.

Why in the browser

The models are downloaded once from this domain and inference runs on your device with ONNX Runtime Web: on the graphics card if the browser supports WebGPU, and otherwise on the processor with WebAssembly. The camera, the microphone and any photos you use never leave your browser.

There are three reasons for doing it this way:

  1. Privacy with no small print. If the image never leaves the device, there is nothing to safeguard on a server.
  2. Cost. The server behind this website has no graphics card. Running the models on your device is what makes it possible to offer the demos without charging or limiting use.
  3. Honesty. These are not screenshots or recorded videos: if a model gets your drawing or your accent wrong, you see it live.

Nothing is downloaded when you open the page. Each demo tells you how large its model is and waits until you ask for it: the Try-on, about 6.7 MB; the Voice, less than 400 KB.

What I learned building it

Measure before you show. None of the figures on the model cards is typed in by hand: they come from the metrics files that each training run generates, with the official test set evaluated only once. When I have not measured a figure myself, like the detector's mAP, the card says so.

Exports fail too. The published ONNX versions of D-FINE gave broken results, so I exported it again from PyTorch. Before publishing it I compared the reduced versions (fp16 and int8) with the original on 46 images: they match on practically every detection.

The licence decides before accuracy does. The best-known family of detectors is licensed under the AGPL, which would force me to open-source this website. I explain it in detail in Data and models you can use on a commercial website.

What it means for a business

The Lab is not a product: it is the test bench where I learn what works outside a slide. Camera-based detection, for example, is the basis for very concrete things in a warehouse or industrial unit: counting, flagging that something is where it should not be, or anonymising images before storing them. If you have a case like that, we start with a free diagnosis and see whether it makes sense.

More from the blog

Shall we look at it in your business?

Tell us which process eats your hours and we will see whether AI solves it, with a free assessment.