Vision

Vector has a far better vision system than the robot’s public interface suggests. Understanding which parts reach a server, and which never leave the robot, is most of the work of building anything that watches.

Everything on this page was read from the firmware source and the protocol definitions rather than measured on a robot.

The camera and the pipeline

The camera is 1280×720 colour. Frames arrive from the camera daemon into the engine process, where each is wrapped in a cache that will lazily hand out grey or colour at full, half (640×360), quarter or eighth size, so a cheap detector can ask for a cheap image.

There is no fixed frame rate. A dedicated vision thread processes one frame at a time and waits for the next; frames that arrive while it is busy are dropped and counted. Throughput is whatever the enabled detectors leave over.

Detectors are switched on by subscription, not by a global flag. Behaviours and any connected client subscribe to a named vision mode at a frequency, and a scheduler merges every subscription into a per-frame plan using a table of per-mode costs. Only one mode is on by default — marker detection, every fourth frame. Faces are requested by so much of the robot’s own behaviour tree that they are effectively always on while he is awake.

Anything a connected client asks for is refused unless that client holds behaviour control, and every client-requested mode except image streaming is dropped the moment control is released. Modes the robot’s own behaviours request run regardless.

What is on board

Faces

A commercial face library, statically linked: detection, tracking, landmarks, recognition, smile, gaze, blink and expression. It produces a face observation with an identifier, a 3D pose, an image rectangle, a name once enrolled, an expression from a fixed set, a histogram of expression confidences, and landmark points for both eyes, the nose and the mouth.

Recognition runs asynchronously, which is why the identifier for one person flickers between a negative tracking id and their enrolled id from frame to frame. Anything consuming face events has to smooth that; Ember’s brain does, preferring an enrolled match over stranger noise.

The album holds twenty named faces with ten entries each. It lives on the robot and survives reboots.

Pets

A separate commercial detector finds cat and dog faces — not bodies. Up to four at a time, face sizes between 60 and 240 pixels. The robot’s own behaviour tree only subscribes to it while he is exploring, so out of the box he looks for cats only when he has gone wandering.

It produces a pet observation with a type, a score and a rectangle. None of it reaches a server on stock firmware; there is no message for it in the robot’s external protocol. Exposing it is a firmware change, which is what EmberOS does.

Markers

Anki’s own printed fiducial markers, on the cube faces, the charger and any custom object sheets. This is the only detector on by default, and it is how he finds his charger to dock. Marker sightings feed an object world that localises things and broadcasts observations. Reaches the server as object events.

The cube’s own events — tapped, moved, turned over, connected — come from the cube’s accelerometer over Bluetooth, not from the camera.

Motion

Frame-to-frame brightness ratio per pixel, with a ground-plane centroid and three peripheral regions, driven by an impulse-decay activation so a single flicker does not register. Output is an image centroid and area fraction, the same point projected onto the ground in millimetres relative to the robot, and per-region centroids and areas.

The robot sends this. Older server builds could not read it, because the vendored protocol stubs stopped short of the field. Ember regenerated those stubs from the firmware’s own definitions, which is what made motion, enrolled face changes, camera settings and unexpected movement readable.

Laser dot, illumination, bright colours

A laser-dot detector looks for a small very bright spot and maps it to the ground plane. An illumination classifier decides “lights on” or “lights off” from brightness percentiles over a short window of frames — that is what powers his reaction to someone turning the lights off. A bright-colour detector exists and nothing subscribes to it.

None of these are in the stock external protocol.

Person and hand

Two small TensorFlow Lite models run asynchronously on the robot: a person detector on a 128×128 input over a 6×6 grid, and a hand classifier. Their output is a “salient point” with a position, a score and an area fraction.

They are built in and working, but they have few customers: the person detector runs when you say “come here” or ask for a photo, not continuously. Salient points have no message in the stock external protocol either.

There is also a dormant offboard path in the firmware: the engine can write a frame to a cache directory and poll for a JSON list of detections computed elsewhere, feeding them back as first-class perceptions. The helper that would answer is a no-op in the shipping build.

What the camera does not do

Drop-off detection is not vision. It is four downward-facing infrared sensors. Obstacle and range detection is a single forward time-of-flight sensor, reported in every robot state message along with the status bits for picked up and cliff detected. The camera contributes a memory of where edges were, not collision avoidance.

What reaches Ember

On stock firmware, the useful list is short: face observations, face id changes, object (marker) sightings, photos, robot state including the proximity sensor, and the camera feed itself. Motion, enrolled-face changes and camera settings are sent by the robot but need current protocol stubs to decode. Pets, laser, illumination and salient points do not leave the robot at all.

EmberOS closes most of that gap in two ways:

• New events in the firmware’s external protocol for pets, laser points, illumination and salient points, converted the same way motion already was.
• Always-on detectors, configured by a file on the robot rather than requested by a client holding behaviour control. This is the mission applied to the robot itself: the detections flow, and whatever wants them subscribes.

Everything that arrives is re-emitted by the server as vision.* events. Rate limits are applied per robot so a face in view does not produce thirty events a second.

The server-side recogniser

The robot can tell you there is a face, a cat, or something moving. It cannot tell you that the thing you are holding up is your keys. Ember adds a second, separate layer for that, and labels it plainly as the server’s opinion rather than the robot’s.

• Model: the image half of CLIP ViT-B/32, exported to ONNX and quantised to int8, about 85 MB, running on the CPU. Roughly 16 ms per crop in a batch, and about 0.4 seconds for a 640×360 frame’s twenty-odd candidate windows on a modest container. A general-purpose embedding was chosen over an ImageNet classifier because the point is to learn an arbitrary object from a handful of examples.
• Teaching: the dashboard’s Teach tab saves labelled crops. Each example is embedded once and cached, and the index is rebuilt in the background after every new snapshot.
• Matching: cosine similarity to the label’s centroid and to its nearest example, averaged. A hit needs to clear an absolute threshold and to stand out from the other windows in the same frame by a margin. Without that margin the background alone scores 0.85 to 0.9 against a crop taken in the same room, and everything matches everything.
• Candidates: the robot’s own boxes from the last second — face, object, motion, pet, salient — padded slightly, plus a small three-scale grid so an object held up in front of him is found even when the robot has no box for it.

The model is downloaded once by the installer and is not in the repository. Results are drawn on the live view as a separate dashed layer labelled as the server’s, so the two opinions are never confused.