AI at the edge
Nine milliseconds, always, beats a hundred and eighty when the link is up
A model in a vehicle or on a machine cannot wait for a network. The budget is fixed before anything is chosen, and the model shrinks until it fits.
Why the device and not the cloud
Three reasons, and they compound. Latency, because a control loop that has to cross a network has a worst case measured in seconds rather than milliseconds. Availability, because a machine that stops working when the connection drops is a machine nobody trusts. And data, because the cheapest way to keep something private is not to transmit it.
The cost is that everything is now a budget. Memory, compute and power are fixed by the hardware you can actually buy, and the model has to fit inside what is left after the rest of the product has taken its share.
The budget
The model shrinks until the budget fits
The order of work is the opposite of the usual one. Measure what the loop has to do and how often, subtract what the rest of the firmware needs, and what remains is the model's allowance.
Then the model is fitted to that allowance through quantisation, pruning, or picking a smaller architecture and training it properly. Each step is measured against the evaluation set, so you can see exactly what accuracy was traded for what milliseconds.
The result is usually smaller than people expect and better than they fear. A model that has to be right about a narrow thing does not need to be large.
The choices
What the hardware decides for you
The chip is chosen early and it is hard to change, which makes this the expensive row to get wrong.
| Constraint | What it fixes | Where it usually binds first |
|---|---|---|
| Latency budget | How much compute a single inference may use | Control loops, where the deadline is hard |
| Memory | How large the model may be at all | Small microcontrollers, long before compute does |
| Power | How often it may run | Anything on a battery |
| Thermal | Sustained rate rather than peak rate | Sealed enclosures, and it is always found late |
The last row is the one that surprises people. A part that runs fast on a bench runs slower in a closed box, and the box is usually already tooled.
Or write here
Three fields, and a person reads it
Name, an address we can answer to, and what you are trying to build. You get a reference back straight away.
What has to decide without a network?
Tell us the deadline and the hardware. Those two settle whether this is possible before anybody writes code.