What the ONNX runtime does to a model

Most of what looks like Foundry Local behaviour is actually ONNX Runtime behaviour. The Foundry Local Core API handles model acquisition, caching, and lifecycle; the moment inference starts, it hands off. Which means the interesting questions — why this model chose the CPU, why the same binary is fast on one laptop and ordinary on another, why a downloaded file has to be converted before it will run at all — are questions about the runtime underneath.

The runtime is worth a chapter of its own, because it is the layer that repays understanding and the layer most teams skip.

ONNX is usually described as a model format, which is true and slightly misleading. What the format actually encodes is a computation graph: a set of operator nodes and the tensors flowing between them. The weights matter, but the graph is what the runtime reasons about.

That framing explains the behaviour that follows. When a model is loaded, ONNX Runtime does not execute nodes top to bottom. It analyses the graph first, applies optimizations, and then decides — node by node, subgraph by subgraph — which piece of hardware should run each part. A model is less a program to be interpreted than a plan to be scheduled.

The runtime also supports quantized models directly, which is the mechanism that makes on-device inference practical at all. Reduced precision cuts memory footprint and increases throughput on hardware that has no room for a full-precision model. Every model in the Foundry Local catalog has been through that process before it is ever downloaded.

An execution provider is a hardware acceleration library that ONNX Runtime can delegate work to. The runtime's extensibility model is built around them: instead of implementing every operator for every accelerator, it asks each provider what it can handle and allocates accordingly.

The mechanism has a name worth knowing, because it explains the partial-acceleration case that confuses people. The runtime calls a capability interface on each provider to determine which nodes or subgraphs that provider can execute. A model is therefore not "on the GPU" or "on the CPU" — it is partitioned, with supported subgraphs assigned to the accelerator and everything else left elsewhere.

Providers are supplied as a priority-ordered list, and the runtime falls back down that list. The last entry is always the same one:

That last row is the one to internalize. The CPU provider is always present, which means an on-device model degrades instead of breaking. A missing driver, an unsupported accelerator, a machine older than the install base was scoped for — none of these produce a crash. They produce a slower answer.

This is excellent engineering and a genuine operational hazard at the same time. A silent fall back to CPU looks identical to a working deployment right up until someone complains about latency. Any team shipping on-device inference across mixed hardware should be logging which provider was actually selected, not assuming the fast path was taken.

The fallback that guarantees a model always runs is the same fallback that hides a model running badly.

Apple silicon is the case that best rewards knowing the internals, because the obvious guess is wrong. There is no Metal execution provider. GPU acceleration on macOS goes through WebGPU, and WebGPU is implemented through Dawn, Google's native WebGPU implementation, which compiles WebGPU compute shaders into Metal Shading Language for execution on the GPU.

Three optimizations make that chain viable rather than merely clever: half-precision arithmetic for throughput, GPU-side tensor management to avoid shuttling data across the CPU boundary, and graph capture for repeated inference passes.

The payoff is portability. The same WebGPU provider targets Direct3D on Windows, which means one code path covers GPU acceleration on both platforms. The cost is a layer of indirection between the model and the metal — worth knowing about when profiling, and not worth worrying about otherwise.

Execution providers have to get onto the machine and be registered with the runtime before they can be used, and the answer to "who does that" differs by platform in a way that matters operationally.

On Windows, Foundry Local integrates with WinML, which sources hardware-matched provider plugins from the operating system and Windows Update, registers them with ONNX Runtime, and negotiates driver compatibility. Plugin providers download on first run and update themselves when new versions land. That is a real convenience and a real dependency: the acceleration story is partly owned by the OS update channel, not solely by the application.

On Linux and macOS, the Core API registers providers directly, and the SDK bundles the plugins it needs for each target platform. Registration happens internally during model loading. Fewer moving parts, and the application package carries more of the weight.

A detail that surfaces immediately in practice: the first run of foundry model list downloads the execution providers matched to the machine's hardware before it prints anything. That is not a hang. It is the runtime acquiring the acceleration layer it just discovered it needs.

The most common early friction has nothing to do with hardware. A model is sitting on disk in a format that works perfectly well with another local runtime, and Foundry Local shows no interest in it.

The reason is upstream of any bug. ONNX Runtime executes ONNX graphs. That is the contract. Weights distributed in another serialization — whatever its merits, and several are excellent — are not an ONNX graph, and no amount of configuration turns one into the other at load time. The conversion is a real build step with real inputs and outputs.

Microsoft's documented answer is to convert first: Olive takes Safetensors or PyTorch sources from Hugging Face and produces optimized ONNX targeted at a specific device and execution provider. Precision is chosen at that step — fp16, fp32, int4, int8 — not at run time.

Which reframes the whole complaint. The runtime is not refusing a model. It is declining to guess at a compilation step that has genuine choices in it — target device, target provider, target precision — and that a developer should be making deliberately. That step is the subject of the next episode.