Anytime Inference Planner

Static ML serving policies force a bad tradeoff. Always running the large FP32 model is accurate but sheds most requests once traffic passes its service limit. Always running the small INT8 model has headroom but loses accuracy you did not need to lose. In practice the right precision depends on the current load and the current request's deadline. That is a control problem, not a "pick a model at deploy time" problem.

Anytime Inference treats variant selection as a constrained optimisation: maximise expected accuracy subject to a deadline, with the budget enforced against an M/M/c sojourn-time bound and the measured backlog.

A Python control plane watches queue depth and CPU load and routes each request to one of several profiled model variants. Inference runs in-process in a C++ ONNX Runtime engine.

• Variant profiles. Service-time distributions measured per variant through the serving path itself, not estimated from FLOPs. Every variant is cross-checked against a separate ONNX Runtime session.
• Load signal. Queue depth plus a smoothed CPU-load estimate. Backlog is what the admission math actually consumes.
• Selector. Picks the highest-accuracy variant whose expected sojourn time still admits under the deadline given the current backlog.
• Engine. In-process C++ ONNX Runtime holding warm sessions for every variant, bound through pybind11. A pure-Python backend is used automatically when the extension is not built.

DistilBERT-SST-2 and MiniLM-L6 from a 4-worker pool on an Apple M4 Pro, real SST-2 validation traffic, 39 ms deadline. Offered load is a fraction of the measured pool capacity of 310 rps. Goodput counts only requests that completed inside the deadline; p95 is the adaptive policy's.

At 95% of capacity that is 6.5x the goodput at 49% of the compute cost, for at most 0.92 accuracy points — the gap between the variants is 91.06% against 90.14% on SST-2 validation, and only the shifted fraction of traffic pays it.

Below ρ ≈ 0.5 the two policies are indistinguishable: every request stays on the most accurate variant and the adaptive policy matches the baseline exactly. It only starts trading once the queue says it has to.

Repeating the sweep moved goodput by at most 6%, and service times carry a 1-4% run-to-run spread, so the third digit is noise.

A second lane decodes. GPT-2 124M is exported with its KV cache in the graph signature, and that cache is held in a fixed arena of blocks — so admission can refuse a sequence it cannot hold, and eviction can pick a victim on evidence rather than on hope.

Time to first token and time per output token are ~35x apart at FP32, so they are never reported as one number. A 1024-token prompt is prefilled in chunks of 256; TPOT is measured against 960 cached tokens, where a decode step costs the most. Perplexity is WikiText-2 over 32,736 tokens.

INT8 is the variant to serve here, and it does not dominate: it is fastest in both phases, but INT4 is smaller and FP32 is 0.06 perplexity better. It also reverses the encoder finding on this host, where INT8 was strictly slower — and reversing is not overturning. Model, shape, and quantisation recipe all differ between the two measurements, so what it shows is that the encoder conclusion does not generalise, not which difference caused that.

What the fitted cost model does isolate is narrower and more useful. Precision moves the cache-independent part of a decode step — 4.71 ms at FP32 against 3.06 at INT8 — and barely touches the part that grows with the cache, 3.62 against 4.37 µs per cached token. That is why INT8's lead narrows from 28% at 128 cached tokens to 11% at 960: the part it shrinks is a shrinking share of the step.

The arena is not a speedup and is not offered as one. Feeding the graph's own present tensors straight back costs no gather at all. What blocks buy is an occupancy number a policy can act on, and the last column is what that costs.

Batching decode steps is worth 3.5x the tokens per second at 128 cached tokens and 1.7x at 960, because only the cache-independent part of a step amortises across a batch while the rest is per sequence. Batching and threading compound: a batch of one is a skinny GEMV with little for a thread pool to divide, so the same points read 3.1x and 1.4x with the decoder session pinned to one thread.

Under load the picture is different, and better. Against an open-loop Poisson arrival stream, batching holds time to first token near its unloaded value while one-at-a-time decoding collapses — at 80% of measured capacity, p95 TTFT is 0.53 s batched against 20.2 s serial. Past saturation the policy that also limits concurrency wins again by as much: 71% of requests met both service targets at 1.3x capacity, against 13% for the same batch width over an arena large enough never to evict.

Queueing bound over ad-hoc heuristics. A closed-form sojourn-time bound gives a decision rule you can defend and calibrate. Trade-off: it assumes Poisson arrivals, and bursty production traffic violates that. Pessimistic service-rate estimates are the lever that compensates.

Variant per request, not per window. Some adaptive systems batch into windows and switch globally. Per-request lets one request drop to INT8 while the next stays accurate if the queue drains in between.

In-process C++ engine, not a subprocess. Warm sessions behind a pybind11 boundary remove an IPC hop per request, which matters when the deadline budget is 39 ms. Cost: an ONNX Runtime crash now takes the control plane with it, so the Python fallback backend exists partly as an escape hatch.

Measured through the serving path. Every number above comes from the server, not from a microbenchmark of the model in isolation, because the gap between those two is exactly where serving claims usually go wrong.

• Wire the decoder lane into the server. The continuous batching scheduler runs many generations over one arena, but it is not yet behind the serving endpoint. That is the next piece of real work.
• Online learning of service rates. Rates are calibrated offline today. In practice they drift with warm caches, memory pressure, and neighbour workloads. An exponentially-weighted online estimator would keep the queueing model honest.
• Multi-tenant fair queueing. The queue is FIFO. Under multi-tenant load, weighted fair queueing stops one tenant's burst from starving another's deadlines.
• Fold profiling into CI. The pipeline producing service-rate and accuracy profiles is manual. Running it on every model change and diffing the profiles would catch quantisation regressions before they reach serving.