Anytime Inference

Static ML serving policies force a bad tradeoff. Always running the large FP32 model is accurate but drops most requests once traffic passes its service limit. Always running the small INT8 model has headroom but loses accuracy you did not need to lose. In practice the right precision depends on the current load and the current request's deadline. That's a control-theory problem, not a "pick a model at deploy time" problem.

Anytime Inference is a serving stack that treats variant selection as a constrained optimisation: maximise expected accuracy subject to a deadline, with the deadline budget enforced by an M/M/1-based admission controller.

Python control plane, C++ ONNX runtime. The control plane monitors CPU load in real time and routes each request to FP32 or INT8. The runtime holds warm ONNX Runtime sessions for both variants and returns results over a base64-framed JSON stream.

• Load monitor. Background thread sampling psutil.cpu_percent with an exponential moving average. Smoothed load is the planner's feedback signal.
• Selector. Picks the highest-accuracy variant whose expected service time, inflated by current CPU pressure, still admits under the M/M/1 sojourn bound.
• Admission controller. Computes E[W_q] + 1/mu from the observed arrival rate and the candidate variant's service rate. Rejects the request when this would exceed the deadline budget. Derived directly from the M/M/1 stationary result.
• Runtime client. Spawns the C++ binary as a subprocess and exchanges base64-framed JSON over stdin/stdout. A pure-Python ONNX Runtime backend is used automatically if the binary is not present.

M/M/1 admission control over ad-hoc heuristics. Closed-form service-time bounds give a defensible decision rule. Trade-off: assumes Poisson arrivals. Bursty production traffic can violate that; the runtime can be configured with pessimistic service-rate estimates to compensate.

Variant is per-request, not per-window. Some adaptive systems batch requests into windows and switch variants globally. Per-request lets a fast request drop to INT8 while a subsequent slow one stays on FP32 if the queue drains.

Two variants (FP32 and INT8), not more. Adding FP16 or mid-tier quantisation would give finer control but complicates the calibration story. Two well-characterised variants is enough to show the mechanism works.

C++ runtime as subprocess, not a shared library. Subprocess isolation means an ONNX Runtime crash does not take down the Python control plane. Cost: an IPC hop per request. Base64-framed JSON is honest overhead but small compared to inference itself.

Deadline is the SLO, accuracy is the objective. Backwards from how a lot of ML serving is framed. When you name the deadline as the contract, the "right" model becomes whichever one meets it with the best accuracy. That's a real user promise, not a benchmark.

Reject over lie. When the admission controller cannot promise the deadline, the request is rejected with a clear response, not silently served slow. In production this feeds a retry-with-backpressure loop on the client, which is easier to reason about than tail-latency alerts on a chatty p99 dashboard.

Reproducibility over marketing numbers. Every result comes from a script that fresh-seeds, sweeps offered load, and reports mean and confidence intervals. The demo runs on a synthetic ONNX model so anyone cloning the repo can verify.

Sweeping offered load against a fixed 45 ms deadline (FP32 12 ms and INT8 5 ms service profiles, 4 workers) compares the adaptive policy with an FP32-only baseline under the same Poisson arrival stream:

Once offered load approaches the FP32 service limit, the baseline sheds the majority of requests. The adaptive policy keeps completion at around 99 percent by routing to INT8 when the queueing math says FP32 will not admit. Mean compute cost per served request falls by roughly 45 to 57 percent under heavy load with no loss in deadline hit rate.

Results are from the serving harness. Numbers vary run to run because routing and admission decisions are driven by live CPU load.

• Online learning of service rates. Currently service rates are calibrated offline and passed to the controller. In practice service rate drifts with warm caches, memory pressure, and neighbour workloads. An exponentially-weighted online estimator would keep the M/M/1 model honest.
• More variants. Adding FP16 (and possibly INT4 for aggressive load-shedding) gives the selector a smoother accuracy-versus-throughput curve. Adds calibration and drift-check burden.
• Multi-tenant fair queueing. The current queue is FIFO. Under multi-tenant load, weighted fair queueing prevents one tenant's burst from starving another's deadlines.
• Fold profiling into CI. The offline pipeline that produces service-rate estimates is manual. Running it on every model change and diffing the resulting profiles would catch quantisation regressions before they hit serving.