Research with Intel. The methods behind the numbers.

How we served Inkling, a 975B-parameter mixture-of-experts model, across eleven Intel Core Ultra PCs with a resident MoE engine built on OpenVINO’s iGPU operations. Covers throughput under load, first-token latency, and prompts from 1k to 64k tokens.

Aggregate decode at 88 concurrent streams

60.29 tok/s

Median time to first token at 15 streams

6.05 s

Intel AI PCs, 64GB each

11

The system design behind Cascadia. Every PC schedules and runs its own requests, so there is no central control plane in the path. Nodes join a certificate-based peer-to-peer network and serve a model whole, as load-balanced replicas, or split across machines as a pipeline.

Throughput on a three-node NPU testbed, against one node

3.1×

Throughput on a four-node deployment, against one node

4.1×

Serving modes: whole model, replicas, pipeline

3

Splitting a model by layer into precompiled OpenVINO stages, one per PC, so a fleet can serve models no single machine can hold. Adds speculative decoding for stateful models and interleaves users across stages.

Model served at interactive speed on four Intel AI PCs

70B

Throughput for two users on a two-node pipeline

1.79×

Speculative output matches the plain pipeline, token for token

Exact