Serving a 975B-parameter model across eleven Intel AI PCs

Open-weight models keep getting bigger. Inkling by Thinking Machines has almost 1T parameters, and models of that size are normally trained and served on data-center hardware.

Pooling the memory and compute of several machines makes it possible in principle, but getting a model this large to run efficiently across a fleet of ordinary PCs is the hard part. Intel wanted to know whether its new Core Ultra machines were capable of doing it.

We ran Inkling on eleven Intel Core Ultra X7 PCs, each with an Arc B390 iGPU and 64GB of RAM (704GB in total), connected over Gigabit Ethernet.

Inkling is a sparse Mixture-of-Experts model with 41B active parameters per token across 66 decoder layers. Cascadia split those layers into six-layer shards, one per machine: Machine 0 holds the embeddings and the first six layers, Machines 1–9 each hold six sparse layers, and Machine 10 runs the last six and picks the next token.

A custom MoE engine for Core Ultra

On each machine, Cascadia splits the work between the CPU and the iGPU. The CPU handles attention, routing tokens to experts, and request tracking; the iGPU runs the expert calculations and compressed projections using OpenVINO’s fused MoE operations, while Cascadia keeps control of routing.

Expert weights are INT4, attention projections and the output head are INT8, and the residual stream stays at FP32.

Each request generated 128 tokens with greedy decoding at Cascadia’s default 1,024-token context. Aggregate throughput climbed with concurrency and peaked at 60.29 tok/s with 88 concurrent streams.

Single-stream speed was modest at roughly 5–10 tok/s, which makes this setup best suited to multi-user serving and background work. The point was never single-user speed: it was proving that an organization’s existing fleet of PCs can serve a model this large at all.

The paper, the engineering write-up and the source code are all linked below.