Inside Our Distributed LLM Inference Research for Intel PCs

Many organizations have hundreds or thousands of Intel-powered PCs with integrated GPUs, NPUs, and unified memory. Yet, most of that computing power sits idle when the machines aren’t in active use.

Last week, we announced Cascadia: a distributed AI inference stack to leverage that compute. This was built on novel research tackling the unique problem space of performing distributed inference on heterogeneous hardware.

Our first research paper, titled “Pre-Compiled Pipeline Shards for Distributed LLM Inference on Intel AI PC Fleets”, covers the explorations and optimizations which led to Cascadia. In practice, running large models across machines is a difficult task; however, many of the challenges we faced were on the software side, not the hardware side.

This blog post gives a high-level overview of the research paper and the optimizations we discovered.

Many useful local models can fit on a single machine, but more powerful models can grow in size rapidly. Llama 3 70B, for example, is ~36GB in size (INT4 compressed). If a machine has enough RAM/VRAM to load it into memory, it can perform inference as expected.

If a model is too large to fit in a single machine, however, it can also be split into parts and spread across multiple machines. At a high level, most LLMs are constructed of layers. When an LLM ingests some input, it passes through the model’s layers and generates something called intermediate activations. These are used to eventually construct and provide a response.

These layers can be split into groups, called shards, and each shard can be evaluated on a different machine.

This effectively makes it possible for a group of computers to pool their resources and run a model larger than any single machine could. These devices need to communicate with each other over a network to coordinate the inference. The entire process is called pipeline-parallel sharded inference.

Most models with open weights can be manually split into shards and formatted for the inference runtime being targeted, such as llama.cpp. For our implementation, we exported the shards for OpenVINO, an open source toolkit and runtime which optimizes inference for Intel silicon.

In our research, we took a Llama 3.1 8B model, split its layers into shards, and pre-compiled and exported these shards to OpenVINO IR (the format for running models on OpenVINO). We also used INT4 quantization with each shard to make them more efficient for running on commodity Intel hardware.

This effectively solves the problem of splitting larger models and running them across different Intel-based devices. However, making it possible doesn’t immediately make it a performant or realistic solution.

Upon initial testing, exporting Llama 3 8B as a 1-stage shard (all 32 layers in a single shard) performed markedly worse (13-23% slower) than simply running the model without sharding it. Both tests took place on a single machine.

This was an interesting piece of data as the model and its weights are effectively the same in both scenarios. We realized that our exported shards were missing a specific pattern that OpenVINO expected (called beam_idx), and instead of throwing an error, OpenVINO was falling back to a slower, default pattern.

We had to perform post-export graph surgery to include this operation for each shard, and this brought the performance back up to a similar level as a single machine running a single model.

This test only covers use of a single shard on a single machine though, and upon splitting a model into more shards, the inference performance decreases in proportion to the number of shards you’re deploying. This is due to the latency of needing each machine to communicate to generate each token.

In order to alleviate the latency induced by adding shards and machines, we worked on two techniques in particular: mask-based speculative decoding, which reduces the amount of computation (and network requests) required, and pipeline micro-batching, which drastically improves throughput in setups with more than one user or agent.

Speculative decoding is a technique by which a small model predicts the next few tokens, and a target model decides whether to accept or reject them. This has the benefit of sending multiple (speculative) tokens at once, instead of the standard output of a single token at a time.

If the speculative tokens are accepted, the system doesn’t have to perform as much computation and, in the case of a sharded and/or distributed setup, the amount of network requests required to produce an output.

Each time a token is generated by a model, it is appended to all of the previous tokens generated. A representation of these tokens is cached in each layer of the model, to prevent it from having to re-compute the current state of a response from scratch each time. The cache each layer has is called the KV cache (Key-Value cache).

Typically in the case of speculative decoding, the system will discard the generated tokens and prevent them from being unnecessarily cached. In the case of OpenVINO, the rejected tokens enter the KV cache and require an additional operation to remove them which adds overhead.

In our research, we decided not to remove the tokens from the KV cache; but simply mask them instead so they are ignored on each pass. When implemented, the inference speed rose from an average of 22.66 tok/s to ~29.98 tok/s when running on a single machine.

For a model split into three shards without mask-based speculative decoding, the average performance was 21.56 tok/s. When adding the spec optimization to the three shard setup, it rose to 28.38 tok/s.

All of the benchmarks discussed so far have been for tests on a single machine. However, the benefits compound when put into practice with a fleet of machines with multiple users or agents.

In a standard single-user setup, a prompt passes through the first shard, and the next, until it produces a final token at the end. A single request can’t, however, keep every shard busy at once. If the first shard is processing some input, the second one is waiting for it. Likewise, once the first shard is finished generating activations, it is waiting for input again.

In a situation like this, a lot of machines can sit idle. These periods are sometimes referred to as pipeline bubbles.

In a multi-user (or multi-agent) system, a coordinator can be used to interleave requests to idle shards via micro-batching. For example, if Shard 1 processes part of a request from Alice and passes it to Shard 2, it can now start processing a request from Bob while waiting for the next token for Alice’s request.

It’s important to note that micro-batching doesn’t aim to find optimizations for single-user inference speed, but instead makes a fleet of computers much more efficient. It also means that the more machines and input streams in the system, the more useful the setup is.

Given that the basis of the research is focused on inference across a distributed, networked set of Intel-based devices, we tested our explorations on a set of three Intel AI PCs across the same Wi-Fi network.

In our benchmarks, a single user using a model split into two shards, across two machines, saw an average of 16.33 tok/s. As soon as a concurrent user was added to the pipeline, the aggregate throughput increased to 29.34 tok/s. With three stages and concurrent users, it reached 64.67 tok/s.

Keep in mind that this is aggregate throughput, not for a single stream setup. The benefits compound much more quickly when inference is being requested from multiple users/agents. When adding the third Intel AI PC and third stream, the aggregate tok/s reached 64.67.

Network latency and distribution generally added ~20ms of per-token overhead, which is somewhat expected. We managed to reduce this in some places, for example when sending logits.

Logits contain the final scores for what the next token should be, and are sent back to the coordinator. They are 501 KB in size, but we took a “greedy” approach and selected only the top result (8 bytes in size). This is a much smaller payload to be sent over Wi-Fi or WAN connections.

We also tested the approach at a larger scale: using Llama 3.1 70B (~36GB, INT4 quantized) and split it across four Intel AI PCs, which ran on our setup successfully.

Our results show that local inference is perfectly capable of extending past a single machine, and that larger, more powerful models can be deployed on a network of machines. For multi-user or multi-agent setups, pipeline-parallel sharded inference can lead to a much higher aggregate throughput.

That doesn’t mean every single AI workload needs to be distributed across all employee PCs. The general rule-of-thumb we advise is to shard a model across the minimum number of machines required, and in the case of multi-tenant usage, have multiple clusters of machines. This approach is also considerably cheaper than paying for inference from a cost standpoint.

This paper forms part of the foundation for Cascadia: an open-source runtime built for the express purpose of running distributed inference on Intel machines, based on this research. We’ve already started onboarding design partners, and we are making considerable progress on performant distributed inference.

We’re also working on a lot of adjacent research related to memory usage and quantization, and we’ll be releasing a second research paper on Cascadia’s unique architecture over the coming weeks.

• Read the full research paper on arXiv
• Book a demo call with us
• Check out the GitHub repository
• Follow us on X