Mesh LLM: Distributed AI Computing on Iroh

When people picture running a large language model, they picture a data center. Racks of GPUs that belong to someone else, a metered API, and a bill that grows every month you succeed. You send your prompts off to a black box and hope the price, the model, and the privacy policy all stay the way they were when you signed up.

For a lot of teams that is a bad trade. You give up control over when models change, where your data goes, and what hardware runs your workloads. And as usage grows, so does the bill, with no lever to pull except "pay more."

Mesh LLM is a different shape. It pools the GPUs and memory you already have, across as many machines as you want to add, and exposes the whole thing as one OpenAI-compatible API. Start one node. Add more later. Let the mesh decide whether a model runs on the box in front of you, routes to a peer, or splits across several machines.

The Problem: AI is Expensive, and It's Somebody Else's

The popular models are monoliths. Most people reach them through a UI or an API key and pay a large provider to run everything. That is convenient, and it is also a surrender. You do not control when the model gets updated, what memory it runs in, or what hardware sits underneath.

Plenty of businesses and services that depend on these models want the opposite: more control, more pluggability, lower cost. They have GPUs sitting in offices, in closets, under desks. What they are missing is a way to make those machines act like one.

Mesh LLM: Run the Models Yourself

The pitch is simple. Run bigger models without buying bigger GPUs. Share compute privately with your team, or publicly with the world, to power agents and chat. Point any OpenAI client at http://localhost:9337/v1 and stop caring where the work actually happens.

Under the hood, Mesh LLM distributes model compute across a mesh of iroh endpoints. A request can be served three ways:

Run it locally, on this machine's GPU.
Route it to a peer that already has the model loaded.
Split a model too big for any single box across several machines, as a pipeline.

How It Works

The architecture is pluggable. Plugins declare what they provide in a manifest, the runtime starts them, routes calls, and exposes their capabilities over MCP, HTTP, inference, and mesh events. The catalog ships with 40+ models, from half-a-billion-parameter models that fit on a laptop to 235B mixture-of-experts giants.

For the giants, Mesh LLM has a split mode (internally, "Skippy"). A model gets partitioned by layer ranges into stages: layers 0 to 15 on one node, 16 to 31 on the next, and so on down the pipeline. Activations flow from one stage to the next, so several modest machines can run a model none of them could hold alone. The OpenAI client never sees any of this. It still just talks to localhost.

continue reading below...

How It Uses Iroh

Every node, whether it serves models or only sends requests, boots an iroh endpoint. That endpoint is the node's identity, a public key, and its only network surface. There is no central server. iroh handles the hole-punching, NAT traversal, and relay fallback needed to open a direct, authenticated QUIC connection between any two nodes, wherever they sit.

To keep that working across the open internet, Mesh LLM runs two iroh relays in different regions, so nodes that cannot reach each other directly always have a fallback path nearby.

The whole protocol rides on QUIC's ALPN negotiation. There are three:

mesh-llm/1: Main mesh for gossip, routing, HTTP tunnels, and plugin channels.
mesh-llm-control/1: Owner control plane for config sync and ownership attestation.
skippy-stage/2: Latency-sensitive activation transport for split models.

Inside the main mesh-llm/1 connection, everything is a bidirectional QUIC stream tagged with a single leading byte that says what kind of stream it is. One connection carries gossip, inference, route queries, and peer-lifecycle events, all demuxed by that first byte.

The neat part is what this buys you. iroh gives authenticated, NAT-traversing QUIC between any two machines, addressed by public key. So "route to a peer" and "stream activations to the next pipeline stage" become the same primitive as "talk to localhost," just with a different endpoint ID. The networking stops being something you have to think about.

iroh provides the secure transport. Mesh LLM builds its own gossip layer on top, so it controls exactly who gets admitted to the mesh, which versions are compatible, and which peers to trust.

Market Context and Competition

Mesh LLM enters a crowded field. The "inference gold rush" has attracted heavy investment, with companies like Baseten recently valued at $13 billion, and Inferact from the creators of vLLM. French startup ZML, endorsed by Turing Award winner Yann LeCun, recently released LLMD, an inference server that runs models across Nvidia, AMD, Google TPU, Apple Metal, and Intel Arc chips at maximum speed.

Meanwhile, Prime Intellect raised $130 million in a Series A round led by Radical Ventures, with participation from Nvidia Ventures, Intel Capital, and Dell Technologies Capital. Their platform parallelizes training across large GPU clusters using PyTorch's FSDP2, targeting enterprise agent development. The market is clearly moving toward distributed, open-source-friendly infrastructure.

Mesh LLM differentiates by focusing on peer-to-peer networking rather than centralized orchestration. It doesn't require a cloud provider or a dedicated cluster. Any machine with a GPU can join the mesh, and the system handles NAT traversal, authentication, and routing automatically via iroh.

Technical Deep Dive: The Iroh Protocol

Iroh is a dial-any-device networking library that provides authenticated, NAT-traversing QUIC connections between any two machines, addressed by public key. Mesh LLM uses this as its transport layer, building a custom gossip protocol on top for peer discovery and admission control.

The protocol uses three ALPNs for different traffic types: mesh-llm/1 for main mesh operations, mesh-llm-control/1 for owner control plane, and skippy-stage/2 for latency-sensitive activation transport. Inside the main connection, a single leading byte demuxes streams for gossip, inference, route queries, and peer-lifecycle events.

This design means "route to a peer" and "stream activations to the next pipeline stage" become the same primitive as "talk to localhost," just with a different endpoint ID. The networking stops being something you have to think about.

Market Context and Competition

Getting Started

Users can install the lightweight software (about 18 MB) and either join the public mesh or configure private deployments. The system presents itself as localhost:9337/v1 to any standard OpenAI client.

A mobile app is coming, built on iroh's Swift SDK. The plan is to speak ACP, the emerging agent standard, so other clients can join the mesh too. The throughline is the same one that motivated the whole project: more peer to peer, fewer closed servers, and no lock-in.

Iroh is a dial-any-device networking library that just works. Compose from an ecosystem of ready-made protocols to get the features you need, or go fully custom on a clean abstraction over dumb pipes. Iroh is open source, and already running in production on hundreds of thousands of devices.