Open Source Experiential Gateway Turns AI Usage Into Smarter Routing

An Open Source Answer to the AI Gateway Problem

The AI infrastructure landscape is consolidating rapidly. Stripe's recent acquisition of OpenRouter for over $7 billion underscores the strategic value of the layer that connects applications to multiple AI models. But this consolidation also raises questions about neutrality and control for developers who adopted such gateways to avoid vendor lock-in.

Enter Experiential, an open source project that offers a compelling alternative. Billed as an "open OpenRouter," it provides a unified control plane for closed, open-source, local, and custom models. The project, which recently hit version 0.6.0, has garnered significant attention with over 380 stars on GitHub and active development.

What Is Experiential?

At its core, Experiential is a gateway and router designed for agent workflows. It allows developers and enterprises to use hosted, bring-your-own-key (BYOK), and local models through a single OpenAI-compatible API. This simplifies integration and avoids the need to rewrite systems when switching or mixing model providers.

The platform offers granular control over which users and agents can access which models, for what use cases, and with what spending limits. This is a critical feature for organizations managing multiple teams, projects, or external agents that consume AI resources.

Perhaps the most distinctive feature is its ability to turn production traffic into a better model. By collecting OpenTelemetry traces from agent usage, Experiential can build a custom router or fine-tune an open source model optimized for quality, speed, and cost. This moves beyond simple model selection into continuous optimization based on real-world data.

Technical Deep Dive: The Rust Native Data Plane

A significant recent development is the migration to a Rust-native data plane. The project has deleted its Python gateway data plane entirely, replacing it with a compiled Rust engine that serves every route. This transition delivers substantial performance and reliability gains.

The commit history reveals a meticulous engineering effort. The team froze encoder parity contracts as committed golden bytes, ensuring that the Rust implementation produces byte-identical output to the previous Python version. They also moved readiness accounting to the native components and introduced a fixed worker-thread pool for ledger connections.

This last optimization is particularly noteworthy. In a 608,000-request soak test, the previous Python-based bridge saw file descriptors grow to around 350 and RSS memory to roughly 300 MB. The new Rust implementation with bounded worker threads kept gateway.db file descriptors flat at 14 and RSS at 118 MB, while serving 62% more requests in the same wall time.

The migration also eliminated the FastAPI and uvicorn dependencies, reducing the attack surface and operational complexity. The native engine now handles everything from request admission to settlement, with exactly-once semantics and robust failure handling.

How It Works: From Setup to Optimization

Getting started is straightforward. A simple pip install experiential followed by the exp command launches a local gateway. The first-run wizard guides users through provider selection, model choices, and budget configuration before issuing a one-time API key.

For those who prefer a managed service, Experiential offers a hosted platform at platform.experientiallabs.ai, which serves the same API at api.experientiallabs.ai. The hosted platform also supports a four-step onboarding that includes uploading LLM traces as telemetry and connecting BYOK inference provider keys.

The optimization workflow is where Experiential differentiates itself. Users can collect OpenTelemetry traces from their existing coding agents or download a public dataset like the terminal-tasks OTLP traces. The exp build command then simulates routing decisions against these traces, and exp optimize can fine-tune an open source model using the collected data.

Positioning in a Crowded Market

The model gateway and router space is becoming increasingly competitive. Workweave's Router, for example, focuses on per-action routing with an in-process ONNX embedder and HMM policy sidecar, promising cost reductions of 40-70%. OpenRouter, now part of Stripe, offers a commercial solution with a vast model catalog.

Experiential's open source approach and focus on turning usage data into optimization signals set it apart. It addresses a key concern raised in the Hacker News discussion: the cost of switching models due to lost cache benefits. By allowing users to build custom routers from their own traffic, Experiential can potentially mitigate this issue, though the caching question remains a valid consideration for heavy production workloads.

Why It Matters

The consolidation of AI infrastructure into a few large players is a defining trend. Hugging Face, another access layer, is reportedly exploring a sale at a $13 billion valuation. This pattern suggests that owning the plumbing between AI models and users is becoming as valuable as building the models themselves.

Experiential offers an alternative path. By providing an open, self-hostable gateway with advanced optimization capabilities, it empowers organizations to maintain control over their AI infrastructure. The project's focus on using production traffic to improve routing and models could lead to more efficient and cost-effective AI deployments.

As the AI landscape evolves, tools that provide transparency, control, and continuous improvement will likely gain traction. Experiential is positioned to be a significant player in this space, offering a sophisticated, open alternative to commercial gateways.