Why Agents Need Special Inference and How TokenSpeed Speeds Up LLMs

When you run a standard neural network for text or code generation, engines like vLLM usually have more than enough capabilities. But as soon as it comes to autonomous AI agents, the picture changes. Constant tool calls, dialogue branching, repeated context passes, and growing KV-cache quickly bring standard inference to its knees.

In May of this year, the LightSeek team released TokenSpeed on GitHub. The developers set out to create a specialized engine for agent workloads that combines the speed of TensorRT-LLM with a clear and simple Python interface.

What Breaks in Standard Engines When Working with Agents

Agent scenarios differ significantly from chatbot dialogue. A bot receives a request, generates a single response, and releases resources. An agent, on the other hand, operates in cycles:

• Forms thoughts and selects a tool
• Waits for a response from an external API or database
• Analyzes the received result and takes the next step

This causes context to grow continuously, forcing the server to recalculate long token chains or maintain massive memory volumes for the KV-cache. If you run dozens of such agents simultaneously, even powerful accelerators start idling while waiting for data transfer.

TokenSpeed Architecture and the Solution

The TokenSpeed authors didn't create another thin wrapper over PyTorch. They rewrote critical system components to extract maximum performance from the hardware.

First, they separated the control loop from execution. The request scheduler is written in C++, while higher-level execution remains in Python. Each request's state, KV-cache ownership transfer, and timing are tied to a strict finite state machine. The C++ type system checks cache resource reuse safety at compile time, completely eliminating memory leaks.

Second, the engine includes a static compiler for distributed computing. Developers don't need to manually write parallelism logic through torch.distributed. Simply placing annotations at module boundaries is enough—the compiler automatically generates commands for inter-processor communication.

Third, they reworked the low-level kernels. The authors implemented their own version of the Multi-head Latent Attention (MLA) algorithm, optimized for NVIDIA Hopper and Blackwell architectures. It minimizes latency during active work with long contexts.

Numbers and Real-World Tests

The developers provide concrete benchmarks on current models. In May, the project demonstrated a speed of 580 tokens per second on the Qwen3.5-397B-A17B model in agent tasks.

Looking at comparison charts with TensorRT-LLM on NVIDIA B200 chips when running the Kimi K2.5 model, TokenSpeed wins on throughput at the same latency level.

It's interesting to see how quickly the project adapts to new releases. For example, support for Kimi K3 models and FP4 inference for NVIDIA and AMD GPUs was added literally on the day of their official release.

Integration into Existing Code

From an entry point perspective, TokenSpeed uses AsyncLLM. The architecture minimizes CPU overhead for processing incoming HTTP requests, so the server doesn't get overwhelmed at high RPS.

The server startup example looks familiar to anyone who has worked with vLLM:

python3 -m tokenspeed.entrypoints.openai.api_server \
    --model Qwen/Qwen2.5-7B-Instruct \
    --port 8000

After that, you can connect to the server using the standard OpenAI client, which simplifies integration into existing agent frameworks like AutoGen, CrewAI, or LangChain.

Is It Worth Deploying to Production

Currently, the repository has about 17,000 stars and nearly 50 open issues. This is a young, dynamic project that it's too early to call a conservative industry standard.

It's definitely worth trying TokenSpeed if you already have AI agent infrastructure deployed and you've hit the performance ceiling of vLLM on long contexts. If you need a simple server for serving a basic chatbot, standard tools will suffice for now.