How to fit a million tokens into a regular GPU using Kimi Linear

Have you ever wondered why working with long texts in LLMs is so expensive? It's all about the KV-cache. When you feed a massive document to a model, its "memory" (the cache) swells to obscene sizes, devouring all your video memory. The folks at MoonshotAI decided it's time to do something about this, and rolled out Kimi Linear. This isn't just another "improved" model—it's an attempt to rethink the attention architecture so we can work with a million-token context without buying a server farm.

What's the problem with standard attention

The standard Full Attention mechanism (the one in classic transformers) is a hungry beast. Its complexity grows quadratically with text length. Want a context twice as big? Get ready to spend four times the resources. Popular solutions like Flash Attention or DeepSeek's MLA (Multi-head Latent Attention) help, but don't solve the problem radically when it comes to truly long sequences.

Kimi Linear takes a different approach. The developers used a hybrid method combining the strengths of transformers and RNN-like structures.

How Kimi Delta Attention works

The core of the project is the Kimi Delta Attention (KDA) mechanism. Without diving into hardcore math, it's an evolution of the Gated DeltaNet concept. The main trick here is smart "forgetting."

In a standard RNN, memory is constrained by a fixed state size. KDA employs a gating mechanism that determines which past information should be retained in this compressed state and which can be discarded. This enables the model to maintain high accuracy even across vast distances, where conventional linear models start to "drift" and lose the thread of the narrative.

What this delivers in practice

The developers introduced an architecture where KDA and MLA (Global Attention) are blended in a 3:1 ratio. This combination achieved several impressive results:

1. Memory savings. KV-cache requirements dropped by 75%. This is critical when deploying the model on your own hardware.
2. Generation speed. On a 1 million token context, token throughput increases up to 6x compared to standard architectures.
3. Honest context. Benchmark tests on RULER show the model genuinely "sees" and uses information throughout all 128k (and up to 1M) tokens, not just pretending to.

The charts below demonstrate how Kimi Linear (blue line) pulls ahead in speed as context length grows:

Trying it out

MoonshotAI didn't hold back and released model weights on Hugging Face. There's a base version and an Instruct variant with 48 billion parameters. Thanks to the Mixture-of-Experts (MoE) architecture, only 3 billion parameters activate during computation, making the model surprisingly lightweight for its class.

To get started, you'll need the latest PyTorch and the fla-core library. The code looks fairly standard for anyone who's worked with transformers:

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "moonshotai/Kimi-Linear-48B-A3B-Instruct"
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto",
    trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)

# Обычный чат-шаблон
messages = [
    {"role": "system", "content": "You are a helpful assistant provided by Moonshot-AI."},
    {"role": "user", "content": "Расскажи, в чем преимущество линейного внимания перед обычным?"}
]

input_ids = tokenizer.apply_chat_template(
    messages, 
    add_generation_prompt=True, 
    return_tensors="pt"
).to(model.device)

generated_ids = model.generate(inputs=input_ids, max_new_tokens=500)
response = tokenizer.batch_decode(generated_ids)[0]
print(response)

If you need to deploy this in production, the model plays nice with vLLM. You can spin up an OpenAI-compatible API with a single terminal command, specifying a massive max-model-len.

Is it worth downloading

The project looks promising for those building RAG systems or analyzing long logs and documents.

Who should definitely take a closer look:

• Those who've hit the memory ceiling of GPUs when working with long contexts.
• Developers who care about response speed (TPOT) in real time.
• Researchers looking for alternatives to standard transformers.

On the downside, the architecture is relatively new, and support in third-party tools (like quantization or specific optimizers) may not arrive immediately. But having ready-made KDA kernels in the FLA library is encouraging.

Kimi Linear is a good example that algorithm optimization can still yield bigger gains than simply stacking more teraflops. If you need to "digest" an entire library or massive codebase in a single pass—this is probably one of the most interesting tools available right now.