How to Run an Orchestra of AI Agents in a Regular Chat

When you move from a single AI assistant in your code editor to multiple autonomous agents, you quickly hit a control problem. Agents start spawning background processes, consuming tokens, and when something goes wrong, they quietly break the environment somewhere in the depths of Docker. It's even worse when you need to give them access to API keys or GitHub: handing over a personal token to a script that generates and executes code on its own is, to put it mildly, stressful.

Recently I came across the AgentTeams project from the AgentScope team. It's a runtime platform that packages neural network agents into a unified team environment based on the Matrix protocol. Instead of abstract background daemons, you get a regular messenger where you, a manager agent, and specialized worker agents all sit in the same chat.

What's the Architecture Idea

AgentTeams doesn't try to invent its own engine for running LLMs or write another wrapper over prompts. The project works as a container orchestrator. The entire system is built around the Manager-Workers concept:

1. The manager receives your high-level tasks, breaks them down into subtasks, and assigns them to specific workers.
2. Workers execute tasks in isolated environments.
3. The Matrix messenger serves as the unified bus for all messages.

As a result, you see absolutely all the correspondence between agents. If a worker goes off track, you just write to them in the chat @alice измени логику валидации and adjust the course on the fly, without restarting the entire chain.

Without Access to Real Keys

The main problem with most autonomous agents is security. For an agent to make a Pull Request on GitHub or call a third-party API, you usually pass the corresponding secrets directly into environment variables.

The AgentTeams developers solved this through a built-in Higress AI Gateway. The interaction scheme looks like this:

• The worker receives only a temporary consumer token.
• All external requests to LLM APIs, GitHub, or MCP servers go through the gateway.
• Real API keys and personal tokens are stored inside the gateway and never reach worker containers.

Even if the code inside a worker tries to read environment variables and send them out, it will only find the local gateway access token.

Different Runtimes in One Room

An interesting feature of the platform is the ability to combine different types of agents in one room. Currently, three runtimes are supported under the hood:

• OpenClaw on Node.js for tool calling and task coordination.
• QwenPaw on Python, focused on lightweight tasks and browser automation.
• Hermes for autonomous code writing with an isolated terminal sandbox environment.

This provides decent flexibility. You can assign a deterministic OpenClaw agent as the group leader, and connect Hermes for writing and debugging code. Agents communicate with each other through built-in mentions and a shared MinIO file store. The file system significantly saves on context window: agents don't pass large chunks of code or logs directly in message texts, but drop links to files in the storage.

How It's Structured Inside

Architecturally, AgentTeams is delivered as a set of interconnected services:

• Controller is responsible for resource lifecycle and declarative configuration.
• Higress AI Gateway manages traffic and stores authorization data.
• Tuwunel and Element Web provide Matrix server and web client functionality.
• MinIO stores artifacts generated by agents.

Docker is sufficient for running the platform in local mode. Minimum system requirements are 2 CPU cores and 4 GB of RAM. If you plan to spin up 3-4 workers at once, it's better to budget for 8 GB of RAM or more.

Installation is launched with a single command:

bash <(curl -sSL https://raw.githubusercontent.com/agentscope-ai/AgentTeams/main/install/agentteams-install.sh)

The script will ask for a key to the chosen LLM provider (any OpenAI-compatible endpoints work), configure local services, and provide a link to Element Web at http://127.0.0.1:18088.

For more serious installations, the project has an officially supported Helm chart for Kubernetes. When deploying to K8s, you can define workers and teams directly through Custom Resource Definitions (CRDs) in the style of familiar manifests.

helm install agentteams higress.io/agentteams \
  -n agentteams-system --create-namespace \
  --set credentials.llmApiKey=<your-api-key> \
  --set credentials.adminPassword=<your-admin-password> \
  --set gateway.publicURL=http://localhost:18080

Before launching the controller, Helm automatically runs a preflight check: it sends a short test request to the specified model to fail the installation immediately if the API key is invalid or limits are exhausted.

Impressions and Verdict

The project looks fresh against the backdrop of endless console scripts. The concept of using a classic messenger instead of yet another web dashboard turned out to be unexpectedly convenient, especially given that you can connect to Matrix from any mobile client like FluffyChat or Element Mobile.

The downsides include hardware requirements: running a bunch of containers along with MinIO and the gateway quickly eats up memory on a local machine. If you need a simple assistant for hints in your IDE, AgentTeams will clearly be an overkill solution.

But for teams trying to build a full pipeline of multiple AI developers while not wanting to leak API keys or lose control over what's happening, the tool is definitely worth checking out.