How to Train LLM-Based Agents with RL on Your Own GPUs

Training language models with Reinforcement Learning has long been associated with massive clusters and closed laboratories. While SFT (Supervised Fine-Tuning) has become routine for ML engineers, reliably running PPO or GRPO for multi-step agents has always been quite a challenge. You either had to write your own wrapper code around vLLM and DeepSpeed, or put up with the limitations of ready-made frameworks.

Researchers from Berkeley Sky Computing Lab decided to simplify this process and released the SkyRL project as open source. This is a full-featured framework that covers the entire RL fine-tuning cycle: from trajectory generation in isolation to updating model weights during runtime.

What the framework consists of

The developers didn't build a monolithic system. Instead, they split the project into several independent layers that can be easily combined for your specific tasks.

The central layer skyrl handles training orchestration on your hardware. It combines a training engine and distributed inference. The main feature of this layer is support for the open Tinker API standard. You can write a training script once and then run it on a local server with NVIDIA or AMD GPUs as well as in the cloud.

Layer skyrl-gym provides an interface for creating training environments. Unlike the classic Gymnasium for robotics, here environments are adapted for text and code interactions. Out of the box, you get support for SQL databases, Python interpreters, search engines, and Linux terminal.

Layer skyrl-agent focuses on long-horizon tasks. It helps train agents that perform dozens of sequential steps, call external utilities, and adjust their actions based on environment responses.

Results in practice

A quality framework is defined not by the number of abstractions but by real metrics. The project team demonstrated the library's capabilities on the Text-to-SQL task.

The authors took a standard 7-billion parameter model and performed RL training through the SkyRL-SQL pipeline. The dataset contained only 653 training examples. The model didn't just memorize correct SQL queries—it learned to execute them in a real DBMS, receive execution errors, and fix syntax on the next step. As a result, this small model outperformed commercial GPT-4o and o4-mini services in generation accuracy on specialized benchmarks.

Another interesting example is integration with the Harbor environment for training command-line agents. The model learns to work in a real terminal: edit files, run tests, find bugs in repositories, and deploy environments without human involvement.

Technical implementation details

In traditional RL training of models, inference and training often block each other. The model generates responses, the process stops, gradients are computed, weights are updated, and only then does a new iteration begin. SkyRL implements a fully asynchronous mode with so-called in-flight weight updates.

While the actor generates new tokens and interacts with the environment, the training process simultaneously updates model parameters. New weights are transmitted to the inference service on the fly, which dramatically reduces downtime for expensive accelerators.

The library doesn't try to reinvent the wheel from scratch. Under the hood, it uses proven solutions from the open-source community: ideas from veRL, OpenRLHF, and Search-R1.

Where the project has already found application

Despite its young age, the framework has already become the foundation for several third-party research projects.

A team from Stanford used SkyRL to create Biomni-R0. This is an agent capable of solving complex biomedical tasks and building logical chains at an expert level. The CodeScout project applied the framework for precise bug localization in code within the SWE-Bench benchmark. Projects like OpenThoughts-Agent and Endless Terminals are also being developed on top of the library, where RL is used for procedural task generation in Linux terminals without manual data labeling.

Where to start experiments

You'll need a Linux server with multiple GPUs and PyTorch installed. The installation process is standard for Python projects:

git clone https://github.com/NovaSky-AI/SkyRL.git
cd SkyRL
pip install -e .

Then you can run one of the ready-made training recipes. For example, to launch the simplest RL loop on math or SQL tasks, just call the corresponding script from the examples folder:

python -m skyrl.train --config-name=math_ppo

If you want to integrate your own environment, simply inherit from the Gymnasium API class and define the step() and reset() methods. The model will receive observations as text or system responses and return generated text as the action.

Is it worth adopting for your projects

SkyRL will be useful for R&D teams and engineers who have outgrown standard SFT and want to squeeze maximum performance from medium-sized models. Teaching an LLM to use tools in a specific domain area through RL is a viable way to beat general-purpose proprietary models in accuracy and query cost.

The main caveat: the project is in active development. The repository has around 400 open issues, and the architecture of individual modules continues to evolve. If you need a ready-made turnkey solution with a "just works" button, you might want to wait a bit. But if you're willing to dig into the code and are looking for a flexible foundation for your own RL agents, SkyRL will save months of development.