Making AI Agents Test Hypotheses and Run Benchmarks with OpenResearch

Machine learning and algorithmic experiments typically follow a repetitive cycle. You read a fresh paper on arXiv, come up with an architecture modification, create a new git branch, start training, and wait for the plots. If the idea didn't work, you revert the changes and try again. After a couple of weeks, your repository accumulates a dozen abandoned branches, and your head is full of tangled parameters and metrics.

The alphaXiv team has released OpenResearch (utility orx). The project aims to automate this cycle. The idea is to turn coding agents like Claude Code, Codex, or OpenCode into researchers that independently study literature, modify code in isolated branches, run tests, and log artifacts.

How It Works

OpenResearch follows a local-first approach. The entire SQLite database, run history, logs, and experiment tree live locally on your machine. The source code is written in Rust, so the CLI runs fast and doesn't drag along a heavy runtime.

The developers offer a cloud account only for team collaboration or renting their pre-configured server pool. For personal runs, nothing needs to be sent anywhere.

The tool addresses four main challenges of automated research:

• Parallel branches via git worktree. If you give an agent multiple tasks simultaneously, the project won't turn into a mess of conflicts. Each direction gets an isolated working tree.
• Experiment tree. Each run is tightly linked to a commit, diffs, metrics, and generated files. Results are saved in immutable form.
• Flexible infrastructure selection. The same code snapshot runs on a local CPU, a remote server via SSH, or in Slurm, Ray, Kubernetes, and Modal clusters.
• Integration with scientific sources. The agent can search for papers by keywords and download publications directly via DOI or arXiv identifier.

Launching a Workspace

You can install the utility on macOS or Linux with a terminal script:

curl -LsSf https://openresearch.sh/install.sh | sh
orx up

The command orx up starts a local server and opens the web dashboard at address http://127.0.0.1:4791. If you prefer graphical interfaces without dealing with a browser, the project has a ready-made desktop application.

When you need to run heavy computations on a remote GPU machine, the workspace is launched via SSH:

orx up --remote user@gpu-box

In this mode, the service listens on the local interface on the remote server. You won't need to open ports to the outside. But keep one detail in mind: there's no built-in authentication on the remote instance, so any user with access to that machine will be able to connect to the open port.

Connecting Agents

To give the agent you're using commands for working with the repository, run the skills installation:

orx install-skills

After that, the agent starts understanding the context of research tasks and executes specific commands:

# Поиск релевантных статей
orx discover keyword "mixture of experts routing"

# Загрузка конкретной публикации
orx paper 2401.12345

# Запуск эксперимента и просмотр логов
orx exp run exp_01h8abc
orx logs run_01h8xyz

In autonomous search mode (autoresearch), the agent itself formulates a hypothesis, makes code changes, runs a pass, reads metrics from logs, and decides where to go next.

Privacy and Telemetry

Release builds of the CLI send anonymized usage statistics by default, salted with a random installation ID. This doesn't include file paths, repository names, tokens, or prompt contents.

If you don't need metrics collection, telemetry can be easily disabled with one command:

orx telemetry off

In builds compiled directly from source code, analytics sending is disabled by default.

Limitations and Caveats

The project is fresh, with around 600 stars in the repository. Documentation is concise in some places, so you'll need to dig into the code to understand the purpose of certain parameters.

The second point concerns autonomous cycles. If you leave an agent unattended overnight, it can easily spend your entire API token balance, stuck trying to fix library incompatibilities in tests. It's wiser to first test the setup on short iterations with explicit limits on the number of steps.

Who This Project Is For

OpenResearch is worth trying for ML engineers and researchers who are tired of manually maintaining spreadsheets with run results and switching branches in Git. The tool provides a convenient framework that unites the terminal, remote server execution, and paper-reading skills into a single workflow.

The source code is open under the MIT license, and the repository is available on GitHub: alphaXiv/openresearch-cli.