Harbor + Docker Sandboxes: Run Safe, Reproducible Agent Evals

Key takeaways

Harbor can now run every eval task in Docker Sandboxes

Docker Sandboxes are MicroVM environments to run agents like Claude Code, Codex, Cursor, and more safely

With Sandboxes, define custom network policies for fair, reproducible evals

Run your evals locally or in the cloud: starting for free on Docker Sandboxes on local surfaces and scaling to Cloud Sandboxes

To evaluate an agent, you give it a real task, let it work, and check the result. Along the way, the agent installs packages, edits files, and runs commands. That’s untrusted, AI-written code, and a benchmark runs it thousands of times.

And agents don’t always play fair. An agent can sometimes earn a passing score on an eval by gaming the check instead of doing the work, for example by tampering with tests or overwriting the file that records its score. This is called reward hacking, and it’s a growing concern across the eval community.

So running agent evals needs two things:

An eval framework that runs each task, lets the agent work, checks the result, and collects the scores and logs.

An environment for every task that’s clean and isolated, so evaluating agents is safe and consistent.

Harbor gives you the first. It’s an open-source eval framework. Give it a benchmark like Terminal-Bench, and it simplifies the process of running evals.

Now, Docker Sandboxes gives you the second: a safe way to run those eval tasks, on your laptop or in the cloud.

Here’s what this means for you

Strong isolation by default. Docker Sandboxes were designed from the ground up for deeper isolation. This means you can run every task in a microVM, not a shared container. A microVM has its own kernel, so whatever the agent does stays inside its sandbox. It can’t break out to reach your machine, other tasks, or anything else outside its environment.

Fair, reproducible evals. Configurable network policies let, for example, a task download what it needs during setup, then cut the agent off from the internet while it works. This way, every agent can be tested under the same conditions.

Parallel runs, no machines to manage. With Docker Cloud Sandboxes, teams can run many tasks at once without needing to manually provision anything. Run dozens of concurrent workers simultaneously.*

Develop locally, scale in the cloud. The same tasks run locally or in Docker Cloud Sandboxes. Build on your laptop, then run your tasks in the background, and at scale, without changing anything.

Try it

Install Harbor with Docker Sandboxes.

uv tool install "harbor[sbx]"

Activate Cloud access for your Docker account and create a personal access token. Select Cloud Sandboxes under access permissions. Copy and save your token. It’s shown only once.

Set your Docker ID (your Docker login username) and PAT (personal access token), then run a task in Cloud Sandboxes.

export SBX_DOCKER_USERNAME="your-docker-id"

export SBX_DOCKER_PAT="your-personal-access-token"

harbor run -t hello-world/hello-world -a oracle -e sbx

To run locally, install the Docker Sandboxes CLI, run sbx login, and add --ek target=local to the command above.

Today’s integration supports single-container tasks that run on CPUs. GPU and multi-container support are coming soon.

To learn more, see the Docker Sandboxes docs and the Harbor docs.

*We’ve tested Harbor with 64 concurrent workers on Docker Cloud Sandboxes.