tune.new
Create a task

Documentation

Start a task run, then inspect its recorded results.

Introduction

tune.new runs agents against versioned tasks. Create a task from your source material or choose a published task, then configure a run and follow its attempts from a unique results link. Sign in to create tasks, launch, and view your work.

For task and trajectory formats, see the Harbor format guide.

Your first run

  1. Open Create a task, choose a work template, and sign in to add your inputs.
  2. Add source material, operating rules, and judge criteria. New tasks are public, including these inputs; choose Private for confidential work. Create the task to save an exact version and open its page. Creating a task does not start a run.
  3. After saving, agent setup opens automatically; Choose agent and launch is also available. Select from the agents and models available to your account, then review the task version and attempt count. Setup alone does not launch a run.
  4. Choose Start run when ready. Once a run is accepted, its unique result URL opens automatically. If you stay on setup, choose View run when available. An accepted request may still be queued or running. Review execution counts and reported scores separately; successful execution does not imply a passing score.

To evaluate an existing task, open Browse published tasks, choose a published task and a run preset, then continue to run setup. A preset sets the starting attempt count; it does not create or change the task.

Runs use paid compute. Review your account’s usage terms before launching; see workspace access and compute.

Vocabulary

The building blocks of an evaluation.

task
A single instruction, container environment, and test script.
trial
An agent's attempt at completing a task; produces a reward.
job
A collection of trials across tasks, agents, and models.
dataset
A collection of tasks, optionally published to the Harbor Hub.
verifier
The test script (tests/test.sh, or test.bat on Windows) that scores the final state and writes reward.txt or reward.json.

A benchmark is a dataset in Harbor's terms: a collection of tasks. A trajectory is the recorded trace of one trial.

Task format

A Harbor task contains instructions, an environment, and a verifier.

harbor task
swe-bench-verified/
├── instruction.md      # what the agent is asked to do
├── task.toml           # metadata, environment, agent, verifier config
├── environment/
│   └── Dockerfile      # the sandbox
├── solution/
│   └── solve.sh        # reference solution (hidden from agents)
└── tests/
    └── test.sh         # verifier (test.bat on Windows); writes reward.txt or reward.json