8. A Baseline for Judging the Agent's Setup

Keeping the tasks you judge the agent by

Your shared-list app has picked up a lot of setup by now. There is a CLAUDE.md that tells every session how to install, run and test the project. A hook runs the type check after every edit. The reproduce-bug skill turns a vague report into a clean reproduction note. The GitHub connection lets the agent read issue twelve for itself. Each of these went in because a task needed it, and each one made that task go better.

Now a different kind of question comes up. Say you wonder whether a cheaper model would do just as well. Or you want to know if raising the reasoning effort is worth what it costs. Or you rewrite a line in CLAUDE.md and want to know if things got better or worse. This chapter answers how you can tell.

The honest answer is that you can't tell by feel. One good session after a change proves very little. The agent's work varies from run to run, and the tasks you happen to give it this week are not the tasks you gave it last week. So a change can look like a win because the task was easier. To judge a change to your setup, you need two things you keep on purpose. You need a fixed set of tasks, and you need the results you got on them before. Then a change is something you measure against that record instead of something you guess about.

There is a second idea that matters just as much. When you compare setups, cost means the cost of getting a change finished. That includes the token bill, but it also includes the minutes you spend reading the diff, and the minutes you spend fixing what the agent got wrong. A run that costs a few cents less but leaves you twenty minutes of repair is not cheaper. Keep that in mind, because the whole chapter leans on it.

You will build this record first, then use it to pick a model and an effort level for each kind of work in the app.

Choosing the tasks

The tasks in your baseline should be real. Anthropic's own guidance on testing agents says to draw tasks from work you have actually done, such as old pull requests, issues and bugs, rather than inventing a big set of made-up exercises. Their full advice is to collect somewhere between twenty and fifty tasks over time. You don't need that many to start. Three is enough to catch real differences in a small app, and you can add more as the backlog grows.

Your app's history already hands you three good ones, and they cover different kinds of work.

The first is a small validation rule. Back at the start, the app let you add an item with an empty name, and the agent added a rule that rejects it. That task is small, touches one place in the code, and has a clear right answer.

The second is a bug fixed from a written reproduction note. The fast-add bug, where a second item added quickly vanished from the screen, came with a note from the reproduce-bug skill. That note lists the steps, the data, what happened and what should have happened. This task tests whether the agent can work from a careful written report.

The third is a small feature that touches the database layer. The done checkbox needed a migration, which is a script that changes the database's structure, plus a function in the database client and a checkbox in the interface. This task is the hardest of the three, because a mistake can hide in any of those layers.

Notice what these three have in common. Each one was already finished and checked, so you know what a correct result looks like. That is the reason to use your own history. You aren't guessing what success means; you already saw it.

Writing each task down

Make a folder in the repository called evals, and inside it, one small folder per task. Each task folder holds four things.

The first is the starting commit. This is the commit just before the real fix landed. For the empty-name task, it's the commit before the rejection rule existed. You find it in git log by looking for the fix and taking the commit before it. You write down its full commit hash, the long string of letters and numbers Git uses to name a commit, so the task always starts from exactly the same code.

The second is the exact request text. Write the words you will give the agent, word for word, in a plain text file. Don't paraphrase it differently each time. If the wording changes between runs, you are no longer testing the setup; you are testing your phrasing. For the fast-add task, the request can point the agent at the reproduction note, which you copy into the task folder too, so the agent gets the same input every time.

The third is the success criteria, and these have to be testable. "The bug is fixed" is not a criterion. For the fast-add task, the criteria come straight from issue twelve. No item vanishes when two are added quickly. A reload shows the same list the screen showed. For the done-flag task, say they are these. The new column exists after the migration runs. Ticking an item and reloading keeps it ticked. Unticking and reloading keeps it unticked. Existing items still load.

The fourth is the check that decides the task. Write down which check settles it, and in what order. For the empty-name task, it is the full test suite plus the type check. For the done-flag task, it is the suite, the type check and the clicked journey: add, tick, reload, untick, reload. You learned in an earlier chapter that a passing suite does not prove the journey works. So when the journey is part of the deciding check, write that down, and don't skip it.

There is a useful trick here. Tests you wrote when you first fixed each bug can live in the task folder as hidden checks. The regression test for two back-to-back submits, for instance, can be copied into the fast-add task folder. After the agent finishes, you drop that test into the code and run it. The agent never saw it, so it can't write code that only fits that one test. Anthropic's guidance makes the same point: keep the grading material where the agent cannot read or change it.

The results record

Beside each task's description, keep a results file. Each time you run the task, you add one entry. Each entry holds the same few things.

It records which configuration ran, meaning the model and the effort level. It records whether the task finished, meaning the deciding check passed. It records the size of the diff, counted as lines added and removed. It records any regressions, which are things that used to work and now don't, such as another test that broke. It records the minutes you spent reviewing and repairing. And it records the tokens used and the cost.

The repair minutes are the field people are tempted to skip, and the one this chapter keeps coming back to. Time it honestly. Start when you open the diff, stop when the change is something you'd commit. If you had to fix a line or ask the agent for a second round, that counts.

Diff size matters for a quieter reason. A run that passes with a forty-line diff and a run that passes with a four-hundred-line diff are not the same result. The big one takes longer to review, and it may have changed things it had no reason to touch.

Replaying a task cleanly

A baseline is only useful if every run starts from the same place. If one run happens in your main checkout with half-finished work lying around, and another starts from a clean tree, the comparison means nothing.

The tool for this is a Git worktree. A worktree is a second working folder attached to the same repository, checked out at whatever commit you choose. Your main folder stays untouched while the agent works in the other one. Anthropic's testing guidance uses this exact approach.

Here is how one replay goes, step by step.

1. Create a worktree at the task's starting commit with git worktree add, pointing it at a scratch folder outside your project and giving it the detach option and the starting commit hash, so it isn't tied to a branch.
2. Copy your current CLAUDE.md and the dot-claude folder, with its settings, hook and skill, from your main checkout into the worktree, remove the evals folder from the worktree, and commit that, so the agent works on old code with today's setup and can't see the grading material.
3. Open a terminal in the worktree, start Claude Code with the model and effort you are testing, give it the exact request text, and let it work without extra hints.
4. When it says it's done, read the diff, add the hidden checks, run the deciding checks, time your review, and write the results entry.
5. Remove the worktree with git worktree remove, so the next run starts fresh.

Step two is the one to understand. The starting commit is old. On that commit, CLAUDE.md might not exist, or the hook might not exist yet. But you aren't testing what the agent could do with last month's setup. You are testing today's setup against a problem whose answer you already know. Copying the current configuration in is what makes that possible. Anthropic's guidance also warns against starting Claude Code with the bare option for these runs, because bare mode skips loading CLAUDE.md, skills and hooks, and those are the very things you are testing.

When the three tasks are written down and have one results entry each, commit the evals folder. Now a later session, or you in a month, can rerun the same tasks and compare against the same record. Add one line to CLAUDE.md saying that changes to the agent setup are checked against the tasks in the evals folder before they are adopted. That way the routine survives even when you forget it.

What the model and effort settings change

Now you have something to measure with, so you can turn to the first real choice it helps with: which model runs the work, and how hard it thinks.

Recall what Claude Code is. It's an application, running in your terminal, that sends your request to a Claude model and lets that model use tools such as reading files, editing them and running commands. The application stays the same whichever model you pick. The model is the part that reads your code, decides what to do and writes the changes. So switching the model changes the thinking, not the tools.

Claude Code gives you short names, called aliases, for the main model families. Sonnet is the standard coding model. It's the one Anthropic recommends for most software work, and it balances skill, speed and cost. Opus is the more capable model. It's meant for subtle debugging, large changes across many files and careful design work. It costs more per token and takes longer to answer. Haiku is the fast, cheap model, for light edits, quick scripts and simple sorting jobs. There is also an alias called opusplan, which uses Opus while you plan and Sonnet while it writes the code. That is a sensible pairing when the hard part of a task is deciding what to do rather than typing it.

The reasoning-effort setting is separate from the model. It controls how much the model thinks before it answers. The levels are low, medium, high and extra high. Higher effort means the model spends more tokens working through the problem, which costs more and takes longer. Sometimes that catches a mistake. Sometimes it just spends more on a problem that was already easy.

You can think of these as two dials. The model is how capable the worker is. The effort is how long that worker stops to think before acting. A capable worker who rushes and a less capable worker who thinks carefully can land in similar places on easy work, and very different places on hard work. That's exactly the kind of claim your baseline can check.

Setting them in the terminal

In the terminal interface, there are a few ways to set these, and they apply at different scopes.

Inside a session, type slash model. It opens a picker that shows the models your account can use. You move with the arrow keys, and the same picker lets you adjust reasoning effort. If you press the letter s, the switch applies only to this session and isn't saved. There is also slash effort, which sets the effort level directly to low, medium, high or extra high.

When you start a session, you can pass the choice as an option on the command line. Typing claude, then dash dash model, then sonnet starts a Sonnet session. Typing claude, then dash dash effort, then high starts one at high effort. These options apply to that run only.

For a lasting default, use the settings file. Claude Code reads a key called model and a key called effort level. In your project's dot-claude settings file, which is committed and shared with anyone who clones the repo, you can set model to sonnet and effort level to medium. In your personal settings file in your home folder, you can set your own default across all projects. There's also a local settings file inside the project's dot-claude folder that stays uncommitted, for overrides only you want.

When these disagree, the order is simple. A command-line option wins over the files. The project's local file wins over the shared project file. The shared project file wins over your personal file. And if your organization deploys managed settings, those can limit which models are allowed at all.

There is one more way that's handy for a single hard question. If you include the word ultrathink in a request, the model reasons more deeply on that one turn, and your session settings stay as they were.

For your baseline runs, use the command-line options. They make each run's configuration obvious, and they don't leave anything changed behind you. You can write the exact launch line into the results entry.

Reading cost and usage

To fill in the tokens and cost field, you need to read them from the session. What you see depends on how you pay.

If you sign in with a subscription, such as Pro, Max or Team, type slash usage. It shows bars for how much of your allowance you have used, along with when the rolling five-hour window resets and how much of your weekly limit is gone. On a subscription you don't see dollars per task. You see a share of a limit.

If you use an API key and pay by the token, slash usage shows the session's cost in dollars, the time spent waiting on the API, the lines of code changed and the input and output tokens for each model. The slash cost command shows a similar running total for API accounts.

There's also slash context, which you met earlier when you carried the done-flag work across sessions. It shows how full the context window is, which is how much of the model's working memory the conversation has used. That matters for cost too, because a long conversation sends a lot of earlier text with each new request.

Two cautions. First, the numbers in the terminal are estimates made while the program runs. For real billing, the Claude Console usage page, or your cloud provider's console if you go through one, is the record that counts. For comparing runs in your baseline, the terminal estimates are fine, as long as you read them the same way every time. Second, on a subscription, write down the change in the usage bar between the start and end of a run. That gives you a rough share of your allowance per task, and that share is your cost.

What a usage limit does to a run

Limits can stop work partway, and that affects a baseline.

On a subscription, when you use up the five-hour allowance, Claude Code blocks new requests and tells you how long until it resets. If you have turned on extra usage, through slash extra usage or your account settings, the session moves to pay-as-you-go billing and keeps going. If you haven't, it waits. A weekly limit works the same way across both interactive and non-interactive sessions. On an API key, if your workspace or organization reaches a spending cap set by an administrator, requests start failing with a rate-limit error until the budget is raised. For runs without a person watching, there is an option called max budget in US dollars, which stops the run once the session's cost reaches the amount you set.

What this means for your baseline is easy to state. A run that hits a limit partway didn't fail the task. It was interrupted. Don't record it as not finished. Mark it as interrupted, throw it out and rerun it after the limit resets. Otherwise you'd blame the configuration for something the account did. It also means expensive configurations eat into your allowance faster, so running the full baseline twice under Opus at high effort may use a real share of a five-hour window. Plan the runs for a time when you have room.

Running the comparison

Now put the two halves together. You'll run the baseline under two configurations. The first is Opus at high effort. The second is Sonnet at medium effort, which is cheaper. The results below are illustrative, a supposition about how a run like this could go, not a measured claim about these models.

Run each task twice under each configuration. That's twelve runs in total: three tasks, two configurations, two runs each. The reason for two runs comes up again in the failure modes, but in short, a single run can be lucky or unlucky.

Say the empty-name validation task goes like this. Both configurations finish both runs. The diffs are small, around fifteen lines each, and in each case the agent added the check and a test. No regressions. Your review takes about three minutes each time, because the change is easy to read. Sonnet at medium uses far fewer tokens. On this task, the cheaper configuration is simply better. Same result, less cost, same review time.

Say the fast-add bug from the reproduction note also comes out even. Both configurations read the note, find the place where the second save replaced the first item, and fix it. The hidden regression test for two back-to-back submits passes in all four runs. One Sonnet run added an extra change to a nearby function that wasn't needed. That added two minutes of review while you worked out it was harmless. Still a fair result for the cheaper setup.

Now say the done-flag task, the one touching the database, splits them. Opus at high effort finishes both runs. The migration adds the column, the database client function reads and writes it, and the checkbox works through a reload. Sonnet at medium passes the suite and the type check both times. But on one run, the clicked journey fails. You tick an item, reload, and it comes back unticked. When you look, the interface sends the new value, but the database client function that saves the item never includes the done column in what it writes. The suite passed because no existing test covered saving the flag. You spend fifteen minutes finding and fixing it. On the other run it works.

This is why you wrote the clicked journey into the deciding check for this task. Without it, both configurations would have looked like they passed.

Deciding by total cost

Here is the rule for adopting a change. Adopt it only if all three of these hold. The finished behavior holds, meaning the deciding checks pass as well as they did before. No regressions appear. And the total cost, counting tokens and your review and repair time together, goes down.

To compare total cost, you need to put your minutes and the token bill on the same scale. The simplest way is to give your time a dollar value per hour. Say you decide your time is worth sixty dollars an hour. Then each minute of review or repair costs a dollar.

Now say, on an API key, the done-flag task costs about a dollar fifty in tokens per run under Opus at high effort, and about forty cents under Sonnet at medium. Say your review takes five minutes for each clean run. Under Opus, each run costs a dollar fifty in tokens plus five dollars of review, so six fifty. Across two runs, that's thirteen dollars. Under Sonnet, the clean run costs forty cents plus five dollars, so five forty. The broken run costs forty cents plus five dollars of review plus fifteen dollars of repair, so twenty forty. Across two runs, that's about twenty-six dollars. The cheaper configuration costs about twice as much once you count your time. The token savings were real, about two dollars, and they were swamped by fifteen minutes of repair.

On the validation task, the math runs the other way. Review time is the same for both, so the token savings are the whole difference, and Sonnet at medium wins.

If you're on a subscription, the reasoning is the same. You just weigh a share of your allowance against your minutes instead of dollars against minutes.

So the decision isn't one model for the whole app. It's a default for each kind of task. Small validation rules and bug fixes from a written reproduction note go to Sonnet at medium. Changes that touch the database layer go to Opus at high effort.

Writing the defaults down

Put the everyday default in the project settings file: model set to sonnet and effort level set to medium. That covers most work in the app. A settings file holds one default, though, not a rule per kind of task. So the per-task rule goes into CLAUDE.md, where both you and the agent will read it. Add a short line. It says that changes touching migrations or the database client are run with Opus at high effort, either by starting the session with those options or by switching with slash model before the work begins, and that this rule was chosen by the evals baseline, with the date. Then add the twelve results entries to the results files and commit everything together, so the record shows why the defaults are what they are.

When you next think about changing something, such as trying the opusplan alias for database work or rewriting the reproduce-bug skill, the routine is already in place. Rerun the baseline twice under the new setup, compare against the committed record, and adopt the change only if it passes the same three tests.

Where the comparison goes wrong

There are three common ways to fool yourself with a baseline, and each one showed up in the example above.

The first is comparing single runs. The agent doesn't produce the same work every time, even with the same model, effort and request. In the example, Sonnet broke the done flag on one run out of two. If you had run it once and it happened to be the good run, you'd have concluded it was just as reliable as Opus and switched your database work to it. Anthropic's testing guidance names two ways of counting this. One asks whether at least one of several attempts succeeded, which tells you what a model can do. The other asks whether all of them succeeded, which tells you whether you can rely on it. For your own app, the second matters more. You want the change to come out right without you rerunning it. Two runs per task is the minimum that shows any variation at all, and for a task you care a lot about, more runs tell you more.

The second is success criteria vague enough that both configurations pass. If the done-flag task had said "add a done checkbox" and the check had been "the suite passes," the broken run would have counted as finished. The comparison would have shown two equal configurations and sent you toward the cheaper one. Good criteria name the behavior a user would notice, like ticking, reloading and seeing it still ticked. When both configurations pass every task easily, that's also a hint. Either your setup is in good shape, or your tasks are too easy to tell setups apart. Anthropic's advice is to move tasks that always pass into a regression set you still run, and add harder ones to the main set.

The third is treating a lower token bill as savings while review time went up. This one is easy to miss, because the token bill shows up on a screen and your minutes don't. A configuration that writes bigger diffs, makes changes nobody asked for, or needs a second round of fixes can cost less in tokens and more in total. The only defense is the repair-minutes field, timed honestly, every run.

The baseline itself is a small thing: three task folders, some text files, a results record. What it changes is how you make decisions about the agent. Before, a new model or a rewritten skill was something you tried and judged by how the next session felt. Now it is something you test against tasks with known answers, compared by finished work, regressions and the full cost of getting there. The next chapter uses this record to judge bounded subagents, and after that you start moving work off the laptop onto a hosted machine.

One last thing before you run the baseline again: Claude Code has changed in the past week, and some of it touches this work.

Between the eighteenth and twenty-third of October, Anthropic shipped Claude Code versions two point one point three oh three through three ten. Most of it is fixes and security hardening. Four changes matter for the work in this course.

The first touches usage, which you just learned to read. Version three oh four added warnings for Pro and Max subscribers before you resume a long session with a large context. Resuming a session like that sends a lot of earlier text again, which eats into your allowance. The warning tells you how much of your rate limit the resume will use before it happens. The smallest useful action is this. When you see the warning during a baseline run, start fresh instead of resuming, so the earlier conversation doesn't land on that task's cost.

The second touches browser checks. Version three oh five added a sandbox setting called allow nested user namespaces, for Linux. Claude Code can run commands inside a sandbox, a fenced-off space that limits what those commands can reach. On Linux, that fence could stop headless Chromium, which is the browser that Playwright-style journey tests drive without a window. The new setting lets those tests run inside the sandbox. The action here applies only if you work on Linux with the sandbox on and want the agent to run the clicked journey as an automated test. Read the setting's documentation, decide whether that browser access fits your setup, and then test one journey run before relying on it.

The third touches running without a person watching. Version three oh six fixed sign-in failures in headless runs started with claude dash p. You'll use headless runs once the work moves to a hosted machine. If one failed to sign in before, update first, and then try again before you start debugging anything else.

The fourth touches review. Version three oh nine makes the ultrareview command, which you met as a review pass for clean trees, require explicit trust in the workspace before it runs. A script that called it without that trust will now stop. The action is to grant trust only for repositories you trust, with the documented trust option, and not to switch the check off globally.

One change sits outside this course's path for now. A new Claude API version, dated the twenty-first of October, is generally available. It removes the temperature and sampling settings and changes how keys are sent. That matters only if you write code against the Claude API or the Agent SDK directly. If you do, check those two things before you move to the new version.

To pick these up, update Claude Code. Then rerun one baseline task to confirm nothing in your setup shifted.