Harness first, then the model

Disclosure: I work in Developer Relations at NVIDIA. This is a personal project, built on my own time and budget; NVIDIA did not review this post, and every number in it is my own measurement, not an NVIDIA benchmark.

A team of one

I have assembled a team of researcher and editor agents to be my personal intelligence and briefing team: an always-on system that reads the news, listens to the podcasts, watches the feeds, and files what it learns into a knowledge graph I can query and audit. It has a staff of one — me — which means it only earns its keep if it is cheap, fast, and reliable enough to leave alone. This is the story of how it became all three: by moving from closed frontier models to open weights, tuning the harness first and the model second.

The agent follows more than 30 international news sources and technical blogs, more than 20 podcasts, and a set of social media feeds. An extraction pipeline works through whatever is new and pulls out the companies, products, technologies, people, and events mentioned, along with how they relate: who builds on what, who partners with whom, who announced what. All of it lands in a Neo4j knowledge graph. Any claim the system makes can be traced back to where it came from and when it was true.

The graph is the foundation. A scoring layer rates every new fact against criteria I set, so a queue of thousands of tracked companies, technologies and products stays ordered by where my attention matters. A digest job selects the overnight stories worth keeping, extracts them into the graph, and writes me a morning briefing. A podcast job selects the stories from the graph, writes the scripts, and generates the audio with ElevenLabs.

And when I want to explore the graph, or change how any of this works, I talk to it. A sandboxed OpenClaw agent runs against the same backend, deployed on the NemoClaw pattern — NVIDIA's open-source reference stack for running agents inside policy-enforced OpenShell sandboxes. It carries about 40 tools and works in both directions. It answers questions grounded both in in the graph and in online research with citations. It also operates on the graph: it runs and configures jobs, records new standing instructions when I ask it to, and repairs data.

I built this alone, and I run it alone. Since starting this project a few months ago I have gradually moved it from slow closed frontier models to fast open-weight NVIDIA Nemotron models. To keep it reliable as I shifted toward efficiency, I had three levers to pull: Grounding in truth, tuning the harness, then tuning the model.

The first part is already on the table. It is the knowledge graph itself, with its source, confidence, and validity interval on every fact. An answer can be checked against a record rather than a recollection, and that is what makes the other two steps measurable at all: every harness change and every training run that follows is scored against that ground truth. This post focuses on the other two levers: tuning the harness and tuning the model.

First, though: what does it mean to tune a harness?

Everyone is tuning the harness

"Harness" has settled in as the name for everything around the model: which tools it sees and how they describe themselves, the prompt and context management, the schemas its output must fit, what a failed call returns, the budgets, the loops, the sampling settings. Most people want to treat all of that as boilerplate — write it once, swap models underneath. The finding of 2026 is that the harness is a tunable system in its own right, that the right harness is model-specific, and that tuning it can move results as far as switching to a better model can.

The clearest published arc starts with LangChain and NVIDIA. In July, LangChain tuned its Deep Agents harness for Nemotron 3 Ultra and reached the top accuracy among open models — within a point of the leading closed model's best run on their suite, at roughly a tenth the cost per run. As NVIDIA's announcement put it: "Every gain came from engineering the environment around the model, not the model itself."

More recently, the NVIDIA team behind the coding agent AVO reported 100% on the public set of the ARC-AGI-3 interactive reasoning benchmark, by its own action-efficiency measure — powered by Claude Opus 5, a model that scores about 30% there on its own. That result is self-reported, and NVIDIA's post is careful to note it is not directly comparable to ARC Prize's verified leaderboard, where Opus 5's 30% is still the top verified score. "A frontier language model is only one component of an AI agent," the AVO post reads; long-horizon capability "is a property of the full system." TechCrunch's headline was blunter: the harness, not the model, is now the real hero.

Hero is a strong word for scaffolding. But I had just spent July reaching the same conclusion, so I am inclined to allow it.

Tuning my way to open weights

When I first moved the system's model calls from a closed frontier model to Nemotron 3 Ultra — a 550-billion-parameter open-weights model, 55 billion of them active per token — it was not a plug-and-play replacement. The trace showed two problems feeding each other. Some tools were easy to misuse, and the Nemotron model was more likely to fall for badly phrased tool descriptions. And the model has a specific habit, which LangChain's playbook documents and which is where I took the method from: Nemotron follows guidance weakly when it sits in a standing rule — a system prompt, or a tool description — and strongly when it arrives in the conversation at the point of need, in a tool's return value at the moment it is used.

So I tuned the harness. The Nemotron-specific hints live in a per-model profile, on for Nemotron and off for models that do not need them, so the shared tool surface stays identical for every model.

My behavior suite on Nemotron 3 Ultra — [N] cases per behavior, scored against my own labels — returned to its previous frontier-model ceiling after the harness tuning work, and published token prices ran 4–14× lower depending on the job and which closed model I compared against; against the frontier models I actually migrated off, closer to 10×. And it was incredibly fast at responding, compared to the frontier model alternatives. By the end of July, most of the system's model calls ran on open weights, at the old behavior ceiling.

Can a small model keep up?

The other thing that changed this year is what small open models can do. The 30-billion-parameter class has become fast, cheap, and reliable at exactly the work an agent system does all day, and the pitch is that these models now match or beat much larger ones on that work. Nemotron 3.5 Lightning is the version of that argument I tested: 30 billion total parameters, about 3 billion active per token, for high-volume agent execution, and released in the open in August with weights, training data, and fine-tuning recipes included.

A model that is both blazing fast and an order of magnitude cheaper is not a marginal improvement. But I did not want to discover its weaknesses in production. I froze an evaluation suite of request and response chains from all thirteen jobs and ran the candidate models through it under identical conditions. For agent behavior I had a frontier model coding agent define 28 benchmarks — graph-grounded question answering, false-premise rejection, tool selection, write discipline — at 356 runs per model in the baseline round, scored against my own labels.

The 30B model came in equal to the 550B model on the agent suite while running 35% faster at the median, 24% faster over the full daily cycle, and 8.5× cheaper. Ultra is still what I reach for on frontier reasoning and extraction depth. Lightning's job is high-volume execution, so Lightning gets to run the high-volume throughput jobs like the agent lane and most nightly audit jobs.

A few real gaps remained, all in graph work, where I Lightning showed promise but was still underperforming. Extraction produced 23% fewer relationships than Ultra on identical articles, with high per-article variance. More edges only help if they are true, and I did not measure precision directly; the one quality signal I have is whether an article yielded relationships at all. Told to choose from a supplied list of candidate answers, Lightning went outside the list 8.2% of the time; Ultra, 1.0%.

Another gap dissolved before training began, and the way it dissolved matters. When a query returns nothing, the model should recover — reformulate, try a different tool — rather than loop or assert a confident wrong answer. One paragraph of standing instructions fixed this in the harness: the affected cases ([N] of them) went from 92% to 100%, and wall-clock time fell 44%. Some behaviors belong to the harness — which is the sequence doing its job: the harness takes what it can, and what remains is a short, well-evidenced list of gaps that are actually in the weights. I was curious to see if I could close the remaining gaps with some fine tuning. After all, I had accumulated over 70 million tokens of run logs on Ultra.

A 36-minute fine-tune

My training data was already there. The system logs every prompt and completion in production, so assembling a corpus took no annotation: 20,494 examples, 71.4 million tokens, spanning all thirteen job families and agent sessions with complete tool-call sequences.

Relationship extraction, the weakest measured behavior, made up 9% of its job family's tokens; attribute extraction, which had no measured failures, made up 44%. So I duplicated the relationship-extraction examples for a second pass but trained only a single epoch. Repetition lands on the weak behavior, at about 1/40 the compute of a second full epoch.

The recipe:

Notes for anyone reproducing this on a single GPU:

• Use the recipe's LoRA target modules as documented.
• Enable activation recompute. Peak memory exceeds 140 GB without it.
• The recipe defaults to two multi-token-prediction heads. I trained with one.

What the fine-tune bought

Same frozen suite, third model:

One targeted gap closed outright: staying inside a supplied candidate list reached Ultra's level, 1.4% out-of-list against Ultra's 1.0%. The other narrowed. Extraction moved from 23% below Ultra to 16% below — 216 relationships to Ultra's 257, which is 10% above the untuned base — and, unlike the base model's run, every article yielded relationships, at a median of 35 seconds per article. The company-detail gap narrowed without closing, and it is the one I would read least into: production answers never come from memory, because the graph exists precisely so that no model's recall has to be trusted. I track it as a canary for how much world knowledge survives in a small model, and the honest answer is: less.

Hill climbing in production

In late August I moved the agent lane — the daily briefing job and the interactive agent — to the 30B (extraction and story selection stayed on the 550B), but I soon noticed there was still some more work to do. The 550B's runs had executed a five-step trajectory: survey what landed overnight, read the evidence behind the top items, check my standing watch items, save a tiered briefing with per-claim citations, close out the cost — its benchmark morning produced an 18 KB briefing. The 30B's unattended mornings collapsed to a minimum-viable trajectory: the worst ran two tool calls in thirty seconds, flat-listed the survey into 2.8 KB, and asserted a check it had never run. The eval suite had cleared the cutover — 94.2% against the 550B's 92.4%. Both numbers are real. The suite had no case that scored a multi-step job trajectory, so the axis that regressed was one it could not see.

So the first fix was a scorer that replays the job in an isolated session and reads the service's access log: which prescribed steps actually ran, whether every claimed check maps to a tool call in the same turn — a claim without a call scores as fabrication — and whether the saved briefing carries citations, tiers, and an honest count of what it dropped. That scorer is the hill. From there, tuning is a loop: change one thing, run five interleaved trials per variant on the production stack, keep what climbs, revert what descends.

A checklist-steered agent trajectory on this job now costs $0.02–0.04 against the 550B's $0.385, and performs basically just as well.

Other problems were surfaced in the agent chat logs and fixed. Transcripts showed that the user (me) has asked to "update the graph", it was answered with a six-question clarification form, and it took three escalating instructions before anything was recorded. The fix is a point-of-need directive: when an incoming message matches write intent with a bare object, the relay appends one line — the object is this conversation's latest findings; execute now, then report what was written. Instructed-write follow-through went from 50% to 100%.

What stayed flat got named and routed, not re-prompted. Judgment did not move: under a "why this matters" instruction the model invents relevance for stories that have none, and wording dampens that without fixing the discrimination. That is what the 550B's 10–20× price buys, so the next step will be to implement model routing and have the agent switch to the bigger model when there is a need for judgement calls.

What happens next

First, where this actually stands. Production today is a deliberate mix, split by role: the base Nemotron 3.5 Lightning 30B — not the fine-tune — runs the high-volume agent lane at a fraction of the cost and latency; the 550B holds the seats where it pays its keep. A few cross-check and review jobs run on GLM models, so a reviewer never shares the assumptions of the model it checks. One job stayed on the top of the line frontier model – the podcast's script writer, to get the most high quality prose possible.

Custom weights used to be expensive twice over — costly to make, costlier to serve. The first half has collapsed: 36 minutes on a rented GPU made to fine tune Lightning. The second half is collapsing now. Self-hosters can already do this — NVIDIA NIM serves many private adapters over one hot base model — and at 3B active parameters, a workstation-class box is a viable serving target.

If you want to try this: the Nemotron 3.5 Lightning weights and model card are on Hugging Face, the fine-tuning recipes are in NVIDIA's Megatron-Bridge repository, endpoints are on build.nvidia.com or self-hosted as a NIM microservice, and the sandbox pattern is at github.com/NVIDIA/NemoClaw. LangChain's harness playbook is where I took the method from.