Should you build an agent harness?

I see lots of opinions about it.

My thoughts:

As an AI engineer, learning how to build a harness is one of the best ways to stay ahead and unlock unique value from agents.

If you understand how to build one, you can, at a minimum, transfer that knowledge to tune whatever harness or set of harnesses (closed or open) you use.

In the best case, you apply your domain expertise to build domain-specific harnesses that unlock unique real-world value and solve reliability issues other companies just aren't willing to invest time in. If you haven't noticed, many companies and startups have already started doing this. Harnesses are enablers in that way.

I don't see any drawbacks in learning to build one.

The main pushback against building a custom harness is that models will get better at generating them on the fly, so why build one? Or that companies will provide harness-as-a-service, etc. Now, ask yourself: will you have the level of customization that a proper harness requires? See, you are not building a wrapper here; you are building an important part of your intelligence stack. Something you want to control completely.

Like automated prompt engineering, evals, and many other areas requiring extensive domain knowledge, harness engineering isn't something models are great at (see dynamic workflows from ant as an example). We assume too much that tools will remain static, data won't change, or knowledge will not evolve. A custom harness lets you own these issues and solve them at your desired pace. You simply cannot afford to sit back and wait for model providers to solve this problem for you. The harness is too important to offload.

While general frontier models get better at verifiable (math, code, and the like) tasks, I haven't seen evidence that they solve reliability issues when you apply them to domain-specific and more dynamic environments. This is why you want to understand how the harness works and potentially build your own. I see a lot of companies already doing this in bio, health, legal, and finance.

My other concern about just relying on a model provider to solve the harness for you is vendor lock-in. Right now, we mostly use single models for most tasks, but it's not hard to see a world where we leverage a set of frontier models (open and closed) to address issues like cost and diversity of intelligence. Are you going to rely on some company to build that harness solution for you, or, even worse, trust a single model to do that for you?

I can go on and on.

Building your own harness is about working towards building your own intelligence stack. I don't think that's optional where things are headed if you really want to have a differentiated business or offering.

So where do you get started?

I suggest feeding this list of seminal harness engineering papers to your agent: https://academy.dair.ai/papers/collections/harness-engineering

You can start with something like: "Summarize the main components of an agent harness by researching this list of papers and tools: https://academy.dair.ai/papers/collections/harness-engineering. Then put together a set of visual notes on where to get started to build my own minimal harness using <language_of_your_choice>."

Your thoughts? I want to keep this as an open discussion. Please share any concerns or thoughts. I'll share more thoughts as the conversation evolves.

https://x.com/omarsar0/status/2098809969252450451?s=20