The Human-Shaped Fallacy

“Services are the new software” (Sequoia).

“Service as software” (General Catalyst).

There’s a lot of market commentary right now about service businesses being the new frontier, with AI as the technology that can enable them to scale and drive more attractive margins.

There are two versions of the bet. The first: build an AI service business from scratch and sell the finished work. The alternative, as pursued by Thrive Holdings: buy an existing firm, keep the customers and the revenue, but replace much of the labour and cost base with agents.

In both models, the theses are appealing. Service revenue dwarfs software revenue globally; AI enables automation of traditional human work; and if you can capture software-like margins in a service-scale market, then the rewards are plentiful!

The reality of the opportunity, though, is far more nuanced. In my first post in this series, I argued against a half-rewire and for a refounding: that agentic possibilities are only fully realisable if you look beyond traditional human-shaped work and design for how AI can do so much more.

But in service businesses, the word “human” is doing three jobs at the same time.

To use the example of financial advice, the same human adviser who spends a morning chasing paperwork and writing up meeting notes, is also the human connection and reason the client pays for an ongoing service, and the person who is regulatorily on the hook if the advice is wrong.

A spreadsheet view of the opportunity treats all three as one cost line. But the reality is that some of what is human about a business is not the process, or the order of work, but the output and requirements of the work itself.

This doesn’t diminish the opportunity, but rather sharpens it.

Preserving a human relationship does not require preserving the human workflow that supports it. The relationship, the decision and the responsibility are the output. The workflow underneath is not, and the fallacy, in both forms, is keeping it anyway: either by assuming that agents dropped into a human-shaped process will deliver the full gain, which is the half-rewire again, or by assuming that because the output must stay human, the process must too.

What happens when the work gets cheaper before the process changes?

We’ve had a few examples this year where the AI process has felt upside down. One of the most pointed was a contract negotiation we had. The process of the work remained as it always had: an initial draft from lawyers, our review, the counterparties’ review, iteration, negotiation and completion. But what changed was the hidden role of AI throughout. With multiple parties needing to sign the doc, some armed with legal counsel and others with Claude, we ended up in a cycle of lengthy drafting feedback, negotiation on a growing list of smaller points, often confusion and contradiction, and sometimes unawareness even of what was actually being asked for.

The question beneath the legal document drafting was straightforward: does this capture the commercial intent and protect the relevant parties’ interests? Answering it properly still takes legal expertise. But with everyone armed with AI, it became harder to tell which points actually represented the person raising them.

Dropping ‘shadow agents’ into a human-shaped legal process reproduced the activity of reviewing, not the contribution the review was there to get, which was a decision that somebody was answerable for.

Everyone is still in the process, as messengers between agents.

It is the facade of a human process, and a very easy one to fall into, because a fully formed answer in seconds takes away the incentive to do the hours of thinking the answer used to cost.

Which parts are still there for a reason?

Automating today’s tasks one by one is a legitimate first step, and it is where everyone starts, us included. But many tasks exist only to carry information between people, or to help one person understand what another has done, and once agents produce and consume that information, the need for those tasks changes too. Make them cheaper, and you simply get more of them. The question worth asking of every process, and the one I keep having to ask of our own, is this: once agents can do the work, which parts of the way the company runs are still there for a reason?

In that contract negotiation, the parts there for a reason were two sides answerable for a commercial decision, and a document that stands up. A redesigned version would let each side’s agents work through the drafting within agreed instructions, give each person an agent of their own to explain what the words mean for them, and bring the commercial judgements, passed across from one side to the other, to the relevant humans. It would still need human expertise along the way, because a small change of wording can significantly change the outcome and somebody has to be on top of that. But the human attention would be on those judgements rather than on every intermediate exchange. (Whose drafting agent is the better one, and whether you would rather be using the other side’s... the metaphorical jury is out.)

Don’t get me wrong: this is not an argument for taking people out of every interaction. Within software engineering, a more visible and prominent part of our day-to-day life at Sidekick, the coordination and integration of agents and humans is as important as having AI write the code itself. And our tooling reflects this. Agents don’t need Linear; humans need Linear. Having agents write tickets in Linear and post in Slack is a step forward, and it is the right step wherever a person contributes through those tools, redirects the work or stops it. But is it the right full step? The question is which interactions need a shared human interface, and how much of the agent activity underneath really needs to pass through it; my suspicion is that far less needs to pass through it than does today.

Where is the opportunity?

The same split is where the opportunity sits in any service where the product is the relationship, and financial advice is once again the obvious case to talk through. A client values knowing an adviser and having somebody to call when circumstances change. Everything around that relationship (the context gathering, the chasing, the reconciling, the preparing for the meeting) is the cost of producing it, but because the two have been entwined for so long, people mistakenly assume a personal service must also be an expensive service.

An adviser chasing a letter of authority, for example, is one person ringing another to ask about a third person’s account; where access and permissions exist, an agent can simply ask the system instead, and the chase goes away. Not all of the administration is like that, and some of it still needs a person’s judgement. But most of it does not need to be human-shaped, and little of it needs to wait for the meeting. The client is still spoken to in human cycles, because that is what the client is paying for.

Crucially, though, there is a limit. Some of the human work is simply supporting the service, and agents can take on a lot of that. But much of the human effort is also the service, and it doesn’t disappear just because the support has been automated underneath. Conversations with clients and being there to answer questions and talk through recommendations still take time.

An adviser whose preparation is done for them can know more clients, and know them better, but not an unlimited number.

Then who signs?

For a regulated business, accountability is also a key part of this puzzle, and it can’t be delegated. An executive remains responsible when agents do the work; under the Senior Managers Regime, that is not a matter of preference.

Yet if agents start producing significant volume increases that require a matching increase in human review, the business’s capacity remains limited by how much that accountable human can review. The risk is that the firm becomes much better at producing work, but that the system to stand behind it cannot keep up.

Removing the reviewer doesn’t solve it. Human review is still the control; what has to change is what the reviewer has in front of them and the tools they have for covering more ground. Embedded, not top-down; scalable, not dip-tested.

This chimes with the FCA’s Mills review of AI in retail financial services, published in July, which says it will not be enough to say that a person remains “in the loop”.

We already build the compliance context into the work from the start, so that the accountable person is reviewing something largely right rather than fixing something wrong. Today the evidence behind that review comes the traditional way, from a person going back over the work, and that is the control doing its job. The question now is whether the work can also produce the evidence the reviewer needs to challenge it, auditable and replayable, so that oversight scales without somebody replaying every step by hand, because an agent’s confident account of its own work is not enough on its own.

Agents masquerading as humans

I am still working out where the boundaries belong, and despite the nirvana of agents removing admin work, I do naturally still have an innate fear of letting go of tasks done in my name: the idea that an inbox agent could be sending emails on my behalf without me knowing what they say, because they would look like they came from me, and people would believe I was in control when I wasn’t. I think that’s the distinction: is an agent automating a task, running a process, or masquerading as a human under the hood?

It’s also the basis of some of my recent frustrations in general comms. There’s nothing worse than asking someone for an opinion (their real human judgement and perspectives), or messaging someone, and getting a response back that’s clearly been drafted by an agent. It makes the interaction feel hollow. It’s agents masquerading as humans.

In industries such as financial advice, where the product is the human, this matters more than most. The second the client suspects that the words behind the comms are agent-written, or that there isn’t really a human in control, the relationship and service diminish.

What is going to be clearer and clearer is the distinction between what is authentically human and what is agentically delivered. Substack now estimates the percentage of a post that is written by AI, which is as good a place as any to say how this one was made: my own voice notes, drafted into prose with the support of AI, then cut, redrafted and fine-tuned by me. That is the distinction. The judgement is human. The typing, increasingly, is AI-assisted.

The human-shaped fallacy

The hardest thing as a human is to think in a non-human way.

Even though I consciously know this, much of what I could pat myself on the back for automating this year still looks like human processes that are now more efficient, and the only defence is to keep myself honest about when I am doing it.

So a fully rewired service firm may still need more people as it grows, but for different reasons. The contributions that make the service worth paying for (the relationship, the judgement and the responsibility for the result) don’t disappear when the work underneath them does; they get more productive, and they are the costs worth paying.

An agentic service business does not automatically acquire software economics. Some of the cost of a personal service is the person, and it stays.

Agentifying doesn’t necessarily mean removing the human. It also doesn’t mean keeping a human process.