Stop Counting Chats

A few weeks ago I sat in with a team trying to work out whether their AI rollout was actually working, and within minutes someone had pulled up a usage dashboard in Claude. We looked at the number of chats, then chats per user, then daily and weekly active users, and everyone in the room nodded along as though the picture was getting clearer. It wasn't. By the end we knew precisely how often the tool was being opened and almost nothing about whether it was doing anyone any good.

I've sat in enough of these rooms now to recognise the pattern. Nearly every measure of AI adoption I come across is really a measure of magnitude. How many chats, how many seats are active, how much of the monthly token allowance got spent, whether the team is "maxing out" the API. All of it counts volume, and volume happens to be the easiest thing in the world to count, which is exactly why it's the thing everyone reaches for first.

The trouble is that magnitude tells you nothing about value. Someone who runs the tool twenty times a day might be spinning their wheels, and someone who opens it once might have rebuilt the way they work through an entire week. The number that actually matters has nothing to do with how much AI a person uses and everything to do with whether what they use is any good at the job, and whether that's getting better over time. Effectiveness over magnitude.

Put plainly, the question worth asking is what share of your real work now runs through AI and comes out better for it, and whether that share is climbing month on month. That's adoption. The rest is activity.

When I started bringing my own team onto AI, the first thing I noticed was that people treated it as extra work. They went to it when they were stuck, or couldn't find an answer, or wanted to learn something new, which is all fine, except it leaves the AI sitting on top of the job instead of inside it. The shift that matters is when AI starts doing the work people were already doing anyway: the report that gets written every Friday, the data clean-up nobody enjoys, the first cut of a proposal. That's when effectiveness starts to climb, and it's also where most teams quietly give up, because the first time they point it at real work, it isn't very good.

Here's what they run into. On day one, AI knows a great deal about the world and almost nothing about you. It has never seen your projects. It doesn't know how your programmes actually run, what the last three years of work taught you, or the dozen unwritten rules your team carries around in their heads. So when someone tries it on their real job and gets back something generic, they conclude the tool can't do their work, and they go back to doing it by hand. They're right about that first attempt. They're wrong about what it means.

One of my engineers told me flatly that Claude couldn't handle what he did. So I sat with him, and the first response was genuinely rubbish, I'll be honest about that. But the reason was obvious once we looked. It didn't know the project, and it didn't have the particular capability his workflow needed. So we gave it the context, and we built a skill for that exact task, a small reusable piece of knowledge that taught the AI how the work was done here. Accuracy went from somewhere around 30% to 60%, and he started using it. A week later he had built a second version of that skill himself, we were at 80%, and now he reaches for it every day without thinking about it.

That work has a name. Institutional intelligence. It's the slow, deliberate business of taking everything your organisation knows, the context buried in documents and databases and people's heads, and turning it into something AI can actually use, whether that's a skill, a context file, or a connection to your existing systems. It's the difference between an intern who knows nothing about you on their first morning and a colleague who has been around for years. Almost the entire gap between "AI can't do my work" and "I use it every day" is just this, built or not built. Adoption follows effectiveness, effectiveness follows accuracy, and accuracy follows the institutional intelligence you've actually bothered to create.

So how do you get a team to do that work without it feeling like one more chore? For us, the answer hasn't been training. It's been hackathons. Every two weeks we block three hours on a working day, lay on pizza and good coffee, and hand out a certificate and a Ferrero Rocher at the end. That's the whole budget, and the outcomes have been wildly out of proportion to it. In six weeks my team went from building basic mockups to a working face recognition system, a retrieval setup over our own documents, and a multiplayer game running across the office network. People who had been wary of the whole thing were suddenly building.

The reason it works is the part that sounds backwards. The hackathons matter because they don't matter. Nothing rides on them, no appraisal, no deliverable, no client waiting at the other end, and because nothing rides on them people relax and get curious and start to play. Curiosity does the work that mandates never manage to. Once someone is genuinely interested, you stop having to worry about adoption at all, because they have started teaching the machine who you are without anyone asking them to.

Run different ones for different people. Your power users want a hard problem to disappear into for an afternoon. Most of the team wants something they can actually finish, so give them that. Build this proposal. Make a skill for that monthly report. Let them win, and then let them want the next one.

There's one mistake I made early, and I watch nearly every technical founder make the same one. You figure something out, it changes how you work, and your instinct is to get the whole team doing exactly what you do as quickly as possible. Hold on to that instinct, because it doesn't survive contact with reality. People grow at their own pace, and the goal that actually works is getting each person 10x from where they are today rather than getting everyone to where you happen to be. Those are different ambitions, and only one of them is achievable. The other just wears people out and leaves them feeling permanently behind.

You don't have to do all of it at once either. The pressure to roll everything out in six months is nearly always self-inflicted. Sequence it, and let the early wins create the pull for whatever comes next.

I'm roughly 10% AI and 90% human these days, and not because I held anything back. It's because the AI quietly folded itself into the work until I stopped noticing it was there. That's the target. Not more chats.