Fix the measurement before you change the model

Before you replace the model that makes moderation decisions, make sure you can see what the current one actually does. At Buy Nothing, doing that first changed what I thought the problem was.

Buy Nothing is a gifting platform: neighbours give things away and ask for what they need, free, in local groups. That openness attracts scams and spam. As fractional CTO I am responsible for the automated moderation that reviews new posts, comments, messages and sign-ups every few minutes.

A layered system with one weak layer

The service works in layers. Pattern rules catch the obvious cases, and a small, cheap model filters out what is plainly fine. A larger model classifies what remains. A final set of rules decides whether each result is acted on automatically, sent to a person, or dropped.

In the first half of 2026 that larger model came from Amazon's Nova family, and it was the weak layer. It read ordinary exchanges as threats: a member sharing a phone number to arrange a pickup, a family asking for help after a hard month, a giver announcing that a recipient had been chosen at random. The daily audit, in which a stronger model re-reads the system's flags, concluded that most of them were wrong.

The audit could not see enforcement

The obvious move was to swap in a better model. I looked at the measurements first, and three of them were broken.

• The action taken on each flag was never written back to its record, so the audit could not tell a flag that removed content from one that did nothing.
• The sampler meant to find missed scams kept re-examining content that had already been caught, and counted it as missed.
• The table built to compare two models side by side existed, but nothing was writing to it.

The headline therefore mixed labels the system merely recorded with actions it actually took, and it overstated what was being missed. Those problems need different fixes. I shipped the measurement changes on their own, with no change to what the system decided, and let the audit run on honest numbers before touching the model.

How do you choose a model for your own community?

Public benchmarks do not know what a pickup message looks like. The test that counts is your own traffic, so I ran candidate models in shadow. On a sample of live batches they classified the same content as the production model. Their answers were recorded, and nothing they said was acted on.

Claude Sonnet went first and showed the cost ceiling: too expensive to run on everything a community platform receives. Claude Haiku was the realistic candidate.

Two habits mattered. The first: disagreement is not correctness. Where the models differed, a stronger model judged each case against Buy Nothing's own policy, and I read the verdicts. Haiku was right far more often than Nova on messages and comments, which is where most enforcement happens. It also had weaknesses of its own: it was too quick to call contact sharing a scam, and less sure-footed on posts that tried to sell something.

The second: check what a metric counts. The first disagreement figure treated any difference in confidence score as a disagreement, which made the two models look incompatible. Counting different labels told a calmer and more useful story. The same care caught a sampling setting that had never reached production, so the shadow run was reading every batch instead of a fraction of them. The cost logs gave it away the same day.

The switch was a setting

The pipeline already chose its models from configuration, so the cutover was a configuration change rather than a release. Nova stayed on as the shadow, now watching the other way: any scam it would have caught and Haiku let through would show up in the comparison. Rolling back meant changing one setting and restarting the service, a matter of minutes.

A day later the reverse shadow showed no rise in missed scams, and the adjudicated disagreements favoured Haiku almost every time.

Cost was the other half of the decision. Haiku costs several times more than Nova per call, and nearly all of that is the long, fixed set of instructions sent with every batch. Caching those instructions removed most of the bill. One detail made it work: the default cache lasts five minutes and the pipeline runs every few minutes, so it would have expired between runs. A one-hour cache fits the rhythm. Two weeks later the invoice matched the projection.

What the model does not decide

The new model did not end false positives, and it was never meant to carry the system alone. When the audit found a new cluster of mistakes, the fix was a narrow rule that sends that pattern to a person instead of to an automatic ban. Those rules never wave content through; they only take the decision away from the machine.

Some abuse is invisible to a content classifier. A brand-new account can message stranger after stranger without writing anything that reads as abusive, and a scam ring can post the same bait from account after account. Both are now caught by behaviour: how an account acts, and whether its posts repeat scams that were recently removed.

What carries over

• Before trusting an audit's headline, check what it can actually see.
• Test candidate models in shadow on your own traffic; benchmarks do not know your users.
• Adjudicate the disagreements. Divergence tells you where to look, not who is right.
• Make the model a setting, and keep the old one watching after the switch.
• Price the model with caching, and make sure the cache outlives the gap between runs.

None of this is exotic. It is the unglamorous engineering that lets an AI system make decisions about real people and stay trustworthy while you improve it. It is also where an AI Delivery Review starts.