The Threshold I Set Before I Knew the Answer

I Wrote the Number Down First, Which Is the Whole Trick

A model I built scored 52.1% across 125 real outcomes. The number it had to clear was written down months earlier, before that model had produced a single result. It came up short by a fraction of a percentage point, so I left it switched off. It’s still switched off.

Deciding that took about four seconds, which I mention because “I made the hard call” is the sort of thing people say about themselves and it usually isn’t true. This one wasn’t hard at all. I’d done the hard part months earlier, back when I wrote the threshold down and genuinely had no idea which side of it I’d end up on.

Some context on what this thing even is. I run a prediction system for myself. It forecasts outcomes for a series of recurring scheduled events, and every prediction it makes gets checked against what actually happened. Nobody pays for it. Nobody sees the results but me. Part of why it exists at all is that I wanted to find out whether I’d follow my own rule on a day when following it cost me something. Twice now I’ve found out, and one of those two times I did not enjoy the answer at all.

Let It Run, Just Don’t Let It Matter

Shadow mode is the cheap version of being careful, and I want to describe it plainly, because it’s a lot less clever than the name makes it sound.

A shadow model runs on the same schedule as the live one, off the same live inputs, and writes its prediction to a table that only ever gets appended to. Nothing reads that table. Not the dashboard, not me on a Tuesday afternoon when I’m curious, nothing. When the event happens, the real outcome gets written next to the prediction that was made before it, and a record piles up one row at a time. The 125 outcomes behind that 52.1% arrived exactly that way, one event at a time, while the model’s output reached nothing and changed nothing.

The whole apparatus is one table and one scheduled job, and what it buys me is months of finding out on live data, with no way to quietly stop looking, before I have to decide anything.

The other half of it is a config flag that controls whether a model’s output reaches anything at all, separate from whether the code is deployed. So switching a model off is one setting, not a rollback. (Both models in this essay are sitting in the codebase right now, still computing away, both suppressed. If the underlying data ever changes I can go look again without rebuilding a thing.)

Which sounds like a small engineering detail and isn’t. When shutting a model off is expensive, you will find reasons the number is fine. You’ll decide the sample is too small, or that one bad stretch dragged the average down, or that another month of data would settle it. Every one of those thoughts costs you less than a rollback does, so the rollback loses, every single time. Make switching it off cheaper than arguing about it and none of that reasoning ever has to happen in the first place.

You Can’t Argue With a Number You Wrote Down First

52.1% is either a failure or an encouraging early signal, and which one it is depends entirely on when the threshold got written down. Written first, there’s nothing to discuss. You put two numbers next to each other and you’re done before lunch.

Written afterward, though, 52.1% becomes whatever you happen to need that afternoon. It’s directionally right. Or it’s close enough to justify collecting more data. Or it’s better than the previous version. Or it’s strong in the second half, as long as you agree to look only at the second half.

And every one of those readings is available to a completely sincere person who wants the project to continue, which is what makes this so hard to defend against. You don’t have to lie to talk yourself past a number you picked after you already saw it. Liars you can catch. This is the version where everyone involved means well and the project still doesn’t stop.

This is the fourth step of how I work with clients, and it’s the step that costs me money. Before any build work starts, we agree in writing on one number that defines success, how it gets measured, and over what period. When the period ends, I write down whether it hit. If it didn’t, my recommendation is to stop, in writing.

The threshold is a term of the engagement, same as the fee is. It needs a value, a way of measuring it, and a date, and all three have to exist before anybody writes code. A threshold with no date never closes, because more data is always arriving. A date with no threshold closes into an argument. Both of them go on the page before the work gets funded.

The Time the Record Itself Was Wrong

The 52.1% model was the second time I had to do this. The first time was worse.

An earlier model had been running for months against real outcomes and it looked strong. Strong enough that I’d described it that way out loud, to other people. Then I found a defect in the scoring. My scorer had been pairing some results with the wrong prediction rows. The predictions themselves were sound. The record of how they’d performed was not.

So I fixed the pairing and re-scored every prediction the model had ever made. The number inverted. The model had been losing, and it had been losing the entire time I was reading a record that told me the opposite.

Nobody would have known, and that’s the part that stayed with me, not the embarrassment. The defect was in my own code, in a system with exactly one user, reviewed by no one. Every incentive pointed toward correcting the pairing going forward and leaving the historical record right where it sat. That would have looked like diligence from the outside, and it would have produced a much nicer number.

Turning the model off mattered less than what came after it, which was rebuilding how grading works. Grading now reads from data frozen at the moment the prediction was made, instead of reconstructing after the fact what the state must have been. A prediction and the facts it gets scored against are written together and never recomputed. That whole class of bug can’t happen again, which matters to me more than the one instance I caught, because I have to assume there were instances I didn’t.

What Being Honest Actually Cost Me

Both of those corrections took something off me. The first one took back a claim I’d already made out loud. The second took a model I’d spent months building and wanted to put to work. Nobody forced either one on me, and both were hours of work spent making my own position weaker, on a system nobody was paying me to be right about.

I bring the cost up because the same structure shows up in how I price consulting work, where you can actually check it. The assessment is a fixed fee, agreed before it starts, and it’s the same fee whether I come back and tell you to build something or come back and tell you to build nothing. I don’t credit it against implementation work either.

That costs me money in the obvious case. An assessment that ends in “don’t build this” is an assessment that ends, and I’ve already been paid everything I’m going to get paid. It’s also the only arrangement under which my recommendation is worth reading at all, because a validation gate stops being worth anything the moment passing it is the profitable outcome for the person holding it.

What to Ask the Next Vendor Who Shows You an Accuracy Number

An accuracy claim can’t be wrong unless it arrives with the condition under which the vendor would come back and report a failure. The figure itself is usually real, by the way. I’m not suggesting anyone is inventing numbers. What’s missing is any result that would have produced a different conversation.

You can’t audit a vendor’s model, and you shouldn’t try. What you can audit is whether a gate exists, and a real one has three properties that are all visible in a document: a specific number, a specific way of measuring it, and a specific date by which it gets evaluated. If the answer to “what would make you stop” runs a full paragraph without containing a number, or if the date is “once we have enough data,” what you’re holding is a plan to keep going plus a vocabulary for describing whatever happens as progress.

Then there’s the follow-up, which matters about as much as the question does. A vendor can have a written threshold, and still be the party who decides whether it was met, and still get paid either way. So ask who records the result, on what date, and what happens to the contract if the number comes in low. Someone who’s done this before answers in one sentence and doesn’t need to go check with anybody first.

One question, before the first invoice. What number would make you tell me to stop, by what date will you know, and will you put both of those in the statement of work?