There is a habit in this line of work that nobody quite admits to, which is reaching for the newest model the way you reach for the freshest milk. Later version, bigger number, must be better. You swap it in without really thinking, feel briefly modern for having done it, and move on.
Most of the time it costs you nothing, because most of the time it does not matter. But every so often the new model is not the upgrade. It is the mistake. And the older thing you put down without ceremony turns out to have been the correct answer all along.
Two of my projects taught me this in the space of a few weeks, from opposite directions. In one, an old Llama beat the newer models in a fair fight. In the other, an old Gemini was the right choice precisely because the job was easy, and easy jobs do not want a genius, they want something quick and cheap that turns up on time.
None of what follows runs on real money. These are paper-trading experiments and side projects, built to learn from and to be wrong in public with. Keep that in mind whenever a number sounds impressive. It is fake money behaving impressively.
The model I would never have picked
The first project is the horse one. A panel of models looks at each runner and scores it, several opinions arriving at once, and the disagreement between them is the part that actually works. I have written before about how the arguing matters more than the agreeing. Same system, a little older and a little wiser.
For a while it ran on a pair of newer models, and that was fine, until they developed a wobble. Responses started coming back empty or half-formed, the JSON truncated, the answer simply missing. On a bad morning a third or more of the calls would drop. A system that depends on several voices does not cope well when the voices keep leaving the room mid-sentence.
So I did the unglamorous thing and held a bake-off. Threw an older Llama 3.3 70B into the mix as a fallback, mostly out of desperation. A model I would never have shortlisted on purpose, because why would you, it is old and everyone has moved on.
It won. Not narrowly, and not on a technicality. Clean, properly formed output every single time, where the newer models were dropping a third of their answers on the floor. Over a thirty-day backtest it picked better, forty-three point seven per cent against thirty-eight, across roughly six hundred and twenty-five decisions each. The return came out at plus fifty-five and a half per cent against plus twenty-eight, both flattered by the same optimistic accounting, so the gap between them is at least an honest one. The model nobody would have chosen quietly beat the ones I chose on purpose.
Where the benchmark was lying
The satisfying result is not the interesting part. The interesting part showed up only when I stopped looking at the panel as a whole and tested each model on the exact job it was doing. One of the newer models looked completely fine from the outside. Sensible output, nothing obviously wrong. But scored on two specific sub-tasks in isolation, it was worse than a coin toss, actively wrong more often than right on jobs it appeared to be handling without complaint. The whole-panel view had averaged that failure away and handed me a number that looked healthy. A model can top every chart you have heard of and still be quietly hopeless at your particular corner of the world.
Then there was the optimisation I was certain would work, and did not. Having scored each model per role, the obvious move was a dream team, best model in each seat. It was the worst configuration I tested, below even the plain single-model panel. Being best in isolation did not add up to best in company. The uniform old Llama, doing every job adequately, beat the assembled squad of specialists doing each job brilliantly and refusing to play together. I have watched enough real teams to have seen that one coming, but it still stung in the data.
The job that did not need a genius
The second project comes at the same lesson from the other side, and it is the one that changed how I think about this.
Somewhere in one of my systems there is a gatekeeper. Its whole job is to read an answer the system has produced and decide whether it is good enough to keep or should be thrown back. It runs constantly, called far more often than anything else, so every millisecond and every fraction of a penny it spends gets multiplied by an enormous number. Deliberately dull, high volume work.
So I benchmarked the candidates properly. Accuracy, speed, cost, across a couple of thousand decisions. And the accuracy came back almost identical, everything reasonable landing between ninety-eight and a hundred per cent. Not because the models are secretly the same, but because the task, in this form, is not hard enough to tell them apart. There is a ceiling, everyone is already touching it, and no amount of newer or bigger buys a single point above it.
Which means accuracy stops deciding anything. Once every candidate is tied at the top, the only things left that differ are speed and cost, and there the older, cheaper Gemini 2.5 Flash Lite wins outright. It costs roughly a third of what the newer Gemini 3.1 Flash Lite costs to do the same job to the same standard, and answers faster while doing it. You cannot beat a hundred per cent, so the newer model would have me paying three times over and waiting longer purely for the comfort of a higher version number. That is not an upgrade. It is a tax you volunteer for.
The bit where it bites
The old Gemini that wins that job is on a retirement schedule.
Google has confirmed that Gemini 2.5 Flash Lite, along with the rest of the 2.5 family, will be shut down no earlier than October 2026, with a firm date to follow once the next generation is fully out. The replacement is the newer, pricier tier. Which is to say: the model that is provably the correct choice for a huge slice of everyday work is being taken away, and the thing offered in its place costs more to do the same job no better.
Deprecating that model does not remove a bad option. It removes the optimal one. Easy, high-volume, cost-sensitive calls are not a rare edge case. They are most of what production systems do all day. Quietly retiring the cheap, fast, good-enough model pulls the best tool out of the largest drawer and hands you a more expensive one that does the same work while feeling like progress.
I have already lived the small version of this. A model I loved vanished from an API overnight with no notice, and half my projects got quietly worse for a day before I noticed. That was a free, obscure little thing, and losing it merely stung. Losing a model you have benchmarked, validated, and built a real dependency on, because a vendor decided its version number was too low to keep around, is a different order of loss. And when the best model for a job comes from a single vendor, as they often do, that is not a preference you hold. It is a dependency you cannot unwind, waiting for someone else’s roadmap to switch it off.
The ask
So here is the argument, and it is a plain one. When a company retires a closed model, the model does not stop being good. It stops being available. Gemini 2.5 Flash Lite will be, on the day it is switched off, exactly as capable as it was the day before, still the correct engineering choice for an enormous class of ordinary tasks. The only thing that will have changed is that nobody can use it.
Retiring a model is a business decision, and I understand the economics of not hosting yesterday’s inference forever. But there is a difference between not hosting a model and destroying it. Open source it on the way out. Release the weights, let it live wherever people want to run it, and let those of us who found it to be exactly the right tool for a real job carry on using it.
Do not kill the old models. They are cheap and fast and good enough, and good enough, when the task is easy and constant and everywhere, is not a compromise. It is the whole point. The best tool is rarely the newest one, and it would be a shame to only find that out after the good ones have been switched off.
None of the systems described here run on real money. They are paper-trading and research experiments, built to learn from and to be wrong in public with.
The views expressed in this article are my own and do not represent the views of my employer.