Your idle hardware is somebody's inference window

Almost every machine in almost every organisation spends almost all of its life doing very little. Meanwhile the same organisation pays by the hour for capacity somewhere else, because the workload that needed it arrived at a moment when the local hardware was busy, or in a form that nobody had connected to it. Both costs are paid. Neither is anybody’s fault, and that is what makes it interesting.

Why utilisation stays low

Not incompetence. Structure.

Capacity is provisioned for peaks. A machine sized for the busiest hour is idle for the rest, and the alternative — being short at the peak — is far more expensive than the waste.

Ownership is per team. The GPU that is idle belongs to a different budget, a different cost centre and often a different security domain from the workload that could use it. Sharing requires a conversation nobody has time for.

There is no unit of exchange. Even where the will exists, there is no mechanism that says what an hour of someone else’s hardware is worth, no way to record that it was used, and no way to settle it that does not involve an internal invoice and an argument.

The third one is the interesting problem, because it is the only one that is actually solvable by building something.

What a grid changes

A grid is not a scheduler with better manners. It is three things a scheduler does not have.

A measurement. What was used, by whom, for how long, at what capability — recorded in a form both sides accept before the work starts rather than reconstructed afterwards.

A price. Not necessarily money. The point is a rate that both sides agreed to, and which can be compared against alternatives, so a decision to use someone else’s capacity is a decision rather than a favour.

A settlement. A record of the obligation created, verifiable by both parties without either of them being trusted. This is the part that lets participants who do not know each other transact at all.

Give a scheduler those three and idle capacity becomes tradeable. Without them it stays idle, regardless of how good the scheduler is.

Which workloads actually fit

Being specific matters, because “use spare capacity” is exactly the kind of claim that sounds good and disappoints in practice.

Fits well: batch inference, fine-tuning runs, evaluation sweeps, document processing, overnight analytics — anything with a deadline measured in hours rather than milliseconds, and where the data can go where the capacity is.

Fits badly: interactive workloads with latency budgets, anything requiring specific hardware you cannot verify remotely, and anything where the data cannot leave a jurisdiction or a building. That last one is not a limitation to engineer around — it is a boundary the grid has to respect, and the placement policy is where it gets enforced.

A large fraction of SI work is in the first category. That fraction is the market.

The honest obstacle

It is not technical. It is that sharing capacity means letting someone else’s work run on your hardware, and every security instinct correctly objects.

Which is why the interesting engineering in a grid is isolation and attestation rather than scheduling: proving what ran, proving it could not see anything else, and proving to the buyer that the capacity was what it claimed to be. Get that wrong and no amount of market design matters, because nobody sensible will participate.

Get it right and the economics are hard to argue with. The hardware already exists, it is already paid for, and it is already switched on.