Should I use spot, on-demand, or reserved GPUs?

A rough price map

• On-demand: 1.00x — flexible capacity, no commitment.
• Capacity Blocks / reservations: ~0.35-0.75x — predictable GPU window, paid whether the GPUs are busy or idle.
• Spot: ~0.10-0.50x when healthy — more resumable work per dollar, but interruptions can delay completion.

These are planning ranges, not guarantees. Actual prices vary by instance type, region, date, capacity, and operating system.

The decision in three questions

What one batch workload showed

In one anonymized month-long GPU batch workload, we measured 3,771 jobs, 50,707 VM attempts, and 6,500 preempted VM attempts. Preemption touched 42.7% of all jobs.

There were two separate penalties:

• Extra billable setup — repeated setup after interruption was about 0.74% of estimated compute cost, using recorded provider cost where available and hourly estimates elsewhere. About 76% of preempted jobs had no measured repeated setup; among the rest, the median extra setup was 9 minutes. This excludes any extra data-transfer or egress charges from repeated download or sync work.
• Job latency — wall-clock waiting for a preempted job to get back to running. Across 537 recovery events, recovery added 5,234.1 wall-hours of waiting: median 37.1 minutes, average 9.75 hours, and p95 77.71 hours. This is user-visible delay, not necessarily extra billed compute time, and it can overlap across concurrent jobs.

Spot usually wins on price when jobs can absorb interruptions. Capacity Blocks trade lower flexibility for assured capacity in a paid window: they help when launch risk is more expensive than idle reserved time.

The four options

• Spot — discounted capacity the provider can reclaim at any time. Best when checkpointing makes the workload resilient to losing the VM: resumable training, sweeps, batch inference, experiments. Not suitable for hard deadlines or realtime latency requirements — recovery means downtime while a replacement VM starts.
• On-demand — full-price capacity with no commitment. Best for first runs, jobs without checkpointing, and latency-sensitive work while usage is still unpredictable. It does not guarantee capacity: a region can still be out of the GPU you want.
• Committed discount — AWS Reserved Instances, Google Cloud committed use discounts, Azure Reserved VM Instances. A billing agreement: commit to sustained usage and matching consumption costs less. It holds no GPUs for you — a discounted launch can still fail on capacity.
• Capacity reservation — AWS On-Demand Capacity Reservations, AWS Capacity Blocks, Google Cloud reservations, Azure capacity reservations. An inventory hold: the provider keeps specific capacity (zone, VM type, or block shape) for you, billed whether you use it or not. It does not lower the price by itself; combine it with a discount for that. AWS zonal Reserved Instances are the exception that does both at once.

A practical rollout

• Start with on-demand until the container, data path, and GPU fit are known.
• Move resumable jobs to spot after checkpoint and resume have been tested.
• Measure real GPU-hour usage for a few weeks before buying a commitment.
• Separate discount commitments from capacity reservations in your cost model.
• Re-check quota, live capacity, and provider terms before assuming a reservation applies to a new region or GPU type.

Where anycloud fits

anycloud can run the same submitted workload on spot or on-demand targets depending on the job configuration. It can also choose among valid unpinned targets when credentials and region are left open.

For reserved or capacity-specific setups, be more explicit. Pin the provider, credential, region, or VM type when the reservation only applies there; otherwise anycloud may pick another valid target that does not use the commitment.