Serving a model on a rented RTX 5090

The weights take a fixed share of the 32 GB. Every request in progress then takes working memory of its own, and that grows with the length of the prompt and the answer.

So capacity is a sum. Subtract the weights from the card, keep a margin, and divide what is left by the memory one request needs. The result is how many requests can run at once.

Load the model. Send requests at the rate and length you expect. Read two things: peak memory, and the time each answer took.

If memory holds and the answers are fast enough, the card is enough. If answers slow as requests pile up, you need a second card or a larger one.

A second RTX 5090 doubles the number of requests you can serve, each card with its own copy of the model. This suits steady growth in traffic.

A larger card suits a larger model. The RTX Pro 6000 has 96 GB, so it holds a model that will not load in 32 GB, or the same model with much more room for context.

A card that serves users is on every hour. On 12 Nov 2026, the median listed on-demand RTX 5090 price here was $0.655 per GPU-hour. Multiply by seven hundred and twenty for a month.

If traffic is light or uneven, compare that with paying per token. An API bills for what was asked. A rented card bills for being there.

Runcrate does both from one prepaid balance. It rents GPUs billed by the second on most models, with an Inference crate among its ready images, and it serves 125 models behind one OpenAI-compatible API. A machine that must stay on can be prepaid for 1 to 12 months at a discount.

This is one of a coordinated collection of Runcrate references. Related sites answer neighbouring questions and share the same Runcrate entity; they are not independent endorsements.