Cold-start
Stop paying the cold-start tax on every tenant.
isovert lets tenants safely share one warm model instead of loading per tenant. On a single node we measured two independent levers on the same cost: ~21× at N=8 and ~10× at N=16 from prefix reuse, and 38.1× at 8k and 79.72× at 16k prefix from disk-baked prefix injection. They are alternatives, not additive. We're looking for one serving partner to prove it at fleet scale — and capture the saving first.
The cost we remove
The per-tenant load penalty, at scale
isovert changes the shape of the problem: a shared warm model, provable isolation between tenants, and the per-tenant load penalty largely gone.
Serving cost
Every tenant that needs its own model instance burns GPU-hours idling warm — or pays to reload on demand. The bill scales with tenants, not usage.
First-token latency
A cold tenant waits for a full model load before the first token. The usual fix — keeping everything warm — just moves the cost back onto the GPU bill.
The trade-off
Keep everything warm and it is expensive; accept cold-start and it is slow. Either way you are paying the per-tenant load penalty at scale.
What we measured — and what we didn't
Two levers, and the caveats that make them credible
Two independent mechanisms attack the same prefix-prefill cost. They are alternatives, not additive — never stacked, never combined into a single number. Each is quoted with what it measures.
Scope. Measured on a single hardware configuration (A100, Qwen 2.5 3B), eager execution path, transformer-only — it does not port to state-space models. Both figures are fork-side evidence: piecewise, full capture, N≤8. The shipped isovert-vllm extension is a separate column — N≤4, eager only, V1 runner — and these numbers do not transfer to it without a port run. Cross-hardware and fleet-scale are unproven — that is the design-partnership deliverable, not a claim.
Tenant density — the v2.4 sweep
4 to 32 tenants per GPU, depending on tenant size and latency target
Share one model across tenants instead of one instance per tenant. In the v2.4 density sweep the isolation gate passed at every tenant count tested — all 160 pairs, including at 32 tenants — and halving tenant size roughly doubled tenants per GPU at every latency target the ladder resolved. The gate is a gate on the density number, not a published isolation result; the adversarial isolation evidence is the separate metric-v2 work and the published paper.
| Prefix | 0.50 s | 0.75 s | 1.00 s | 1.50 s | 2.00 s |
|---|---|---|---|---|---|
| 2048 | 8 | 8 | 16 | 16 | ≥32* |
| 4096 | 4 | 8 | 8 | ≥16* | ≥16* |
| 8192 | 1 | 2 | 4 | 8 | ≥16* |
* Passed at the largest count tested — a ladder floor, not a hardware limit. Nothing above the registered ladder was measured, and no out-of-memory boundary was hit anywhere. Rotation-off (arm A) lands on the same rung at most operating points; it gains one rung at 2048 for the 0.75 s and 1.50 s targets and at 8192 below 1.50 s. Cost works out to $0.12–0.25 per tenant-hour at small tenants and moderate latency targets, at the $3.9492/hr H100 rate used.
Scope. Bare H100 80GB — no confidential-computing mode; no cell has ever run in an enclave. Qwen 2.5 3B Instruct, vLLM, eager execution, max_num_seqs 128, prefix lengths 2048 / 4096 / 8192. Median of five boots; cross-boot spread 10–21%, growing with N. Pre-registered before any GPU ran, commit 976aefd. Where we contrast this with one tenant per enclave, that denominator is vendor documentation — NVIDIA's Trusted Computing release notes (one GPU per confidential VM in SPT mode, MPS unsupported, MIG absent from the CC modes) and Azure's single-H100 confidential SKU — not our measurement. We have not measured inside an enclave.
The joint-validation offer
A bounded benchmark on your stack
- →
A bounded, roughly two-week benchmark run on your serving stack.
- →
You set the success bar and the kill criterion up front.
- →
A fast, honest no if it does not port to your hardware — the caveats are the point.
- →
If it holds, you capture the serving-cost saving before anyone else.
Why safe sharing, not just caching
The saving comes from provable isolation
Caching tricks can warm a model, but they can't let different tenants safely share one — shared model state leaks across tenants. The saving here comes from provable multi-tenant sharing, the isovert isolation core, which is what makes one warm model safe to share in the first place.
That means the cost win composes with a security property your regulated customers already want: per-tenant isolation you can prove, on commodity hardware, with no foreign hardware root of trust.