AIC GPU Time-Slicing for Sovereign LLM Inference
Platform: AIC 1.0.1 · Topic: Sharing one GPU server across many users by time-slicing
1. Introduction
AIC lets a customer share a single GPU server across many users by time-slicing it. Users take turns: at any instant, exactly one user is using the GPU. A user may be internal (the customer’s own staff) or external (the customer’s partners, or their own customers).
Each kind of workload is described by a VM template — a template packages a model and its runtime configuration. During a time slice, a VM built from the relevant template owns the GPU and serves the active user. When the slice ends, that VM is powered off and the next slice’s VM boots and takes over the GPU. All workloads here are inference (not training) as the design is for stateless VMs that can be re-used by multiple tenants and stoppable at any time. The CPU and GPU monitoring over the time slice will give information on the usage of the resources during the time slice.
Key physical constraint. The GPU server is a multi-GPU baseboard with a high-speed interconnect — for example, an 8× NVIDIA A100 SXM4 80 GB NVLink baseboard. The whole baseboard is passed through to a single VM (full PCI passthrough) rather than being partitioned, so the GPU behaves as a single, indivisible, serially-shared resource. Time-slicing rotates ownership of that resource among users’ VMs over time.
These are VMs and not Kubernetes nodes.
2. High-Level Slicing Scheme
The design is a pool of persistent, per-template VMs that rotate ownership of one GPU baseboard:
- One VM template per workload — encodes the model, resources, GPU passthrough, and slice schedule.
- One persistent VM instance per template — created once and kept for the lifetime of the engagement. Its disks persist across power cycles.
- Exactly one VM powered on at any time — the VM whose slice is active. All others are powered off and hold no GPU.
- Slice handoff = power off the outgoing VM (releases the GPU) → power on the incoming VM, which re-acquires the GPU, re-initialises the interconnect, and reloads weights into GPU memory.
- Rotation is driven by the platform’s native scheduled actions (recurring power-off / resume), optionally serialised by a state-change hook.
Because weights live on each template’s own persistent disk, a model is staged once — subsequent slices are a local reload, never a re-download.
3. Scheduling Scheme
Rotation uses the platform’s native scheduled actions. No external cron or extra tooling is required.
3.1 Recurring action attributes
Attribute | Meaning |
ACTION | VM action: resume (boot a powered-off VM), poweroff, … |
TIME | Time of day the action fires |
REPEAT | 0=weekly, 1=monthly, 2=yearly, 3=hourly |
DAYS | Selector interpreted per REPEAT (weekly: 0=Sun … 6=Sat) |
END_TYPE | 0=never, 1=after N repetitions, 2=until date |
resume is the action that boots a powered-off VM back up — this is where the cold start (GPU re-acquire + interconnect init + weight reload) is paid.
3.3 The GPU-handoff interlock
All template VMs reference the same GPU baseboard, and the platform treats that GPU as a resource only one VM can hold at a time. Therefore:
- The incoming VM must not start until the outgoing VM is fully powered off and has released the GPU.
- A short buffer between power-off and the next resume (e.g. 5 minutes) absorbs guest shutdown and device release, keeping the handoff clean.
4. Persistent Storage & Contents
Two persistent disks per template VM, deliberately split across storage tiers:
|
Disk |
Backing datastore |
Persistence |
Contents |
Hot path? |
|
OS / system |
Networked Block Storage |
Persistent |
Guest OS, GPU driver + CUDA, inference server, service config |
No — small boot reads |
|
Weights |
Local NVMe (host-local) |
Persistent, read-only |
Model weights/shards, tokenizer, model config |
Yes — the large per-slice load |
Rationale:
- Weights on NVMe keeps the per-slice load fast (local read) instead of pulling the model over the network each slice. Weights are read-only and re-stageable, so host-local placement is acceptable.
- OS on Networked Block Storage gives easy templating and durable persistence, and is off the hot path (boot reads are small).
- Weights staged once. Because each template’s weights disk persists, the model is staged one time and simply reloaded on every subsequent slice — never re-fetched.
What is not stored on the VM (kept in external stores):
- Conversation history / multi-turn context → external store
- RAG corpus / embeddings → external vector DB / object store
- Request logs / metrics / billing → external logging & metrics
- Attention cache → ephemeral in GPU memory; dies with the request
Inference is stateless at the VM level, so no user state is carried across slices — and a user’s data is never on the shared GPU hardware.
