Hybrid inference: one agent, two substrates

Almost every AI infrastructure conversation we join starts as a binary — cloud or on-prem — and ends as an allocation. Some of the workload cannot leave the building. Some of it is spiky enough that owning hardware for it would be absurd. The interesting engineering is in making both true at once without building two of everything.

This post is the architecture we land on most often, and the arithmetic behind it.

Start with five numbers

Placement debates are usually proxies for missing data. Write these down first and the argument tends to resolve itself.

Tokens per day

Input and output separately — they price and batch very differently.

Latency budget

Time to first token, and total, at the 95th percentile rather than the mean.

Concurrency shape

Peak concurrent requests over trough. A ratio of 3 and a ratio of 40 are different businesses.

Data classification

Which fraction of requests carry data that legally or contractually cannot leave a boundary.

Growth curve

Where volume lands in 12 months, with the confidence interval stated honestly.

The two that decide most cases are concurrency ratio and the regulated fraction. A workload with a peak-to-trough ratio of 40 and no regulated data belongs on elastic capacity — owning hardware for that peak means paying for idle silicon 95% of the day. A workload at a steady ratio of 3 with a third of requests carrying regulated data is paying a cloud premium for utilisation it could easily own.

The arithmetic on owning GPUs

The break-even calculation is simpler than vendors on either side would like.

Take an 8-way H100 node. Capital cost lands somewhere around 250,000 USD depending on how you buy, and you should amortise over three years, not five — the useful life of an accelerator is set by what replaces it, not by when it stops working. Add hosting: roughly 10 kW under sustained load, plus cooling, plus rack space, plus the fraction of an engineer who owns firmware, drivers and RMA. Call the fully loaded figure 8,000–10,000 USD per month.

The equivalent on-demand cloud instance runs several times that, so on paper ownership wins immediately. It usually does not, for one reason: utilisation.

Owned hardware costs the same at 15% utilisation as at 85%. Cloud capacity at 15% utilisation costs 15%. The real comparison is:

owned_cost_per_token  = monthly_fully_loaded / (tokens_per_month × utilisation)
cloud_cost_per_token  = published_rate × (1 - commitment_discount)

Under roughly 40% sustained utilisation, cloud almost always wins. Above roughly 60%, ownership almost always wins, often by a factor of three or four. Between those, the deciding factor is not cost at all — it is whether you have someone who wants to own GPU firmware lifecycles.

The portability contract

Hybrid only works if "the same workload" is literally the same artefact. The moment your on-prem deployment diverges from your cloud one, you own two platforms and the arbitrage you were chasing is gone to operational overhead.

Four things must be identical across substrates. Everything else can differ.

The container. One image, one digest, running on both sides. If the on-prem build needs different CUDA base layers, you do not have a hybrid platform — you have two products with a shared name.

The Kubernetes API. EKS on one side, Nutanix Kubernetes Platform on the other, same minor version, same core add-ons. Kubernetes is the entire reason this is tractable: it is the abstraction that makes a rack and a region substitutable.

The delivery pipeline. One Argo CD instance reconciling both clusters from one repository. Environment differences live in a Kustomize overlay — replica counts, node selectors, storage classes — never in a separate pipeline.

Identity and secrets. One OIDC issuer, one secrets authority. Two identity systems is how you end up with an on-prem cluster nobody has audited in eight months.

What legitimately differs: storage classes, load balancer implementation, node instance types, and autoscaling behaviour. Those belong in the overlay, and the overlay should be small enough to read in one screen.

Placement policy at request time

The routing decision is not per-service. It is per-request, and it belongs in a gateway rather than in application code.

policies:
  # Regulated data never leaves the private substrate, at any cost.
  - name: regulated-residency
    match:
      dataClassification: [restricted, pii]
    route:
      target: onprem
      fallback: none            # fail closed, never spill to cloud
    priority: 100
 
  # Interactive traffic prefers owned capacity, bursts when saturated.
  - name: interactive
    match:
      latencyClass: interactive
    route:
      target: onprem
      fallback: cloud
      spillover:
        queueDepthThreshold: 40
        p95LatencyMs: 2500
    priority: 50
 
  # Batch work goes wherever it is cheapest and can wait.
  - name: batch
    match:
      latencyClass: batch
    route:
      target: cloud-spot
      fallback: onprem
    priority: 10

Three properties make this work in practice.

fallback: none on the regulated policy is the important one. The default behaviour of every queue under pressure is to find capacity somewhere, and "somewhere" must be an explicit error rather than a compliance incident. Fail closed.

Spillover triggers on queue depth and observed latency, not GPU utilisation. A GPU pinned at 100% while serving within its latency budget is a GPU doing its job. Utilisation is a cost metric; queue depth is the user experience.

Batch defaults to Spot with an on-prem fallback, which inverts the usual instinct and is right: batch work is exactly what should absorb interruption, and owned capacity is exactly what should absorb the interruptions.

GPU scheduling on both sides

Kubernetes sees GPUs as opaque, indivisible resources by default, which wastes an enormous amount of expensive silicon. Both substrates need the same three mechanisms.

Device plugin and driver lifecycle. The NVIDIA GPU Operator handles drivers, container toolkit, device plugin and DCGM exporter on both EKS and Nutanix Kubernetes Platform. Using it on both sides means one upgrade procedure rather than two.

Partitioning. An embedding model that needs 8 GB should not hold an 80 GB accelerator. MIG carves an A100 or H100 into hardware-isolated slices with real memory and fault isolation; time-slicing oversubscribes with no isolation at all. Use MIG for anything multi-tenant, time-slicing only for development.

apiVersion: v1
kind: ConfigMap
metadata:
  name: mig-parted-config
  namespace: gpu-operator
data:
  config.yaml: |
    version: v1
    mig-configs:
      # Large accelerators split for embedding and reranking workloads.
      embedding-tier:
        - devices: [0, 1, 2, 3]
          mig-enabled: true
          mig-devices:
            "1g.10gb": 7
      # Generation keeps whole devices for KV cache headroom.
      generation-tier:
        - devices: [4, 5, 6, 7]
          mig-enabled: false

Queueing with preemption. Without it, an experiment submitted at 2am holds four GPUs while production autoscaling waits. Kueue or Volcano gives you priority classes, gang scheduling for multi-GPU jobs, and preemption that reclaims research capacity for production within seconds.

Serving is where the real gains are

The largest cost reductions in the systems we have worked on came from the serving layer, not the hardware.

Continuous batching. Naive request-per-forward-pass serving leaves most of the GPU idle. vLLM's continuous batching keeps the device fed by admitting new requests into the batch as others complete. On real chat traffic this is routinely a 5–10× throughput improvement on identical hardware — which is to say, most people's first "we need more GPUs" is actually a serving problem.

Prefix caching. Agents send enormous, near-identical system prompts on every call. Caching the prefix KV state cuts time to first token substantially for any workload with a stable preamble, which is essentially all of them.

Right-sized models per step. An agent loop does classification, retrieval, planning and generation. Only generation needs the large model. Routing the cheap steps to a small model typically removes half the token volume from the expensive path with no measurable quality change — and it is a routing change, not a hardware purchase.

Observability across two substrates

One dashboard, or you have two platforms again. The requirement is a single telemetry pipeline with a substrate label, not two stacks you mentally join.

Emit OpenTelemetry from the gateway, the agent runtime and the serving layer with a consistent resource attribute set:

deployment.environment   prod | staging
cloud.provider           aws | nutanix
k8s.cluster.name         prod-eu-1 | prod-dc-1
service.name             agent-gateway | vllm-generation | retrieval-api
kinekube.tenant          <tenant-id>
kinekube.model           <model-id>
kinekube.data.class      public | internal | restricted

With those attributes, four questions become one query each: cost per tenant per substrate, latency difference between substrates for the same model, spillover rate over time, and whether any restricted-class request has ever been served outside the private boundary. That last one is your compliance evidence, and it is worth designing the label set around it alone.

Export to New Relic if you want it correlated out of the box, or to your own Mimir and Tempo if you would rather own retention. The instrumentation is identical either way, which is the point of standardising on OpenTelemetry rather than a vendor agent.

What we would actually recommend

For most companies, in order:

1. Start on cloud. Your utilisation forecast is wrong and renting is how you buy time to be wrong cheaply.
2. Instrument from day one. Per-tenant, per-model token accounting is nearly impossible to backfill and is the input to every later decision.
3. Fix serving before buying hardware. Continuous batching, prefix caching and model right-sizing routinely reclaim more capacity than the purchase would have added.
4. Revisit at 40% sustained utilisation. That is when owning baseline capacity starts to pay, and when you have twelve months of real data to size it with.
5. Buy for the baseline, never the peak. Own the flat part of the curve, burst for the rest. The whole point of hybrid is that you no longer have to choose.

The failure mode we see most often is not choosing wrong. It is choosing early, with no instrumentation, and then defending the choice for two years because nobody has the data to reopen it.