Programmable KV cache control

Warply does not reimplement paged attention or local prefix caches inside vLLM/SGLang. It orchestrates where prompts go, what gets reused, and what gets measured across disaggregated prefill/decode pools.

Engines own the substrate. Warply owns the control plane.

The problem

Custom inference workloads diverge on cache behavior, not just model size:

vLLM, SGLang, TensorRT-LLM, and Dynamo already ship strong local prefix caching. LMCache and Mooncake extend tiering to CPU, local SSD, and object storage; NIXL moves blocks across tiers.

What's still missing is a programmable layer that:

1. Routes requests to the prefill pool that already holds the matching prefix
2. Coordinates KV transfer between disaggregated pools without destroying hit rate
3. Applies per-workload tiering and eviction policy in Python — not YAML
4. Exposes hit rate, bytes cached, and affinity as first-class metrics

Disaggregation makes this harder. Prefill and decode are separate pools; naive routing loses prefix reuse.

What Warply owns vs. what engines own

Draw the line at placement, routing, tiering, and measurement — not rewriting KV block layouts or building a Dynamo KVBM clone.

Memory hierarchy (Phase 1)

Warply does not implement block allocators or tier movers. It exposes tiering policy; backends execute promotions and demotions.

Use cases: long RAG contexts that exceed GPU memory, warm prefixes across restarts, multi-tenant hot keys on cheaper storage.

Proposed API surface (Phase 1)

### Engine-level routing policy

import warply as wp

engine = wp.DisaggEngine(
    model="meta-llama/Llama-3.1-8B",
    prefill=wp.Pool("1xH100", replicas=2),
    decode=wp.Pool("1xH100", replicas=2),
    backend="sglang",
    kv_transfer="nixl",
    routing=wp.CacheRouting(
        strategy="prefix_affinity",  # route to pool with longest matching prefix
        min_hit_ratio=0.6,           # alert / adapt if below threshold
        tiering=wp.KVTier(
            l1="gpu",
            l2="cpu",           # pinned host — LMCache
            l3="ssd",             # local NVMe — LMCache / NIXL
            l4="object_store",    # S3/GCS — Mooncake / NIXL
        ),
    ),
)

engine.up()
client = engine.client()

### Per-request overrides

client.chat.completions.create(
    model="warply",
    messages=[{"role": "user", "content": "Summarize this document."}],
    extra_body={
        "warply": {
            "cache_key": "tenant:acme:rag-v3",
            "prefix_reuse": True,
        }
    },
)

### Observability

stats = engine.cache_stats()
# → hit_rate, bytes_cached, bytes_by_tier, evictions_by_pool, affinity_map, ttft_p50

Routing strategies (planned)

• `prefix_affinity` — send prefill to the pool with the longest matching cached prefix
• `kv-reuse` — generic cache-reuse score across pools (current landing-page primitive)
• `tenant_isolated` — namespace cache keys per tenant; route within tenant pools only
• `cost_aware` — combine spot pricing with cache affinity (Phase 2)

Why prefix cache is the highest-ROI first feature

1. Disagg breaks naive reuse — without cache-aware routing, every request cold-starts prefill
2. Measurable — hit rate, TTFT, and cost per token prove value quickly
3. Workload-specific — RAG, chat, and agents need different policies
4. Thin competition — few control planes expose programmable cache routing across clouds

Phase 1 pitch: *Warply routes to the prefill pool that already has your prefix — and sets tiering policy across GPU, CPU, SSD, and object storage.*

vs. NVIDIA Dynamo (KVBM and Planner)

Dynamo ships several related components. SSD/object-storage offload is not the Planner.

Warply overlaps Dynamo on routing policy and tiering policy, not on rebuilding KVBM or running under CRDs. Teams already on Dynamo K8s can use `export_yaml()`; teams on Lambda use Warply directly with the same tier backends.

Roadmap placement

Risks and constraints

1. Engine leakage — SGLang radix cache ≠ vLLM prefix cache. Abstract at routing/policy layer; expose engine-specific knobs via escape hatches.
2. Observability first — ship `cache_stats()` before advanced routing; users can't tune what they can't see.
3. Cold start — prefix affinity helps at steady state; document when it doesn't (unique prompts, high churn).
4. Scope — Warply orchestrates cache tiering policy; LMCache, Mooncake, and NIXL own block moves. Do not replace engine block managers or Dynamo KVBM.

Related primitives

• `route="kv-reuse"` — composable primitive on the landing page
• NIXL — Phase 0 disagg path on CUDA; extends to SSD and object-storage tiers
• MORI-IO — Phase 1 disagg KV transfer on ROCm ([ROCm/mori](https://github.com/ROCm/mori))
• LMCache / Mooncake — tiering backends in the ecosystem marquee

See the [Warply GitHub repo](https://github.com/warply-ai/warply) for implementation progress.