Point a naïve scraper at it, push everything into VCF Operations, and you get two things. One is a time-series database (TSDB) full of _bucket{le="0.05"} series nobody will ever graph. The other is a dashboard that tells an operator nothing, in tremendous detail.

This two-parter is the pipeline that fixed that. Part 1 is the design: what to compute, what to drop, and why. Part 2 is the operations story: scheduling, self-monitoring and alerts.

The core decision: percentiles are computed here, not there

A Prometheus histogram is a set of cumulative bucket counters:

vllm:time_to_first_token_seconds_bucket{le="0.1"}   1203
vllm:time_to_first_token_seconds_bucket{le="0.25"}  4871
vllm:time_to_first_token_seconds_bucket{le="0.5"}   7120
...
vllm:time_to_first_token_seconds_bucket{le="+Inf"}  7412
vllm:time_to_first_token_seconds_sum                1812.4
vllm:time_to_first_token_seconds_count              7412

Prometheus turns that into a P95 at query time with histogram_quantile. VCF Operations has no such function: it stores gauges. So the pipeline does the interpolation before pushing:

function Get-Percentile {
    param([array]$Buckets, [double]$Total, [double]$Percentile)
    $target = $Percentile * $Total
    $prevLe = 0.0; $prevCount = 0.0
    foreach ($b in $Buckets) {                    # sorted by le, +Inf last
        if ($b.Count -ge $target) {
            if ($b.Le -eq [double]::PositiveInfinity) { return $prevLe }
            $frac = ($target - $prevCount) / ($b.Count - $prevCount)
            return $prevLe + $frac * ($b.Le - $prevLe)   # linear within the bucket
        }
        $prevLe = $b.Le; $prevCount = $b.Count
    }
}

That’s linear interpolation within the bucket that crosses the target rank, the same approximation Prometheus makes. Out come four flat gauges per histogram: p50_ms, p95_ms, p99_ms and avg_ms (from _sum/_count). Twenty-odd series become four, and they’re the four an operator reads.

It’s applied to time to first token (TTFT), end-to-end latency, inter-token latency, time per output token, prefill time, decode time, inference time and queue time.

Rates need memory

Counters (generation_tokens_total, request_success_total) are useless as absolute values. What you want is tokens per second, and that needs the previous sample. So the script keeps state. llm-metrics-state.json holds the last counters and timestamp for each target, and each run works out the deltas:

tokens_per_sec = (tokens_now - tokens_prev) / (t_now - t_prev)

The same trick goes one step further for live latency. _sum and _count are both counters, so Δsum / Δcount is the mean over the last interval, not the all-time mean the histogram gives you. That’s how you get a live_avg_ttft_ms that reflects the last 60 seconds instead of the last fortnight.

Two guards make this safe:

• Stale-state protection. If the gap since the last run is over 300 s (scheduler stopped, server rebooted), the baseline is dropped. Otherwise you’d get a diluted “per-second” rate, averaged over an hour.
• Restart detection. A counter that went down means vLLM restarted, because counters don’t go backwards for fun. The delta for that cycle is discarded.

Derived metrics: what the raw numbers won’t tell you

Two computed values earn their place at the top of the dashboard.

Queue pressure ratio: (waiting + swapped) / (running + 1). Above 1.0, more requests are waiting than being served, so scale out.

System saturation score (0–100):

$saturation = ($kvCachePct * 0.5) + ($queuePressure * 25.0)
if ($saturation -gt 100) { $saturation = 100 }

A full KV (key-value) cache alone scores 50, and queue pressure of 2.0 scores the other 50. It’s a heuristic, and it’s deliberately one number.

Think of it as a dial that goes red when the engine is about to start swapping requests to CPU memory. That’s the moment latency falls off a cliff. Warning is at 75, critical at 90.

The dial, as Ops draws it: one gauge climbing towards the warning line as KV-cache use and queue pressure rise together. (Test-mode data: see part 2.)

Also derived:

• prefix-cache hit rate, live and all-time;
• average request size, from the dropped http_request_size_bytes _sum;
• uptime in days;
• is_up = 1 on every successful scrape, so the absence of the metric is the alert.

That last one does its most useful work by not turning up.

What gets dropped, and why

The $DropPrefixes list is as important as anything computed:

Everything that survives is truncated to two decimals before the push. It’s a small mercy for the TSDB.

The key hierarchy

Ops shows metrics as a tree, so the names are designed for browsing:

vllm|system|is_up
vllm|throughput|total_tokens_per_sec
vllm|perf|live_avg_ttft_ms
vllm|perf|ttft|p95_ms
vllm|queue|pressure_ratio
vllm|memory|kv_cache_pct
vllm|cache|live_prefix_hit_rate_pct
vllm|process|rss_memory_gb

vllm | category | metric [| percentile]. An operator who has never seen vLLM can find “time to first token, 95th percentile” without a manual.

vllm | category | metric in the Ops metric picker — see part 2 for how this was captured.

Why this matters outside the lab

Organisations putting language models into service soon discover that “is it up?” isn’t the question. The real questions are these. How long are users waiting for the first word? Is the service about to run out of memory? Do we need another GPU before Friday?

This pipeline answers them inside the same VCF Operations console the infrastructure team already lives in. So AI services get the same capacity planning, alerting and dashboards as everything else. There’s no second monitoring stack, and no new team to staff it.

Rules learned

• Never push raw histogram buckets into a gauge-oriented TSDB. Interpolate P50/P95/P99 client-side and push four gauges.
• Counters need state: keep the last sample, compute deltas, and drop the baseline after a long gap (300 s) rather than dilute the rate.
• Δsum/Δcount gives you live mean latency, which is far more useful than the all-time mean.
• One derived saturation score beats six raw gauges at the top of a dashboard. Make it explainable (KV% × 0.5 + pressure × 25).
• The drop-list is a design artefact, not housekeeping. request_params_* alone can double your series count.
• Name for browsing: product | category | metric.

Broadcom documentation

• Super Metric Functions and Operators: the functions an Ops formula can use; none of them is a percentile
• Using the API with VCF Operations: the REST API the pipeline pushes through, and the Swagger reference on the appliance
• Generate a List of All Metrics for the Object: reading an object’s stat keys back, grouped with | as in mem|host_workload
• Symptom Definitions in VCF Operations: metric symptoms at Warning and Critical levels, as for the saturation score

Part 2: running it as a service, monitoring the monitor, and the four alerts.

Lab environment; opinions my own. Config shown is sanitised; the script’s test mode reads a local metrics file instead of a live endpoint.