VCF Operations on The Nested Lab

vLLM metrics into VCF Operations, part 1: don't ship the histogram

Wed, 30 Sep 2026 00:00:00 +0100

vLLM’s /metrics endpoint is generous. Every model instance exposes a few hundred Prometheus series: counters, gauges and histograms. The histograms are the problem, with dozens of buckets each.

Point a naïve scraper at it, push everything into VCF Operations, and you get two things. One is a time-series database (TSDB) full of _bucket{le="0.05"} series nobody will ever graph. The other is a dashboard that tells an operator nothing, in tremendous detail.

This two-parter is the pipeline that fixed that. Part 1 is the design: what to compute, what to drop, and why. Part 2 is the operations story: scheduling, self-monitoring and alerts.

The core decision: percentiles are computed here, not there

A Prometheus histogram is a set of cumulative bucket counters:

vllm:time_to_first_token_seconds_bucket{le="0.1"}   1203
vllm:time_to_first_token_seconds_bucket{le="0.25"}  4871
vllm:time_to_first_token_seconds_bucket{le="0.5"}   7120
...
vllm:time_to_first_token_seconds_bucket{le="+Inf"}  7412
vllm:time_to_first_token_seconds_sum                1812.4
vllm:time_to_first_token_seconds_count              7412

Prometheus turns that into a P95 at query time with histogram_quantile. VCF Operations has no such function: it stores gauges. So the pipeline does the interpolation before pushing:

function Get-Percentile {
    param([array]$Buckets, [double]$Total, [double]$Percentile)
    $target = $Percentile * $Total
    $prevLe = 0.0; $prevCount = 0.0
    foreach ($b in $Buckets) {                    # sorted by le, +Inf last
        if ($b.Count -ge $target) {
            if ($b.Le -eq [double]::PositiveInfinity) { return $prevLe }
            $frac = ($target - $prevCount) / ($b.Count - $prevCount)
            return $prevLe + $frac * ($b.Le - $prevLe)   # linear within the bucket
        }
        $prevLe = $b.Le; $prevCount = $b.Count
    }
}

That’s linear interpolation within the bucket that crosses the target rank, the same approximation Prometheus makes. Out come four flat gauges per histogram: p50_ms, p95_ms, p99_ms and avg_ms (from _sum/_count). Twenty-odd series become four, and they’re the four an operator reads.

It’s applied to time to first token (TTFT), end-to-end latency, inter-token latency, time per output token, prefill time, decode time, inference time and queue time.

Rates need memory

Counters (generation_tokens_total, request_success_total) are useless as absolute values. What you want is tokens per second, and that needs the previous sample. So the script keeps state. llm-metrics-state.json holds the last counters and timestamp for each target, and each run works out the deltas:

tokens_per_sec = (tokens_now - tokens_prev) / (t_now - t_prev)

The same trick goes one step further for live latency. _sum and _count are both counters, so Δsum / Δcount is the mean over the last interval, not the all-time mean the histogram gives you. That’s how you get a live_avg_ttft_ms that reflects the last 60 seconds instead of the last fortnight.

Two guards make this safe:

Stale-state protection. If the gap since the last run is over 300 s (scheduler stopped, server rebooted), the baseline is dropped. Otherwise you’d get a diluted “per-second” rate, averaged over an hour.
Restart detection. A counter that went down means vLLM restarted, because counters don’t go backwards for fun. The delta for that cycle is discarded.

Derived metrics: what the raw numbers won’t tell you

Two computed values earn their place at the top of the dashboard.

Queue pressure ratio: (waiting + swapped) / (running + 1). Above 1.0, more requests are waiting than being served, so scale out.

System saturation score (0–100):

$saturation = ($kvCachePct * 0.5) + ($queuePressure * 25.0)
if ($saturation -gt 100) { $saturation = 100 }

A full KV (key-value) cache alone scores 50, and queue pressure of 2.0 scores the other 50. It’s a heuristic, and it’s deliberately one number.

Think of it as a dial that goes red when the engine is about to start swapping requests to CPU memory. That’s the moment latency falls off a cliff. Warning is at 75, critical at 90.

The dial, as Ops draws it: one gauge climbing towards the warning line as KV-cache use and queue pressure rise together. (Test-mode data: see part 2.)

Also derived:

prefix-cache hit rate, live and all-time;
average request size, from the dropped http_request_size_bytes _sum;
uptime in days;
is_up = 1 on every successful scrape, so the absence of the metric is the alert.

That last one does its most useful work by not turning up.

What gets dropped, and why

The $DropPrefixes list is as important as anything computed:

Dropped	Why
`python_gc_*`, `python_info`	runtime noise; static text can’t be graphed
`process_max_fds`, `vllm:cache_config_info`, `vllm:engine_sleep_state`	static configuration, not performance
`vllm:request_params_*`	cardinality explosion — a series per distinct `max_tokens`/`n`
`vllm:iteration_tokens_total`	redundant with tokens/s
`http_request_duration_highr_seconds`	“high-resolution” = hundreds of buckets; the standard one suffices
`http_request/response_size_bytes` buckets	dropped, but `_sum` harvested for an average
`*_created`	bucket-initialisation timestamps; pure noise

Everything that survives is truncated to two decimals before the push. It’s a small mercy for the TSDB.

The key hierarchy

Ops shows metrics as a tree, so the names are designed for browsing:

vllm|system|is_up
vllm|throughput|total_tokens_per_sec
vllm|perf|live_avg_ttft_ms
vllm|perf|ttft|p95_ms
vllm|queue|pressure_ratio
vllm|memory|kv_cache_pct
vllm|cache|live_prefix_hit_rate_pct
vllm|process|rss_memory_gb

vllm | category | metric [| percentile]. An operator who has never seen vLLM can find “time to first token, 95th percentile” without a manual.

vllm | category | metric in the Ops metric picker — see part 2 for how this was captured.

Why this matters outside the lab

Organisations putting language models into service soon discover that “is it up?” isn’t the question. The real questions are these. How long are users waiting for the first word? Is the service about to run out of memory? Do we need another GPU before Friday?

This pipeline answers them inside the same VCF Operations console the infrastructure team already lives in. So AI services get the same capacity planning, alerting and dashboards as everything else. There’s no second monitoring stack, and no new team to staff it.

Rules learned

Never push raw histogram buckets into a gauge-oriented TSDB. Interpolate P50/P95/P99 client-side and push four gauges.
Counters need state: keep the last sample, compute deltas, and drop the baseline after a long gap (300 s) rather than dilute the rate.
Δsum/Δcount gives you live mean latency, which is far more useful than the all-time mean.
One derived saturation score beats six raw gauges at the top of a dashboard. Make it explainable (KV% × 0.5 + pressure × 25).
The drop-list is a design artefact, not housekeeping. request_params_* alone can double your series count.
Name for browsing: product | category | metric.

Broadcom documentation

Super Metric Functions and Operators: the functions an Ops formula can use; none of them is a percentile
Using the API with VCF Operations: the REST API the pipeline pushes through, and the Swagger reference on the appliance
Generate a List of All Metrics for the Object: reading an object’s stat keys back, grouped with | as in mem|host_workload
Symptom Definitions in VCF Operations: metric symptoms at Warning and Critical levels, as for the saturation score

Part 2: running it as a service, monitoring the monitor, and the four alerts.

Lab environment; opinions my own. Config shown is sanitised; the script’s test mode reads a local metrics file instead of a live endpoint.

Telegraf on VKS: the dependency that isn't in the README

Wed, 23 Sep 2026 06:10:00 +0100

The Windows half of this series was about an agent that works on an OS the vendor hasn’t listed. This one is the opposite: a package on a fully supported platform that does nothing, silently, because of a dependency its own documentation doesn’t mention.

The supported route comes first. The rest is the path underneath, as it ran on f06 (VCF 9.1).

The symptom

Install the Telegraf package on a VKS cluster with the metric proxy flag set (isMetricProxyConfigured: true, which you want when metrics go to VCF Operations). Every Telegraf pod then stays in ContainerCreating:

NAME           PACKAGE NAME                         PACKAGE VERSION         DESCRIPTION                                                            AGE   PAUSED
...
telegraf       telegraf.kubernetes.vmware.com       1.34.4+vmware.2-vks.1   Reconcile failed: Error (see .status.usefulErrorMessage for details)   15m

kapp: Error: waiting on reconcile deployment/telegraf (apps/v1) namespace: tanzu-system-telegraf:
  Finished waiting unsuccessfully:
    Deployment is not progressing:
      ProgressDeadlineExceeded, message:
        ReplicaSet "telegraf-7cd76f96" has timed out progressing.

NAME                      READY   STATUS              RESTARTS   AGE
telegraf-26ppl            0/1     ContainerCreating   0          15m
telegraf-7cd76f96-wbthp   0/1     ContainerCreating   0          15m
telegraf-g4qlm            0/1     ContainerCreating   0          15m
telegraf-h2pcq            0/1     ContainerCreating   0          15m

Describe one, and the reason is FailedMount: two secrets don’t exist.

...
  Warning  FailedMount  5m14s (x13 over 15m)  kubelet            MountVolume.SetUp failed for volume "metrics-proxy-tls-config" : secret "metrics-proxy-tls-config" not found
  Warning  FailedMount  3m12s (x14 over 15m)  kubelet            MountVolume.SetUp failed for volume "telegraf-configs" : secret "metrics-proxy-http-config" not found

Nothing creates them. The package doesn’t. The cluster doesn’t. The PackageInstall reports ReconcileFailed and backs off, and there it stays, waiting for two secrets that aren’t coming.

The supported route first

On VCF 9.1, VKS add-on management installs Telegraf and Prometheus for you. Broadcom’s page on enabling monitoring for VKS clusters (linked at the end) puts it plainly: “After the prerequisites are met, any new VKS cluster is automatically monitored.” It speaks only of new clusters. The prerequisites, and how to check each:

VCF Operations monitors the vCenter and its Supervisor: Enable vSphere Supervisor Collection in the vCenter account’s Advanced Settings, true by default. Look for a vSphere Supervisor adapter instance and VKS Cluster objects in the inventory.
The Metrics Aggregator runs on the Supervisor. VCF Automation installs it by default. Check the Supervisor’s services in vCenter.
The add-on repository offers Telegraf and Prometheus: AddonRepository, Addon and AddonConfigDefinition resources on the Supervisor.

On 9.0.1 and later, Tom Fojta’s Monitoring VKS Cluster in VCF Automation installs the same two packages as add-on tiles in VCF Automation, once the provider enables the Supervisor Management Proxy. His notes include a Telegraf failure over an existing metrics-proxy-tls-config secret: the mirror image of the one above.

f06 took neither route on 23 August. It ran 9.1, but its VCF Automation arrived the next day. Ops had a vSphere Supervisor adapter instance, the Metrics Aggregator wasn’t registered, and nobody looked for add-ons. By nobody, I mean me. The lab’s catalog item installed the standard package itself, following Christian Ferber’s vrealize.it write-up for 9.0.1. That path is the rest of this post.

Two days later, with VCF Automation deployed and the Supervisor rebuilt, the automatic route left a footprint. A cluster created through VCF Automation’s API had Telegraf and Prometheus namespaces within two minutes, none of it mine:

NAME                     STATUS   ROLES           AGE    VERSION
vks-demo01-jn7xr-2gkq9   Ready    control-plane   111s   v1.35.5+vmware.1
--- namespaces ---
NAME                                 STATUS   AGE
...
tanzu-system-monitoring              Active   18s
tanzu-system-telegraf                Active   11s
...

The rebuilt demo01 had them too, before the catalog item’s package stage ran. So the item’s PackageInstalls collided with another kapp app:

=== telegraf ===
kapp: Error: Ownership errors: - Resource 'clusterrole/telegraf-kubelet-metric-access (rbac.authorization.k8s.io/v1) cluster' is already associated with a different label 'kapp.k14s.io/app=1787648814041772277' - Resource 'serviceaccount/telegraf-sa (v1) namespace: tanzu-system-telegraf' is already a
...

I didn’t inspect what installed them (the Supervisor was serving the addons.kubernetes.vmware.com API by then). It’s a footprint, not a test. Check for Telegraf before you install it.

What sits underneath

The two secrets come from the Supervisor Management Proxy, a Supervisor Service on the Supervisor, not in the guest. It runs envoy behind a load balancer on port 10093.

Each guest cluster gets a headless Service pointing at it (supervisor-management-proxy in default), plus the two secrets in kube-system, each with a SecretExport. The package’s SecretImports copy them into tanzu-system-telegraf, and Telegraf posts to https://supervisor-management-proxy.default.svc.:10093/arc/tkgs/metric.

Solid: what f06 showed. Dashed: the 9.1 design, in which the Supervisor also issues Telegraf’s certificates and cert-manager rotates them.

The manual path needs a Supervisor with a load balancer, registry access to projects.packages.broadcom.com, vCenter rights to register Supervisor Services, and cluster-admin in the guest. The cluster is demo01: one control-plane node and two workers.

1. Install the Supervisor Management Proxy

The lab’s catalog item that deploys a Supervisor has an installMgmtProxy tick box: register the service, then install it. Every attempt was refused. The definition shows why:

apiVersion: data.packaging.carvel.dev/v1alpha1
kind: Package
metadata:
  name: supervisor-management-proxy.vmware.com.0.4.1
  annotations:
    appplatform.vmware.com/use-system-vpc: "true"
    appplatform.vmware.com/required_capability: Load_Balancer_Supported
    appplatform.vmware.com/vcenter-version-constraints: '>=9.0.0'
spec:
  refName: supervisor-management-proxy.vmware.com
  version: 0.4.1

The proxy asks for the system VPC. vCenter only places services there that it can verify as published by Broadcom:

=== install on supervisor domain-c9 ===
  install failed: 500
...
      "default_message": "Service supervisor-management-proxy.vmware.com version 0.4.1 is not a trusted service published by Broadcom and cannot be placed on the system VPC.",
      "id": "vcenter.wcp.appplatform.signature_verification.system_vpc"
...

Registered through the API as Carvel YAML, it never passed, even byte-identical to Broadcom’s download. The other Carvel services here carry no such annotation and installed fine:

harbor                       2.14.2+vmware.2-vks.2    no system-vpc annotation  content_type=CARVEL_APPS_YAML
cci-ns                       9.1.0-embedded+739b5075  no system-vpc annotation  content_type=CARVEL_APPS_YAML
velero                       1.9.0-embedded+25369333  no system-vpc annotation  content_type=CARVEL_APPS_YAML

That check is doing its job, and I won’t show how the lab got past it. On a platform you care about, install the proxy the way Broadcom’s proxy page (linked at the end) describes, so vCenter can verify what it places on the system VPC. However it goes in, check it on the Supervisor:

  [30s] service=CONFIGURED  pod=Running ready=1/1
--- LB services in proxy ns ---
  workload-metrics-loadbalancer: type=LoadBalancer ip=10.26.20.21 ports=10093
...

Then check it in vCenter. The install record holds only the namespace the platform chose (the base64 is namespace: svc-supervisor-management-proxy-acqyd):

--- current install record ---
{
  "desired_version": "0.4.1",
  "current_version": "0.4.1",
...
        "default_message": "Reason: ReconcileSucceeded. Message: Reconcile succeeded.",
...
  "yaml_service_config": "bmFtZXNwYWNlOiBzdmMtc3VwZXJ2aXNvci1tYW5hZ2VtZW50LXByb3h5LWFjcXlkCg==",
  "service_namespace": "svc-supervisor-management-proxy-acqyd",
  "display_name": "Supervisor Management Proxy",
  "config_status": "CONFIGURED"
}

2. Create the cluster with serviceDomain

clusterNetwork.serviceDomain can’t be added later (step 7). The catalog item now sets it on every demo cluster. Here’s demo01 as rebuilt:

{
  "clusterNetwork": {
    "pods": {
      "cidrBlocks": [
        "192.168.0.0/16"
      ]
    },
    "serviceDomain": "cluster.local",
    "services": {
      "cidrBlocks": [
        "10.96.0.0/12"
      ]
    }
  },
...

3. Add the package repository

Telegraf ships in Broadcom’s VKS standard packages. Register the repository in the guest’s tkg-system namespace:

apiVersion: packaging.carvel.dev/v1alpha1
kind: PackageRepository
metadata:
  name: broadcom-standard-repo
  namespace: tkg-system
spec:
  fetch:
    imgpkgBundle:
      image: projects.packages.broadcom.com/vsphere/supervisor/packages/2025.8.19/vks-standard-packages:v2025.8.19

packagerepository.packaging.carvel.dev/broadcom-standard-repo created

NAME                     AGE   DESCRIPTION           PAUSED
broadcom-standard-repo   60s   Reconcile succeeded

=== packages available ===

NAME                                                                  PACKAGEMETADATA NAME                             VERSION                  AGE
...
telegraf.kubernetes.vmware.com.1.34.4+vmware.2-vks.1                  telegraf.kubernetes.vmware.com                   1.34.4+vmware.2-vks.1    54s
telegraf.tanzu.vmware.com.1.32.1+vmware.1-tkg.1                       telegraf.tanzu.vmware.com                        1.32.1+vmware.1-tkg.1    54s

4. Install Telegraf

Two values matter, from the package’s own schema:

--- telegraf package valuesSchema keys ---
...
  domainName  (default: cluster.local)
...
  isMetricProxyConfigured  (default: False)
  namespace  (default: tanzu-system-telegraf)
...

The catalog item installs it the Carvel way: a values Secret, then a PackageInstall, in a package-installs namespace whose service account has cluster-admin.

    function pkgSecret(name, valuesYml) {
        k8sApply(gk, gTok, "/api/v1/namespaces/package-installs/secrets", {
            apiVersion: "v1", kind: "Secret",
            metadata: { name: name, namespace: "package-installs" },
            stringData: { "values.yml": valuesYml } }, "secret " + name);
    }
    function pkgInstall(name, refName, ver, valuesSecret) {
        ...
        var pi = { apiVersion: "packaging.carvel.dev/v1alpha1",
            kind: "PackageInstall",
            metadata: { name: name, namespace: "package-installs" },
            spec: { serviceAccountName: "pkgi-sa",
                ...
                packageRef: { refName: refName,
                    versionSelection: { constraints: ver } } } };
        if (valuesSecret) { pi.spec.values = [{ secretRef: { name: valuesSecret } }]; }
        ...
    }
    ...
    pkgSecret("telegraf-values",
        "domainName: cluster.local\n" +
        "isMetricProxyConfigured: true\n");
    pkgInstall("telegraf", "telegraf.kubernetes.vmware.com",
        "1.34.4+vmware.2-vks.1", "telegraf-values");

The pods land in tanzu-system-telegraf: one per node from a DaemonSet, plus one Deployment pod.

5. Watch the secrets arrive

This is the dependency. Within two minutes of the proxy reporting CONFIGURED, the secrets and their SecretExports were in demo01’s kube-system. The SecretImports had copied them across:

--- secrets + secretimports in tanzu-system-telegraf ---
  secrets: metrics-proxy-http-config, metrics-proxy-tls-config, telegraf-registry-creds
  secretimport: metrics-proxy-http-config -> ReconcileSucceeded=True
  secretimport: metrics-proxy-tls-config -> ReconcileSucceeded=True
  kube-system secretexport: metrics-proxy-http-config toNamespaces=
  kube-system secretexport: metrics-proxy-tls-config toNamespaces=
...

The pods came up on their own:

  [30s] telegraf pods ready: 0/4  pkgi: ReconcileFailed=True
  [60s] telegraf pods ready: 2/4  pkgi: ReconcileFailed=True
  [90s] telegraf pods ready: 4/4  pkgi: ReconcileFailed=True
...
  [480s] telegraf pods ready: 4/4  pkgi: ReconcileFailed=True

After an hour and forty minutes of FailedMount, all four were Running four minutes after the proxy went CONFIGURED. I’d quite like that hour and forty minutes back.

6. Clear the stale backoff

The PackageInstall stayed in its ReconcileFailed backoff with every pod Running. Bumping an annotation forces a fresh reconcile:

$K=@{Authorization="Bearer $($lg.session_id)"; 'Content-Type'='application/merge-patch+json'}
...
$u="https://${gk}:6443/apis/packaging.carvel.dev/v1alpha1/namespaces/package-installs/packageinstalls/telegraf"
...
# force re-reconcile by bumping an annotation
$patch='{"metadata":{"annotations":{"lab.kick":"'+(Get-Random)+'"}}}'
Invoke-RestMethod -Method Patch -Uri $u -Headers $K -Body $patch -SkipCertificateCheck | Out-Null

...
kicked; polling...
  [30s] ReconcileSucceeded=True Reconcile succeeded

7. Check the output URL

Telegraf still wasn’t delivering. Its log, one error a minute:

  2026-08-23T19:17:56Z I! [agent] Config: Interval:5m0s, Quiet:false, Hostname:"telegraf-7cd76f96-wbthp", Flush Interval:1m0s
...
  2026-08-23T19:20:03Z E! [agent] Error writing to outputs.http: Post "https://supervisor-management-proxy.default.svc.:10093/arc/tkgs/metric": dial tcp: lookup supervisor-management-proxy.default.svc. on 10.96.0.10:53: no such host
  2026-08-23T19:20:56Z E! [agent] Error writing to outputs.http: Post "https://supervisor-management-proxy.default.svc.:10093/arc/tkgs/metric": dial tcp: lookup supervisor-management-proxy.default.svc. on 10.96.0.10:53: no such host
...

Note the trailing dot, and nothing after svc. The name is that headless Service:

--- guest 'default' ns services ---
  kubernetes: type=ClusterIP extName= clusterIP=10.96.0.1
  supervisor: type=ClusterIP extName= clusterIP=None
  supervisor-management-proxy: type=ClusterIP extName= clusterIP=None
...
--- endpoints of supervisor-management-proxy (default ns) ---
  ips=10.26.20.21 ports=10093

The domain part follows the cluster’s serviceDomain, which demo01 lacked. The package’s own domainName was cluster.local all along, and made no difference. Adding the field later was refused:

patch rejected:  { "kind": "Status", "apiVersion": "v1", "metadata": {}, "status": "Failure", "message": "admission webhook \u0022capi.validating.tanzukubernetescluster.run.tanzu.vmware.com\u0022 denied the request: spec.clusterNetwork is immutable and cannot be upd
...

Two fixes, both applied:

Every new cluster: serviceDomain: cluster.local in the spec (step 2). Every add-on that builds a service URL assumes it’s there.
Live cluster: a CoreDNS rewrite rule in the guest’s coredns ConfigMap, mapping the domainless name onto the real one. Ugly, effective, and documented in the cluster’s notes. The reload plugin picked it up:

--- Corefile now ---
.:53 {
    errors
    health {
       lameduck 5s
    }
    ready
    rewrite name supervisor-management-proxy.default.svc supervisor-management-proxy.default.svc.cluster.local
    kubernetes cluster.local in-addr.arpa ip6.arpa {
...

8. Leave the proxy’s values alone

The error moved on, which counts as progress around here. The name resolves now, and nothing answers:

checked at (UTC): 19:35:38
telegraf-26ppl: STILL FAILING (4 errors)
  last: 2026-08-23T19:35:07Z E! [agent] Error writing to outputs.http: Post "https://supervisor-management-proxy.default.svc.:10093/arc/tkgs/metric": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
...

In 9.1 terms, a prerequisite was missing. The Metrics Aggregator, which terminates the cluster’s TLS in that design, didn’t exist on f06 yet. I tried pointing the proxy at Ops by hand, with a value from its definition:

        metricsHTTPRemoteEndpoint:
          title: HTTP remote endpoint configuration for pushing workload metrics
          ...
          properties:
            host:
              ...
            port:
              ...
              default: 443
              ...
            tlsClientSecretName:
              type: string
              description: Name of the secret which contains the TLS configuration required to push metrics to remote endpoint.
              default: ""
            tlsClientSecretNamespace:
              ...
              default: "vmware-system-monitoring"

A PUT on the install record (PATCH returns 404) reconciled:

$vals="namespace: svc-supervisor-management-proxy-acqyd`nmetricsHTTPRemoteEndpoint:`n  host: `"f06-flt-ops01.res.lab`"`n  port: 443`n"
$b64=[Convert]::ToBase64String([Text.Encoding]::UTF8.GetBytes($vals))
$body=@{ version='0.4.1'; yaml_service_config=$b64 } | ConvertTo-Json
...
  $r=Invoke-WebRequest -Method Put -Uri "https://$vc/api/vcenter/namespace-management/clusters/domain-c9/supervisor-services/$svc" -Headers $H -ContentType 'application/json' -Body $body -SkipCertificateCheck -TimeoutSec 120

PUT -> 204
  [30s] CONFIGURED
  [60s] CONFIGURED
  [90s] CONFIGURED

Telegraf timed out as before. Broadcom’s proxy page is blunt: “No additional configuration values for the Supervisor Management Proxy service are required in this case.”

Set at install time, the value did harm. With no tlsClientSecretName, the package renders a nameless SecretImport, and kapp refuses it:

=== supervisor-management-proxy.vmware.com ===
Reason: ReconcileFailed. Message: kapp: Error: Validation errors:
- Expected 'metadata.name' on resource 'secretimport/ (secretgen.carvel.dev/v1alpha1) namespace: svc-supervisor-management-proxy-htarg' to be non-empty (stdin doc 5).

The next day, the Supervisor was rebuilt onto the lab’s planned address range. The catalog item installed the proxy and the Metrics Aggregator with no values, and both went CONFIGURED:

--- ALL LoadBalancer VIPs on the rebuilt supervisor ---
...
  svc-metrics-aggregator-5q0of             workload-metrics-loadbalancer 192.168.144.7
  svc-supervisor-management-proxy-htarg    workload-metrics-loadbalancer 192.168.144.6

demo01 was recreated on that Supervisor with serviceDomain set. This check read a Telegraf pod’s last two minutes of log. The broken state had logged an error a minute:

=== observability packages ===
  cert-manager   ReconcileSucceeded=True
  contour        Reconciling=True
  fluent-bit     ReconcileSucceeded=True
  prometheus     ReconcileFailed=True
  telegraf       ReconcileFailed=True
=== telegraf delivery (last 2 min) ===
  CLEAN - metrics flowing to mgmt proxy

“Metrics flowing” is my script’s wording for no failed writes. And the telegraf PackageInstall is the one kapp refused, so the clean Telegraf wasn’t mine.

9. VCF Operations

Broadcom’s consumption docs show the result in VCF Automation (Manage and Govern > Kubernetes Management > Clusters) and in a Workload Management Activated Cluster Summary tab in VCF Operations. The 9.0.1 write-up adds a switch on the VKS Cluster object, which the catalog item still names at the end of each run:

... | Ops manual step: set 'Pod And Container Monitoring Enabled' on VKS Cluster demo01 in VCF Operations inventory; ...

This record has neither. The switch was never set, and my one capture of a VKS Cluster object (vks-demo01, 1 September, in the two-ways post) counts 0 pods, deployments and DaemonSets. The last hop I can show is Telegraf writing without errors. Not the grand finale I’d hoped for, but an honest one.

Clean-up and repeatability

One owner per add-on. Where add-on management installs Telegraf, don’t add your own PackageInstall. To run it yourself, first set the cluster label addons.kubernetes.vmware.com/automated-monitoring to disabled. The catalog item’s answer to the ownership errors, --dangerous-override-ownership-of-existing-resources=true, only hands the objects to a second owner.
Supervisor Service API on 9.1: service carvel_spec.version_spec.content, version carvel_spec.content, install {supervisor_service, version}, values PUT {version, yaml_service_config}, removal PATCH …?action=deactivate, then DELETE. New definition bytes need a new service record.
A definition outlives the Supervisor. What you register lives on vCenter’s service record, so a rebuilt Supervisor installs the same definition again. Replacing it means deactivating and deleting that record first.
Services break later. In September, the proxy, the aggregator and four other Supervisor Services went to ERROR when the platform replaced their kapp service accounts. Recreating the old ones fixed all six within two minutes.

Why this matters outside the lab

On VCF 9.1, Kubernetes metrics in VCF Operations are meant to be a platform setting, not a job per cluster. Get three prerequisites right, and new clusters arrive monitored.

The layer underneath still matters, because every failure here had the same shape: a generic symptom (FailedMount, a name that won’t resolve, a timeout) caused by a decision made somewhere else.

For a customer, that means a Supervisor built for observability before any cluster asks, a cluster baseline with the fields add-ons assume, and one owner per add-on. That’s exactly what a standard cluster class and a Supervisor build checklist exist to encode.

Rules learned

On 9.1, start with the supported route: Ops collecting the Supervisor, the Metrics Aggregator and the add-on repository. Then check for Telegraf before installing it.
On VKS, the Telegraf package hard-depends on the Supervisor Management Proxy whenever the metric proxy flag is set. FailedMount on two metrics-proxy-* secrets is the tell.
The dependency heals retroactively: install the proxy, bump the PackageInstall annotation, wait.
Set serviceDomain at cluster create. It’s immutable, and every add-on that builds a service URL will assume it.
A name that resolves isn’t a metric delivered. Read the output log for an error-free window, and leave the proxy’s values alone.
When a package “does nothing”, describe the pod, not the package.

Broadcom documentation

Enabling Monitoring for VKS Clusters: the 9.1 route and its prerequisites.
Enabling Monitoring for vSphere Supervisor: the Supervisor collection setting on the vCenter account.
Monitoring VKS Clusters Using VCF Operations: the automated add-ons, the metrics-aggregator service, and where metrics appear.
Disable Automated Monitoring on VKS Clusters Using VCF Automation: the per-cluster label, for clusters where you run Telegraf yourself.
Using the Supervisor Management Proxy Service: what the proxy is for; no extra values for VCF Operations.

Previously: Telegraf on Windows Server 2025. More in the Observability on VCF series.

Lab environment; opinions my own.

Telegraf on Windows Server 2025: unsupported, works anyway

Wed, 16 Sep 2026 12:20:00 +0100

Two facts, both true, and a little awkward in the same room:

The VCF Operations application-monitoring agent (Telegraf, packaged by Broadcom) does not list Windows Server 2025 as a supported OS.
It installs from the VCF Operations UI, runs, and reports on Windows Server 2025, once one missing Windows component is put back.

This post is about the gap between those two. “Unsupported” is a statement about who fixes it when it breaks, not about whether it works. And there is a right way to run unsupported software in production. It starts with knowing exactly what you’re relying on.

What “unsupported” means here

The matrix is a promise. Broadcom has tested this combination and will take a support case on it. Server 2025 wasn’t in the test set at release.

Nothing in the agent is gated on the OS version. It’s Telegraf with Windows inputs (win_perf_counters, win_services, win_eventlog) and an output to Ops. The Windows APIs those inputs use haven’t changed in a decade.

So it works. You just own it.

The one prerequisite: put WMIC back

Windows Server 2025 no longer ships the WMI command-line tool, wmic. It’s been deprecated for years, and it’s now a Feature on Demand rather than part of the base install. The agent’s install bootstrap still calls it. So on a stock Server 2025 build, the install from Ops does not complete.

Add the capability first, then install:

DISM /Online /Add-Capability /CapabilityName:WMIC~~~~
wmic os get caption   # should now answer

(Settings > System > Optional features > Add > “WMIC” does the same thing through the UI.) That’s the whole workaround. Not much of a plot twist, I know.

Note it wherever you keep build standards for Server 2025. It’s a one-line prerequisite for the agent, and it will stay one until the bootstrap stops needing it.

The normal install, from the UI

With WMIC present, the install path is exactly the supported-OS one. No scripts, no manual telegraf.conf, no heroics:

Operate > Workloads > Applications > Manage Telegraf Agents, tick the VM, Agent Actions > Install, supply guest credentials, wait. On the pipeline-built W2025 server it registered as a Product Managed Agent, version 9.1.0.0.3033, with Last Operation Status reading Install Success.

The agent list with the Windows Server 2025 row expanded. Agent Running, product-managed, Install Success, both collection ticks green. Under Custom Monitoring sits a Ping Check, configured from the same screen and already receiving data, with no config file touched.

The object the agent created, as Ops sees it: green, no alerts, CPU and memory properties populated. This is the picture that matters, not the install dialog.

And the proof it’s doing work rather than just existing: a Ping Check run from the Server 2025 guest, availability flat at 100 across the morning, response time steady at a couple of milliseconds.

The “Windows OS on Windows 2025” child object, with Telegraf Availability in the metric tree and AgentManagedType reading Product Managed.

Then make it yours

Two small things turn “it happens to work” into something you can run:

Record exactly what you’re running. Agent build, Telegraf version, OS build and the WMIC prerequisite, in whatever you use for a CMDB. When the support matrix catches up, you’ll want to know whether you’re on the version they tested.

Give it a check that will go flat first. The Ping Check above is configured from the agent row in Ops (Custom Monitoring). HTTP, TCP and UDP checks live there too, and a Custom Script entry on the same screen can run anything on the box. Point one at something trivially OS-dependent, and alert on its absence. If a Windows update changes an API under the agent, that line stops before anything else does.

What to watch, because it’s unsupported

Agent upgrades from Ops. The upgrade path is tested on supported OSes. Take a snapshot before pushing an agent upgrade to the W2025 fleet, and upgrade one first. Optimism is not a rollback plan.
Windows cumulative updates. Performance counter names are stable. Provider GUIDs occasionally aren’t. Watch your canary check after Patch Tuesday.
WMIC on new builds. Any image or pipeline that produces Server 2025 needs the capability added. Without it, the next install will fail the way the first one did.
Service account and UAC. WMIC was the only install blocker we hit. The documented Windows catch is UAC. With UAC on and a non-administrator account in Administrators, the agent stays at “Not Started” and the last operation reads “Download Success”. The bootstrap then has to be run by hand.
Don’t file cases on it. Reproduce on a supported OS first. That’s the deal you made.

Why this matters outside the lab

The practical lesson for customers is about how to adopt something the vendor hasn’t blessed yet. New operating systems arrive before support matrices catch up, and “wait” is often not an option. The approach here is how an operations team gets Windows Server 2025 monitored on day one, without taking on hidden risk:

Find the real blocker (one missing Windows component, not the agent).
Use the standard install path.
Record exactly what you’re running.
Add a check that detects breakage early.
Upgrade one node first.

The same discipline applies to any unsupported-but-working combination.

Rules learned

“Unsupported” means you own the fix path. Decide that consciously, then record versions, add a canary check and upgrade one node first.
Server 2025 ships without WMIC, and the agent bootstrap needs it. Add the capability first, and the ordinary UI install works.
The Windows inputs aren’t gated on the OS version, and W2025 runs the agent fine. Alert on metric absence, not just thresholds.
Configure checks from the agent row in Ops. Leave telegraf.conf alone, so upgrades from Ops stay clean.

Broadcom documentation

Supported Platforms: the operating systems the product-managed agent supports; Windows stops at Server 2022
Windows Server 2025 support for Telegraf Agent in VCF and Aria Operations: Server 2025 not yet supported, and the install that stays “in progress”
Install an Agent from the UI: the Manage Telegraf Agents install, and what UAC changes about it
Communication with Cloud Proxy and vCenter: the vCenter guest-operation privileges the UI install needs
Activate Remote Checks: ICMP, UDP, TCP and HTTP checks under Custom Monitoring
Custom Script: scripts the agent runs on the box, each returning a single integer

Previously: fluent-bit two ways. Next in the Observability on VCF series: the same Telegraf on VKS, and the dependency that isn’t in its README.

Lab environment; opinions my own. Support status as observed at time of writing — check the current matrix.

fluent-bit two ways: VKS add-on and Windows agent, one log endpoint

Wed, 16 Sep 2026 12:00:00 +0100

Logs from a Kubernetes cluster and logs from a Windows server end up in the same place: VCF Operations for Logs. The roads there couldn’t look more different.

On VKS, fluent-bit is a package: declare a values secret, reconcile, done. On Windows it’s a service: install it, write a config by hand, restart, tail the debug log. Same binary, same output plugin, same endpoint. This post puts the two side by side, so the shared shape is easy to see.

The endpoint

VCF Operations for Logs ingests over its CFAPI on port 9543:

https://:9543/api/v2/events

Auth is enabled but optional by default. The endpoint accepts events with or without a token, unless strict authentication is turned on in Global Settings. With that on, it drops any request without a valid Bearer token. Our shippers sent no token, and the lab ran the default.

What matters to fluent-bit is the URI, the port, TLS and a JSON body shaped as {"events":[...]}. Both shippers below produce exactly that. Eventually.

Way 1: VKS package

VKS ships fluent-bit in its standard package repository. On a cluster with the repo registered:

kubectl ... get packages -n tkg-system 2>&1 | Select-String -Pattern 'NAME|telegraf|prometheus|fluent|cert-manager|contour' | Out-String
NAME                                                                  PACKAGEMETADATA NAME                             VERSION                  AGE
...
fluent-bit.kubernetes.vmware.com.4.0.2+vmware.1-vks.1                 fluent-bit.kubernetes.vmware.com                 4.0.2+vmware.1-vks.1     55s
fluent-bit.kubernetes.vmware.com.4.0.5+vmware.1-vks.1                 fluent-bit.kubernetes.vmware.com                 4.0.5+vmware.1-vks.1     55s
fluent-bit.tanzu.vmware.com.2.2.3+vmware.1-tkg.2                      fluent-bit.tanzu.vmware.com                      2.2.3+vmware.1-tkg.2     56s
fluent-bit.tanzu.vmware.com.3.1.9+vmware.1-tkg.1                      fluent-bit.tanzu.vmware.com                      3.1.9+vmware.1-tkg.1     55s
fluent-bit.tanzu.vmware.com.3.2.7+vmware.1-tkg.1                      fluent-bit.tanzu.vmware.com                      3.2.7+vmware.1-tkg.1     55s
...

A values secret configures the output. The package’s own default output is already shaped for CFAPI, so the values are short:

namespace: tanzu-system-logging
fluent_bit:
  config:
    outputs: |
      [OUTPUT]
        Name          http
        Match         *
        Host          f06-flt-log01.res.lab
        Port          9543
        URI           api/v2/events
        Format        json
        tls.debug     4
        tls           on
        tls.verify    off
        json_date_key timestamp

Then a PackageInstall that points at it:

apiVersion: packaging.carvel.dev/v1alpha1
kind: PackageInstall
metadata: { name: fluent-bit, namespace: package-installs }
spec:
  serviceAccountName: pkgi-sa
  packageRef:
    refName: fluent-bit.kubernetes.vmware.com
    versionSelection: { constraints: 4.0.5+vmware.1-vks.1 }
  values:
  - secretRef: { name: fluent-bit-values }

Reconcile succeeded. The proof is in the pod logs, not the Ops UI. Ops for Logs federates its API authentication through the identity broker, so a script has a far easier time checking from the shipper’s side:

[ info] [output:http:http.0] f06-flt-log01.res.lab:9543, HTTP status=200
[ info] [output:http:http.0] f06-flt-log01.res.lab:9543, HTTP status=200

One 200 per batch. That’s the whole VKS side, and it almost felt like cheating. It’s also why the packaged path is the one to recommend. The inputs (container logs, kubelet, systemd) come pre-wired, the parsers are right, and the only decision you make is the output.

One trap (shared with Telegraf, next post). A cluster created without clusterNetwork.serviceDomain gives in-cluster service names of the form ...svc., with no domain. Some add-ons build those into URLs that don’t resolve. The field is immutable after create, so set it explicitly on every new cluster.

The receiving end. One filter (text contains vks-demo01) and a day of the cluster’s logs, one bar per hour.

Every event carries the fields the package’s filters add: cluster, kubernetes_namespace, pod and container. That’s what lets Windows and VKS sit side by side in one explorer later.

Way 2: Windows Server 2025 service

fluent-bit ships a Windows build with a winevtlog input. On the pipeline-built W2025 server I used the portable zip rather than the MSI. No installer, just a folder under C:\ and a service registered with sc.exe.

A small script writes the config below ($LogsHost is the appliance’s FQDN, $LogsPort is 9543). Four of its lines turned out to matter:

The config the script writes: input, filter and output