The Nested Lab

The full VMware Cloud Foundation stack, engineered live. Written from a nested VCF 9.x lab that the team at Comms-care builds and looks after — a platform where every consultant gets a complete VCF instance of their own to demo, prove and break. The standing rule: every command, every error and every fix you read here was captured from a running environment.

What gets covered — because all of it runs in the lab:

Foundation — VCF 9 bringup from the Installer API, SDDC Manager, vCenter and nested ESXi, vSAN ESA, fleet components, identity.
Networking — NSX and NSX VPCs: isolation, overlapping addressing, trunk subnets, transit gateways, load balancers.
Automation — VCF Automation both ways: classic VM Apps (Aria templates, vRO, event broker) and supervisor-native All Apps (CCI blueprints, VM Service, the catalog).
Kubernetes — VKS clusters, supervisor services, Harbor, packages, air-gapped upgrades.
Operations — VCF Operations, Operations for Logs, fluent-bit, Telegraf, and metrics pipelines for things that don’t have adapters (like vLLM).
AI infrastructure — GPU Operator, vGPU licensing and LLM serving on VKS, including in dark sites.

Posts arrive in series; start with The VPC Pod Papers or browse by series.

The support matrix says no; the agent status says Install Success

Telegraf on Windows Server 2025: unsupported, works anyway

The VCF Operations agent support matrix doesn’t list Windows Server 2025. Add one missing Windows component (WMIC) and the ordinary UI-driven install works: agent running, checks green, metrics flowing. What ‘unsupported’ really means, the one prerequisite, and what to watch because of it.

Two fluent-bit shippers — a VKS package and a Windows service — converging on one Ops for Logs endpoint

fluent-bit two ways: VKS add-on and Windows agent, one log endpoint

The same log pipeline for two very different worlds: fluent-bit as a VKS package (values secret, CFAPI output, verified 200s per batch) and fluent-bit as a Windows service on Server 2025 shipping the event log to the same VCF Operations for Logs endpoint. One backend, two configs, and what each side taught me.

Seven demo apps behind seven VIPs on one VCFA-deployed VKS cluster

Seven demo apps, one request: deploying a showcase stack via VCF Automation

KubeDoom, KubeInvaders, kube-ops-view, Pac-Man with persistent MongoDB, podinfo, Goldpinger and a Prometheus stack — deployed onto a VCFA-provisioned VKS cluster in a tenant VPC, each behind its own NSX VIP. The proper tenanted path (not the supervisor shortcut), what the platform does for free, and the traps.

Pac-Man inside a VPC boundary; a LoadBalancer VIP is the one door out

What's a VPC? Let Pac-Man explain

Part 0 of the Pod Papers: an NSX VPC explained with a running game. Private by default, one deliberate door out — and a self-inflicted outage that taught me five green layers can hide one wrong integer.

The trunk-subnet design: one trunk vNIC, binding maps demux VLANs 1610/1611/1612 into VPC subnets

Nested ESXi inside an NSX VPC: the trunk-subnet design

Plain VPC subnets silently blackhole a nested ESXi host. Here’s why — and the trunk subnet + binding map design that makes nested labs work as an ordinary NSX VPC tenant, verified end to end.

The ordering rule: VPC, VPCAttachment, LoadBalancer, then the namespace - swap steps 3 and 4 and VIPs pend forever

The load balancer that must exist before the namespace

VIPs pending forever, a retryable error that never stops retrying, and an ordering rule the docs don’t tell you: in a self-service NSX VPC, the LBService must exist before the namespace that will use it.

Three VPC pods with byte-identical addressing and no route between them

Three datacenters, one IP plan: identical isolated pods with NSX VPCs

Three nested-ESXi pods, byte-identical addressing — same subnets, same VLANs, same host IPs, even the same MACs — with zero reachability between them. How overlapping VPC CIDRs and deterministic subnet realization turn cookie-cutter environments into a first-class feature.

Hub-and-spoke: three private pods reach one shared-services VPC over the transit gateway; the service cannot reach back

Shared services for isolated tenants: PrivateTGW subnets

Three pods with identical private addressing all need the same WSUS, repo and AD. One shared-services VPC with a PrivateTGW subnet serves all of them over the transit gateway — and can’t reach back into any of them. The directional test, and why SNAT is what makes it work.

One blueprint: namespace, trunk topology, two nested hosts, two VIPs — requested as a catalog item

A datacenter in a catalog tile: nested ESXi pods via VCF Automation All Apps

The whole isolated pod — namespace, trunk subnets, binding maps, two dual-NIC nested ESXi hosts with an ISO attached, SSH/HTTPS VIPs — as one VCF Automation blueprint, published to the catalog. Anatomy of the blueprint, the ordering it enforces, and the three things it can’t express.

Dual-NIC nested hosts: what redundancy means when the fabric is virtual

VCF wants two pNICs per host. In a nested lab the second vNIC adds no physical redundancy — so why add it? Because bringup validation and uplink teaming expect it, and because the failover test tells you something real about the trunk. vmnic0 down, 0% loss, and the SSH session watching it never dropped.