The Nested Lab

The full VMware Cloud Foundation stack, engineered live. Written from a nested VCF 9.x lab that the team at Comms-care builds and looks after — a platform where every consultant gets a complete VCF instance of their own to demo, prove and break. The standing rule: every command, every error and every fix you read here was captured from a running environment.

What gets covered — because all of it runs in the lab:

Foundation — VCF 9 bringup from the Installer API, SDDC Manager, vCenter and nested ESXi, vSAN ESA, fleet components, identity.
Networking — NSX and NSX VPCs: isolation, overlapping addressing, trunk subnets, transit gateways, load balancers.
Automation — VCF Automation both ways: classic VM Apps (Aria templates, vRO, event broker) and supervisor-native All Apps (CCI blueprints, VM Service, the catalog).
Kubernetes — VKS clusters, supervisor services, Harbor, packages, air-gapped upgrades.
Operations — VCF Operations, Operations for Logs, fluent-bit, Telegraf, and metrics pipelines for things that don’t have adapters (like vLLM).
AI infrastructure — GPU Operator, vGPU licensing and LLM serving on VKS, including in dark sites.

New here? Start here: reading paths by what you are trying to do. Or browse by product, series or month below.

Browse the lab notes

By VCF product

By series

By month

October 20269

September 202623

Full archive · Search

site-f06.yaml feeds gen_lab_blueprint.py, which writes eight blueprints and two property groups; each request builds a lab in its own VPC with a jump host, dc01, four nested hosts, vCenter, VCF Operations and AD, routed by the VPC gateway with no router VM

Nested labs in VCF Automation 9.1, built from code instead of captured

A VCF Automation 9.1 catalog that builds a complete nested VCF lab per request, each in its own VPC. How it differs from capturing a hand-built lab, and what building from code buys you.

Everything that has to cross the airlock: images, signatures, node OVAs, kernel headers, licences

VKS and the GPU Operator in a dark site: the whole airlock

Upgrading VKS, Kubernetes and the NVIDIA GPU Operator with no internet means five supply chains arriving intact. Each has a trap that passes every check until activation. This is the map.

Org Builder's Home page, four ways in, with the Install or upgrade tile highlighted and its note: Install, upgrade or uninstall an organization from packages

Install, upgrade, uninstall: a whole lab catalog with Org Builder

The nested-lab catalog as three Org Builder packages that install, upgrade and uninstall a VCF Automation organization. How made and found keep an uninstall honest, and what two live runs proved.

Two tarballs that look the same: manifest.json vs oci-layout

imgpkg tar vs OCI archive: why your air-gap tarballs won't load

Two tarballs full of container images, both bound for the same Harbor. imgpkg loads one and rejects the other. The layout difference nobody explains, what survives a copy, and three ways across.

Telegraf runs the script every 60 s; the script's stdout is one integer; the integer is the health of the integration

vLLM metrics into VCF Operations, part 2: monitor the monitor

Part 1’s design only helps if it runs every minute, for ever, and says when it stops. Self-monitoring through Telegraf, batched pushes, the four alerts for day one, and an offline test mode.

Org Builder's Home page, four ways in, with the Create a package tile highlighted and its note: Create a package from an organization

Bring your own image: a step done by hand in Org Builder

Org Builder’s second worked example, the bring-your-own-image case: a team catalog that boots from an image, packaged without the image file, edited to ask for each site’s own copy at the right moment, and installed in production. Plus what VCF Automation’s own OVA export does with the same blueprint.

Org Builder's Home page, four ways in, with the Move an organization tile highlighted and its note: Package one organization and install it into another

Same blueprints, different values: dev to prod with Org Builder

Org Builder’s first worked example: two blueprints and the property groups they read, exported from a development organization and imported into production with production’s own values.

Org Builder's Home page: four ways in, each a tile of its own - Make a site file, Create a package, Install or upgrade, and Move an organization - with links below for the offline site wizard, importing a bundle and editing a package, beside the Install and Package and move steps in the side menu

Introducing Org Builder

Why we built Org Builder, our tool for VCF Automation organizations: catalog services built in one organization have to reach others, and rebuilding them by hand or scripting every API call doesn’t scale. Packages, site files and versioned releases, in brief.

The blueprint designer's palette, Supervisor Namespace, VPC and Workload groups, unfolding into five YAML types and the Kubernetes kinds behind each item

Every resource in the VCF Automation 9.1 blueprint designer: the complete guide

A complete guide to the VCF Automation 9.1 blueprint designer: every palette item, the YAML behind it, every field the platform accepts, recipes and traps. Every snippet validated, dry-run or deployed.

Prometheus histogram buckets in; flat P50/P95/P99 gauges out

vLLM metrics into VCF Operations, part 1: don't ship the histogram

vLLM exposes about 200 Prometheus series per model, and pushing them raw into VCF Operations is a cardinality bomb. A pipeline that computes the useful numbers client-side and drops what can’t be graphed.