NSX on The Nested Lab

Every resource in the VCF Automation 9.1 blueprint designer: the complete guide

Thu, 01 Oct 2026 00:00:00 +0100

The blueprint designer in a VCF Automation 9.1 All Apps organization offers sixteen items in three groups. Drag one onto the canvas and you get a few lines of YAML. What you may write underneath them is spread across the VM Service, VKS, NSX VPC and Automation guides. For some items, it’s written down nowhere at all.

This guide puts it in one place. For each item, you get:

what it creates, and the YAML behind it;
every field the platform accepts;
a minimal snippet, and recipes for the common jobs;
the status fields worth reading;
the traps we hit.

Three things make it more than a list from memory, which, given my memory, is just as well:

The field lists come from the platform. VCF Automation publishes the schema of every blueprint resource type through its API. The designer carries the schema of every palette item. The Supervisor publishes the definition of every Kubernetes kind it serves, and VCF Automation’s VPC API publishes its own. Where they disagree (and they do), the tables follow what the platform enforces.
Every snippet was checked. Each one went through VCF Automation’s validation API, and every Kubernetes manifest through a server-side dry run on the Supervisor. Five test blueprints (all of them in the downloads) deployed every item for real. The remaining recipes are ones our lab catalog deploys every day.
The traps are real. Several of the most useful lines in this guide come from things that went wrong while testing: a NAT rule that ignored its port, a firewall rule that grew an “Any”, a request that waits for an address that can never come.

We checked it on our lab platform, f06:

VCF Automation 9.1.0;
a Supervisor on Kubernetes 1.32.9, with the VM Operator API at v1alpha5;
VKS with ClusterClasses up to builtin-generic-v3.6.0, and Kubernetes releases up to 1.35.5;
NSX VPCs.

Names in the examples (f06, nested-pod, vpc-student05, vsan-default-storage-policy) are ours; use yours.

In the field tables, Values gives a field’s choices and default. Where the schema has neither, it gives an example, marked e.g.: the value from our tested blueprints wherever one of them set the field, otherwise a typical one. For an object or a list, the example is a short YAML flow value built from its fields (... marks the ones left out). And <...> stands for a name of yours.

The shape of a blueprint

An All Apps blueprint is YAML with formatVersion: 2, its inputs, its resources and its outputs:

formatVersion: 2
inputs:
  vmName:
    type: string
    title: VM name
    default: web-01
    pattern: '^[a-z0-9]([-a-z0-9]*[a-z0-9])?$'
  size:
    type: string
    title: Size
    default: small
    oneOf:
      - {title: Small, const: small}
      - {title: Medium, const: medium}
resources:
  namespace:
    type: CCI.Supervisor.Namespace
    properties:
      name: team-a-dev
      existing: true
  vm1:
    type: CCI.Supervisor.Resource
    properties:
      context: ${resource.namespace.id}
      manifest:
        apiVersion: vmoperator.vmware.com/v1alpha5
        kind: VirtualMachine
        metadata:
          name: ${input.vmName}
        spec:
          className: ${'best-effort-' + input.size}
          imageName: ubuntu-24.04-server-cloudimg-amd64
          storageClass: vsan-default-storage-policy
outputs:
  vmName:
    value: ${resource.vm1.object.metadata.name}

What the expressions can read:

Expression	Value
`${input.}`	A request input. An input group from a property group is `${input..}`.
`${resource..}`	Another resource’s property; this also orders the two. `object` holds the live Kubernetes object of a Supervisor Resource.
`${propgroup..}`	A constant property group’s value.
`${secret.}`	A VCF Automation secret, from the organization or the project.
`${env.deploymentName}`, `${count.index}`	The deployment’s name; the instance number in a counted resource.
`${to_k8s_name(env.deploymentName, 63)}`	A string made safe for a Kubernetes name, as Broadcom’s own samples use for `generateName`.

Besides type and properties, each resource has dependsOn (an explicit order) and allocatePerInstance (see count below). formatVersion: 2 also allows metadata, variables, and an output named __deploymentOverview, whose Markdown value becomes the deployment’s overview page.

How the palette maps to YAML

Behind the sixteen items there are only five resource types, and one of them isn’t in the palette. Most items are a type with part of its YAML already filled in. For the workload items, that’s a Kubernetes apiVersion and kind; for the VPC items, it’s a VPC configuration kind.

Palette item	YAML `type`	Pre-filled
Supervisor Namespace	`CCI.Supervisor.Namespace`
VPC group
VPC	`CCI.VPC`
VPC Configuration	`CCI.VPC.Configuration`	nothing: you set `kind`
Attachment	`CCI.VPC.Configuration`	`kind: VPCAttachment`
IP Address Allocation	`CCI.VPC.Configuration`	`kind: VPCIPAddressAllocation`
NAT Rule	`CCI.VPC.Configuration`	`kind: VPCNATRule`
Group	`CCI.VPC.Configuration`	`kind: VPCNetworkSecurityGroup`
Gateway Firewall Policy	`CCI.VPC.Configuration`	`kind: VPCGatewayFirewallPolicy`
Workload group
Supervisor Resource	`CCI.Supervisor.Resource`	nothing: any manifest
Virtual Machine	`CCI.Supervisor.Resource`	`vmoperator.vmware.com/v1alpha5` `VirtualMachine`
Virtual Machine Group	`CCI.Supervisor.Resource`	`vmoperator.vmware.com/v1alpha5` `VirtualMachineGroup`
Virtual Machine Service	`CCI.Supervisor.Resource`	`vmoperator.vmware.com/v1alpha3` `VirtualMachineService`
Subnet	`CCI.Supervisor.Resource`	`crd.nsx.vmware.com/v1alpha1` `Subnet`
Persistent Volume Claim	`CCI.Supervisor.Resource`	`v1` `PersistentVolumeClaim`
Secret	`CCI.Supervisor.Resource`	`v1` `Secret`
Kubernetes Cluster	`CCI.Supervisor.Resource`	`cluster.x-k8s.io/v1beta1` `Cluster`
(not in the palette)	`Util.PasswordEntry`

Four consequences, worth knowing before any of the detail:

Anything the Supervisor understands can go in a blueprint. The workload items are shortcuts. The generic Supervisor Resource takes any manifest the namespace accepts. Our lab blueprints rely on a kind the palette doesn’t offer, SubnetConnectionBindingMap, to carry VLANs.
The VPC items are a closed list. CCI.VPC.Configuration takes exactly the five kinds above. A VPC’s load balancer is a sixth kind in VCF Automation’s VPC API, and a blueprint can’t make one (see VPC).
VCF Automation validates the outside, the platform the inside. The validation API checks the resource type’s own properties: required fields, patterns, the namespace’s two shapes. It doesn’t look inside a manifest or a configs[].spec: in our test, powerState: Sideways passed validation. Those are checked when the request runs.
The designer’s schemas are not the platform’s. The palette’s forms come from schemas bundled with VCF Automation, while the Supervisor and the VPC API check against their own. They disagree in a handful of places, listed next.

Where the designer and the platform disagree

We compared each palette item’s schema with the platform’s definition of the same apiVersion and kind. We went field by field (every bit as gripping as it sounds) and tested every difference:

Item	The designer offers	What the platform does
Virtual Machine	`affinity.zoneAffinity`, `affinity.zoneAntiAffinity`, and VM affinity terms named `...IgnoredDuringExecution`	Rejects them as unknown fields. VM affinity and anti-affinity take `requiredDuringSchedulingPreferredDuringExecution` and `preferredDuringSchedulingPreferredDuringExecution`.
Virtual Machine	no `linuxPrep.password`, `linuxPrep.scriptText`, `crypto.vTPMMode`, `currentSnapshotName`, or disk options at volume level	Accepts all of them.
Virtual Machine	`bootOptions.firmware` as `BIOS` or `EFI`, and `bootOptions.enterBootSetup`	`spec.bootOptions.firmware: Unsupported value: "BIOS": supported values: "bios", "efi"`, and `strict decoding error: unknown field "spec.bootOptions.enterBootSetup"`.
Virtual Machine `v1alpha4`, `v1alpha3`	`network.nameServers`	The field is `nameservers`, lower case.
Subnet	`regionName`, `ipBlockNames`, `description`	Rejects them. Accepts `vlanConnectionName`, which the designer doesn’t show.
Persistent Volume Claim	`accessMode`	`strict decoding error: unknown field "spec.accessMode"`. The field is `accessModes`.
Kubernetes Cluster	`cluster.x-k8s.io/v1beta1`	Serves it, with `cluster.x-k8s.io/v1beta1 Cluster is deprecated; use cluster.x-k8s.io/v1beta2 Cluster`.
Group	`vmSelectors[].selector`, `podSelectors[].selector`	`spec.vmSelectors[0]: Required value: must specify at least one selector`. The field is `labelSelector`; the API also offers `propertySelector`.
Gateway Firewall Policy	`ruleCount`	Computed by the API; not accepted. A rule’s `from` is required, though the schema says empty means any.

None of this stops the designer from saving a blueprint. It surfaces when the request runs.

What every resource shares

Properties on every type

Property	On	What it does
`count`	all	How many instances to create. Default 1.
`allocatePerInstance`	all (beside `type`)	Required for `${count.index}`: without it the validator refuses the blueprint with `should have allocatePerInstance property set to True`.
`dependsOn`	all (beside `type`)	Explicit order. A reference to another resource orders the two already.
`context`	Supervisor Resource items	Required. The namespace the manifest goes into: `${resource..id}`.
`existing`	Namespace, Supervisor Resource items	`true` adopts what already exists instead of creating it.
`wait`	Supervisor Resource items, VPC Configuration	When the resource counts as done.

Recipe, two Secrets from one resource:

  sec:
    type: CCI.Supervisor.Resource
    allocatePerInstance: true
    properties:
      count: 2
      context: ${resource.ns.id}
      manifest:
        apiVersion: v1
        kind: Secret
        metadata:
          name: ${'ref-s-' + count.index}
        type: Opaque
        stringData:
          note: created by instance ${count.index}

The deployment then holds sec[0] and sec[1], and the namespace ref-s-0 and ref-s-1.

`wait`: when a resource is finished

Without a wait, a Supervisor Resource is finished as soon as the Supervisor accepts the manifest. That’s fine for a Secret, and wrong for a VM whose address an output needs. wait takes conditions, fields, or both:

      wait:
        conditions:                      # status.conditions[] of the object
          - type: Ready
            status: "True"
          - type: Ready
            status: "False"
            reason: Failed
            indicatesFailure: true       # this one fails the resource instead of finishing it
        fields:                          # any field of the object, by path
          - path: status.powerState
            value: PoweredOn
        skipWaitOnDelete: false          # true: deletion does not wait for the object to be gone

A Supervisor Resource can also watch log output with executionLogs: progressMessagePattern while a matching message appears, and failureMessagePattern to fail the resource.

The designer pre-fills a wait for some items:

Item	Designer’s default `wait`
Virtual Machine	condition `VirtualMachineCreated` = `True`
Virtual Machine Group	condition `Ready` = `True`
NAT Rule, Group	condition `Realized` = `True`
everything else	none

The VM’s default came from a 9.0 problem: VM status could come back empty, and Broadcom’s KB 435137 gave this wait as the workaround. That condition is true before the VM is even on, let alone has an address.

VCF Automation also waits by itself in one place: a Virtual Machine Service of type LoadBalancer is not finished until it has an external address, with or without a wait. In a VPC without a load balancer, that address never comes, and the request stays in progress until it times out. Ours was still PARTIAL after ten minutes, with the service on the Supervisor.

Reading a resource’s live state

Every Supervisor Resource exposes the object as the Supervisor sees it under object, and a VPC Configuration exposes its objects under configs:

outputs:
  vmIp:
    value: ${resource.vm1.object.status.network.primaryIP4}
  vmHost:
    value: ${resource.vm1.object.status.nodeName}
  lbIp:
    value: ${resource.webLb.object.status.loadBalancer.ingress[0].ip}
  publicIp:
    value: ${resource.publicIp.configs[0].spec.allocationIPs}

Outputs are computed once, when the request finishes, and not refreshed afterwards. A VM that waited only for PoweredOn finished before its guest reported an address, and its IP output stayed empty for good. Waiting on the condition that marks the guest’s network as configured fixes it. This one gave the output 172.30.0.2:

      wait:
        conditions:
          - type: VirtualMachineGuestNetworkConfigSynced
            status: "True"

Checking a manifest before you deploy it

The fastest check we found is a server-side dry run with strict field validation, against the namespace the blueprint will use:

kubectl apply --dry-run=server --validate=strict -n  -f vm.yaml

It runs the Supervisor’s schema checks and admission webhooks, flags unknown fields, and creates nothing. It works for every workload kind. It does not work for the VPC kinds: VCF Automation’s VPC API ignores dryRun and creates the object, as one of our probes found.

Supervisor Namespace

type: CCI.Supervisor.Namespace. Creates a vSphere Namespace through VCF Automation, inside the project’s allocation, or adopts one that exists. Everything in the Workload group needs one: their context points at it.

The schema has two shapes, and the validator enforces them: an existing namespace takes name and existing: true and nothing else; a new one needs generateName, className, regionName and vpcName.

Property	Notes
`generateName`	New namespace. Prefix; VCF Automation adds `-` and five random characters (`ns-ref-bstk7`). Must match `^[a-z0-9]([-a-z0-9]*[a-z0-9])?$`, so it cannot end in a hyphen.
`className`	New namespace. The namespace class: limits, VM classes, storage classes and content libraries.
`regionName`	New namespace. The region.
`vpcName`	New namespace. The VPC its workloads use: an existing VPC’s name, or `${resource..name}`.
`name` + `existing: true`	Existing namespace. `name` alone matches neither shape.
`zones[]`	Per vSphere Zone: `name`, `cpuLimit`, `cpuReservation`, `memoryLimit`, `memoryReservation`, all five required.
`storageClasses[]`	`name` (the storage policy as the namespace names it) and `limit`.
`vmClasses[]`, `contentSources[]`	VM classes and image sources beyond the class’s own.
`sharedSubnetNames[]`, `infraPolicyNames[]`, `segName`, `description`	Shared subnets, extra infrastructure policies, an Avi Service Engine Group, a description.

Every field the platform accepts (25)

What's a VPC? Let Pac-Man explain

Wed, 16 Sep 2026 08:30:00 +0100

Before this series gets into trunk subnets and binding maps, it’s worth ten minutes on the thing everything else stands on: what an NSX VPC actually is, seen from the tenant’s chair. No slides, just a game of Pac-Man. Strictly for educational purposes, you understand.

The lab has Pac-Man running twice on VCF 9.1. One copy runs on a VKS (vSphere Kubernetes Service) cluster I built by hand with kubectl. The other runs on a cluster deployed through VCF Automation’s catalog. Both live inside VPCs, and both were reachable when I started:

http://192.168.144.15/  ->  Pacman in HTML 5 Canvas
</span></span><span class="line"><span class="cl">http://192.168.144.23/  ->  <title>Pacman in HTML 5 Canvas
</span></span></code></pre></div><figure class="nl-video">
  
    
  </video>
  <figcaption>Pac-Man, live at <code>192.168.144.23</code> — a VIP on the VPC load balancer. The only door in.</figcaption>
</figure>

<h2 id="a-vpc-is-a-private-universe-with-a-door-policy">A VPC is a private universe with a door policy</h2>
<p>Think of an NSX VPC as a tenant’s own routed network space. It has its own
subnets, its own gateway and its own address plan. The tenant carves it out,
rather than filing a ticket with the network team. Three rules define it:</p>
<ol>
<li><strong>Private by default.</strong> A <code>Private</code> subnet is reachable only from inside
the same VPC. Nobody outside can route to it: not other tenants, not
other VPCs in the same org, not the corporate network.</li>
<li><strong>Your addresses are your business.</strong> Private subnets aren’t advertised
anywhere, so two VPCs can use <em>identical</em> CIDRs. (This is the superpower
the rest of the series is built on.)</li>
<li><strong>Every door out is deliberate.</strong> Traffic leaves through the transit
gateway (with source NAT), or arrives through a <strong>LoadBalancer VIP</strong> from
an external block the provider allocated. Nothing is exposed by accident.</li>
</ol>
<p>Pac-Man’s pods sit on a private subnet. The only reason <code>192.168.144.23</code>
answers is a Kubernetes <code>Service</code> of type <code>LoadBalancer</code>, which NSX turns
into a VIP on the VPC’s load balancer. So let’s remove the door.</p>
<h2 id="before-close-the-door">Before: close the door</h2>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-fallback" data-lang="fallback"><span class="line"><span class="cl">$ kubectl patch svc pacman -n pacman -p '{"spec":{"type":"ClusterIP"}}'
</span></span><span class="line"><span class="cl">service/pacman patched
</span></span><span class="line"><span class="cl">NAME     TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
</span></span><span class="line"><span class="cl">pacman   ClusterIP   10.106.219.213   <none>        80/TCP    4d13h
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">$ curl -m 5 http://192.168.144.23/
</span></span><span class="line"><span class="cl">curl: timed out / unreachable
</span></span></code></pre></div><p>The pods are running. The service exists. The game is fine, <em>for anything
inside the VPC</em>. From my desk, it has simply gone, ghosts and all.</p>
<p>That’s the whole VPC model in one <code>curl</code>. The boundary isn’t a firewall rule
somebody wrote; it’s the absence of a route.</p>
<h2 id="after-open-it-again">After: open it again</h2>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-fallback" data-lang="fallback"><span class="line"><span class="cl">$ kubectl patch svc pacman -n pacman -p '{"spec":{"type":"LoadBalancer"}}'
</span></span><span class="line"><span class="cl">service/pacman patched
</span></span><span class="line"><span class="cl">NAME     TYPE           CLUSTER-IP       EXTERNAL-IP      PORT(S)        AGE
</span></span><span class="line"><span class="cl">pacman   LoadBalancer   10.106.219.213   192.168.144.23   80:31467/TCP   4d13h
</span></span></code></pre></div><p>Same VIP, handed straight back. NSX programmed a virtual server and pool on
the VPC’s load balancer, and the supervisor stitched them to the cluster’s
NodePort. Door open. Lesson over, or so I thought.</p>
<p>And then the game <em>didn’t load</em>.</p>
<h2 id="the-outage-i-gave-myself-this-is-the-useful-bit">The outage I gave myself (this is the useful bit)</h2>
<p>Everything was green:</p>
<ul>
<li><code>kubectl get svc</code>: LoadBalancer, VIP assigned</li>
<li><code>kubectl get endpoints</code>: pod IPs present</li>
<li>NSX: virtual server up, pool members healthy</li>
<li><code>iptables</code> on the node: NodePort rules identical to a working
neighbour service</li>
</ul>
<p>Five layers, all green, and <code>curl</code> hung. The control experiment was the
Pac-Man at <code>.15</code> on the hand-built cluster: untouched throughout, and still
playing.</p>
<p>The culprit, it turns out, was me. My “harmless” <code>ClusterIP</code> patch earlier
had included a <code>ports</code> list. A merge patch with <code>kubectl patch</code> <strong>replaces
arrays</strong> rather than merging them, and my array said <code>targetPort: 80</code>.
Pac-Man listens on <strong>8080</strong>.</p>
<p>So every layer above was faithfully forwarding traffic to a port nothing was
listening on. And every layer reported success, because <em>its</em> job was done.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-fallback" data-lang="fallback"><span class="line"><span class="cl">$ kubectl patch svc pacman -n pacman --type=json \
</span></span><span class="line"><span class="cl">    -p '[{"op":"replace","path":"/spec/ports/0/targetPort","value":8080}]'
</span></span><span class="line"><span class="cl">service/pacman patched
</span></span><span class="line"><span class="cl">$ curl -s http://192.168.144.23/ | grep -o '<title>.*'
Pacman in HTML 5 Canvas

Instant recovery. The diagnosis walked the whole paravirtual chain: the VIP, the supervisor’s VirtualMachineService, the NSX virtual server and pool, the NodePort iptables rules, and finally the pod. It’s exactly the walk you’ll need one day:

Layer	Check	What “green” hides
VIP	`kubectl get svc` EXTERNAL-IP	nothing about the backend
NSX LB	virtual server + pool status	pool health is TCP to the NodePort, not the pod
Endpoints	`kubectl get endpoints`	it lists pod IP:targetPort — read the number
Node	`iptables -t nat -L KUBE-SERVICES`	rules can be perfect and point at the wrong port
Pod	`kubectl exec ... ss -ltn`	the only place the truth lives

Bonus find on the way: kube-proxy and Antrea on that cluster had dropped their API watches days earlier (http2: client connection lost). Neither re-established its informers until it was restarted. It didn’t cause this outage, but it’s the kind of thing you only find when you’re forced to look.

Why this matters outside the lab

If you run a platform for more than one team, this is the feature you’ve been asking the network team for. A VPC gives each team, project or customer its own private network space. They create it themselves, in minutes, and nothing in it is reachable from outside until they publish it.

Security teams like it for the same reason developers do. Exposure is a deliberate, auditable act, not a side effect of plugging something in.

What organisations do with it once they have it:

Per-team sandboxes that can’t see each other, provisioned without a ticket.
Partner or supplier environments, isolated from the corporate estate but hosted on the same platform.
Multi-tenant hosting, for service providers and internal IT alike, with isolation enforced by topology rather than a growing pile of firewall rules.

Rules learned

A VPC’s boundary is the absence of a route, not a rule. Private subnets are unreachable from outside by construction, which is also why identical CIDRs across VPCs just work.
A LoadBalancer service is the deliberate door: an NSX VIP from the external block, programmed per service. Flip the type and the door closes, with nothing else to clean up.
kubectl patch (merge) replaces spec.ports; it doesn’t merge it. Patch a single field with --type=json, or leave the array out.
Five green layers can hide one wrong integer. Keep a working control (here, the untouched .15 instance) and compare layer by layer.
Read kubectl get endpoints as IP:targetPort. The port is the part people skim.

Broadcom documentation

Virtual Private Clouds Overview: the subnet access modes, and the NAT a private subnet needs to reach outside.
Create a VPC: private CIDRs, local to each VPC and allowed to overlap between VPCs.
Transit Gateways: how VPCs reach each other and the outside network.
Deploying Supervisor with VCF Networking with VPC: the VPC, load balancer and SNAT IP behind a namespace, and the LoadBalancer services NCP provides.
Create vSphere Namespaces on VPCs without SNAT and Load Balancer: without the VPC load balancer, LoadBalancer services cannot be deployed at all.

Next in the Pod Papers: nested ESXi inside a VPC, where “private by default” meets a host that fakes its own MAC address.

Lab environment; opinions my own. Output captured live, trimmed for length, never edited for outcome — including the outage.

Nested ESXi inside an NSX VPC: the trunk-subnet design

Wed, 16 Sep 2026 08:20:00 +0100

The host booted clean. Management IP configured, services up, DCUI happy. And every single packet it sent, ARP included, died silently. The host was having a lovely time; it just couldn’t tell anyone.

That’s how my first attempt at running nested ESXi inside an NSX VPC ended. The failure is nasty precisely because nothing looks wrong. If you’re building nested vSphere labs on VCF 9 with VPC networking, this post is the map of the minefield. It’s also the design that gets you across it, verified live.

The setup

VCF 9.1, with the vSphere Supervisor on NSX VPC networking. The goal was to deploy nested ESXi hosts as ordinary VM Service VMs inside a tenant’s VPC. No physical fabric changes, no provider tickets, no special treatment. It’s the kind of thing you want for training pods, cert-study labs, or reproducing customer issues.

Nested ESXi needs what physical ESXi needs: a management network, vMotion and vSAN. Traditionally, those are VLANs trunked to every host. But a VPC is an overlay world, and there are no VLANs to trunk. So what happens if you just attach the nested host’s vNIC to a normal VPC subnet?

Failure #1: the silent blackhole

Here’s the trap. A standard VPC subnet port gets address bindings. NSX pins the exact IP and MAC it allocated to that vNIC, and SpoofGuard drops everything else.

ESXi’s vmk0 doesn’t use the vNIC’s MAC. It makes up its own:

vmk0
   MAC Address: 00:50:ac:1e:00:8c     <- NOT the vNIC MAC (04:50:56:...)

So every frame the management interface sends carries a MAC the port doesn’t own. NSX drops all of it (ARP, ping, everything), while the host itself boots green and reports healthy. There’s no error anywhere, which is somehow worse than a bad one. You just can’t reach it, ever.

(There’s a second trap stacked on top. Our VPC subnets run with DHCP deactivated, so the appliance also sits at “waiting for DHCP” unless you inject static addressing through OVF guestinfo.* properties. NSX can give a VPC subnet a DHCP server or relay (Add a Subnet to a VPC); ours had neither. More on that below.)

The design that works: a trunk subnet + binding maps

The fix isn’t a hack. It’s a first-class NSX VPC construct that’s barely documented in the wild: SubnetConnectionBindingMap.

The idea:

Create one ordinary VPC subnet to act as a trunk (sn-trunk). The nested host’s vNICs attach only here.
Create a normal VPC subnet for each traditional network: sn-mgmt, sn-vmotion and sn-vsan.
Bind each of those to the trunk with a binding map carrying a VLAN tag. The nested host’s vSwitch tags frames exactly as it would on metal. The binding map strips the tag and delivers the frame into the right subnet.

It’s pure L2 demultiplexing. One vNIC carries N VLANs, and the VPC never routes on a tag. The physical fabric never sees any of it, because the 802.1Q header rides inside the Geneve overlay.

A tenant can create all of it through the supervisor, as Kubernetes objects:

# sn-trunk and sn-mgmt are ordinary Private Subnets; the interesting object:
apiVersion: crd.nsx.vmware.com/v1alpha1
kind: SubnetConnectionBindingMap
metadata: {name: bm-mgmt}
spec:
  subnetName: sn-mgmt          # the map is a child of the VLAN subnet...
  targetSubnetName: sn-trunk   # ...and points AT the trunk
  vlanTrafficTag: 1610

That direction is easy to get backwards, so it’s worth saying twice: the binding map belongs to the VLAN subnet and points at the trunk, not the other way round.

On the nested host, nothing exotic: plain virtual switch tagging (VST), just like physical. It never suspects a thing.

Name                Virtual Switch  Active Clients  VLAN ID
------------------  --------------  --------------  -------
Management Network  vSwitch0                     1     1610
vMotion             vSwitch0                     1     1611
vSAN                vSwitch0                     1     1612

The same three VLANs as the nested host sees them.

And because there’s no DHCP in our VPC subnets, the nested-ESXi appliance gets its identity through OVF properties in the VM Service spec:

bootstrap:
  vAppConfig:
    properties:
      - {key: guestinfo.ipaddress, value: {value: "172.30.0.40"}}
      - {key: guestinfo.netmask,   value: {value: "255.255.255.224"}}
      - {key: guestinfo.gateway,   value: {value: "172.30.0.33"}}
      - {key: guestinfo.vlan,     value: {value: "1610"}}

Does it actually work? The receipts

Two nested hosts, vNICs on sn-trunk, three VLANs. From host one:

[root@esx01:~] vmkping -c2 172.30.0.41            # mgmt, VLAN 1610
3 packets transmitted, 3 packets received, 0% packet loss
[root@esx01:~] vmkping -I vmk1 -c3 172.30.0.71    # vMotion, VLAN 1611
3 packets transmitted, 3 packets received, 0% packet loss
[root@esx01:~] vmkping -I vmk2 -c3 172.30.0.101   # vSAN, VLAN 1612
3 packets transmitted, 3 packets received, 0% packet loss

The transcript that matters: fail the first NIC, and every VLAN keeps flowing on the second — captured live.

Two more results are worth knowing before you design around this.

Untagged frames are dropped. I put a probe vmk on the untagged port group, using the address NSX itself had allocated to the trunk port. The result was 100% loss and an empty ARP table, while tagged traffic flowed happily beside it, as if to make a point. Every network your nested host uses needs a VLAN and a binding map. There is no untagged fallback.

Failover behaves like real hardware. With two vNICs on the trunk, teamed active/active, esxcli network nic down -n vmnic0 moved every VLAN onto vmnic1 with zero loss. The SSH session I was watching from never dropped. A vmk MAC moving between trunk ports mid-flow is exactly what MAC-pinned standard ports would blackhole. The trunk carries it fine.

Why this matters outside the lab

Running whole vSphere environments inside a VPC turns the platform into something most customers never had. It’s a way to stand up complete, isolated copies of infrastructure on demand, with no physical fabric change and no waiting for anyone. That’s what makes it commercially interesting:

Training and certification labs where every learner gets a real vSphere environment, not a shared one.
Reproducing a customer problem on a like-for-like copy, rather than on their own estate.
Rehearsing upgrades and migrations end to end before the change window, then throwing the copy away.
Vendor and feature evaluations with real behaviour, at zero risk to production.

This is the design Comms-care uses to give every consultant a dedicated environment. The same pattern scales to a classroom or a proof-of-concept factory.

Rules learned

A nested ESXi vNIC on a standard VPC subnet is dead on arrival: vmk0’s made-up MAC loses to SpoofGuard, silently.
Attach nested-host vNICs only to a trunk subnet, with one binding map per VLAN. The map lives under the VLAN subnet and points at the trunk.
No DHCP in our VPC subnets, so bootstrap addressing goes through guestinfo.* (appliances) or cloud-init (Linux). Static IP plans are a feature in a lab anyway.
ESXi’s default TCP/IP stack has one gateway. Set per-vmk override gateways (esxcli ... ipv4 set -g) so vMotion and vSAN use their own subnet’s gateway.
Recreating a VM reallocates its NSX addresses. Pin what you depend on.
MTU: everything here ran at 1500. Raise the trunk and the nested vDS before you do vSAN at any real scale.

Broadcom documentation

Create a SubnetConnectionBindingMap CR on the Supervisor: the binding map CR and its vlanTrafficTag, shown for VKS egress subnets.
Creating a Child Segment: the NSX mechanism underneath: a binding map on the child segment, pointing at its parent with a VLAN ID.
Understanding SpoofGuard Segment Profile: port address bindings, and traffic dropped when its MAC or IP does not match them.
Add a VPC Service Profile: the DHCP settings and segment profiles, SpoofGuard included, that a VPC’s subnets inherit.
Deploy VMs with Configurable OVF Properties in vSphere Supervisor: OVF properties set through the VM Service’s vAppConfig transport.
Configure the VMkernel Adapter Gateway by Using esxcli Commands: a gateway per VMkernel adapter, set with esxcli.

Next in this series: what happens when you want ten of these labs, with byte-identical IP plans, firewalled from each other by construction. That’s where NSX VPCs go from “workaround” to genuinely better than physical.

Lab environment; opinions my own. Everything above was captured from a live VCF 9.1 environment — output trimmed for length, never edited for outcome.

The load balancer that must exist before the namespace

Wed, 16 Sep 2026 08:10:00 +0100

Everything was green. The VPC: realized. The namespace: ready. The VMs: powered on, endpoints populated, ports listening. And the LoadBalancer services sat at . For an hour.

This is the story of the least helpful error message in my recent memory, what it actually means, and the one-line ordering rule that would have saved an afternoon. If you’re doing self-service NSX VPCs on VCF 9 with the vSphere Supervisor, you will hit this. Bookmark accordingly.

The setup

The VPC was tenant-created, through VCF Automation’s Cloud Consumption Interface (CCI) API. A supervisor namespace was pinned to it, with a couple of VirtualMachineService objects of type LoadBalancer to publish SSH and HTTPS for the workloads inside. Standard stuff: the exact pattern that works out of the box in the org’s default VPC.

The Kubernetes side looked perfect:

$ kubectl get endpoints -n pod-a
NAME           ENDPOINTS                        AGE
esx01-access   172.30.0.40:443,172.30.0.40:22   6m36s
esx02-access   172.30.0.41:443,172.30.0.41:22   6m35s

Endpoints resolved. VIPs: nothing. The only clue was a recurring event:

Warning  FailedRealizeNSXResource  service/esx01-access
Generic error occurred during realizing network for Service

“Generic error.” Wonderful.

Digging: what NCP actually wants

The supervisor’s network container plugin (NCP) told the real story in its logs:

nsx_ujo.ncp.nsx.policy.lb_layer4_service Lb Service not Found for Namespace pod-a
NCP00270 Failed to process virtual ip for service ...: Lbs pod-a is not found
Encountered retryable error ... : Lbs pod-a is not found

NCP wants an NSX LBService in the namespace’s VPC. In the org’s default VPC, one exists, because the platform created it when the VPC was born. In my self-service VPC? Nobody had created one.

Fair enough. That’s actually documented behaviour, once you know where to look. A fresh VPC needs a LoadBalancer object. Before that, it needs a VPCAttachment to a connectivity profile with the service gateway enabled. Without the attachment, the load balancer creation itself fails, with a much better error message.

So I created the attachment, then the LBService. NSX: Realized=True. Problem solved?

Warning  FailedRealizeNSXResource  service/esx01-access
Generic error occurred during realizing network for Service

No.

The actual bug-shaped behaviour: a snapshot, not a lookup

Here’s the part that costs you the afternoon. That “retryable error” retries the lookup in NCP’s cache, not the discovery. NCP snapshots the VPC’s load balancer inventory when the namespace is created. An LBService that appears afterwards is never discovered, however long you wait:

Recreating the Kubernetes services: no effect.
Tagging the LBService with the nsx-op/* ownership tags the working ones carry: no effect, because the cache doesn’t re-read NSX.
Restarting NCP would force a full resync, but supervisor system pods are protected. Even Administrator@vsphere.local gets a Forbidden, which is a humbling thing to read.
Mutating the namespace to nudge a resync: also blocked, by the supervisor’s namespace validation webhook.

As a tenant, there is exactly one fix: delete and recreate the namespace, now that its VPC has a load balancer. Fifteen minutes of rebuild, for want of one ordering rule.

And the control experiment proves the rule. A namespace created after its VPC already had an LBService got its VIPs assigned without any drama:

esx01-access   VIP=192.168.144.34   22 OPEN · 443 OPEN
esx02-access   VIP=192.168.144.35   22 OPEN · 443 OPEN

Once the ordering is right, this is what “working” looks like. It’s the pod’s state a couple of minutes after a correctly ordered deployment:

What the requester sees once the order is right.

The ordering rule

For every self-service VPC that will publish LoadBalancer services, create these in order, before the namespace:

1. VPC                                   (vpc.nsx.vmware.com/v1alpha1)
2. VPCAttachment                         (connectivity profile w/ service gateway
                                          — LB creation errors without it)
3. LoadBalancer   {regionName, vpcName}  (the step everyone misses)
4. ...and only THEN the Supervisor Namespace

Encode it in whatever provisions your VPCs: a script, a pipeline, an operator. It’s four API calls, which is a good deal cheaper than an afternoon. And it turns a silent, undiagnosable into a platform that just works.

Why this matters outside the lab

Nobody buys a platform for its ordering rules. But this is exactly the kind of edge that decides whether self-service provisioning feels reliable or flaky to the people using it.

In a customer deployment, the answer isn’t a blog post. It’s provisioning automation that already does the four steps in the right order, every time, so a tenant never sees a VIP stuck at . Knowing where the sharp edges are, because you’ve been cut by them in a lab, is most of what an experienced delivery partner is for.

Rules learned

In a self-service NSX VPC, the LBService must predate the namespace. NCP discovers load balancers when the namespace is added, and never again.
FailedRealizeNSXResource: Generic error on a Service means: go and read the NCP logs. The real message (Lbs is not found, NCP00270) is there.
VPCAttachment (service gateway) is the prerequisite for the load balancer itself. That one, at least, fails loudly.
Retro-tagging NSX objects to look “owned” doesn’t help a cache that never re-reads. Recreating the namespace is the only tenant-level fix.
While you’re at it: new namespaces also reject VM creation until image status.disks syncs (~1–3 minutes after content library attach). Build the wait into your automation and both sharp edges disappear.

Broadcom documentation

Deploying Supervisor with VCF Networking with VPC: what NCP creates for a namespace given no VPC: a VPC with its load balancer and SNAT IP.
Create vSphere Namespaces on VPCs without SNAT and Load Balancer: without the VPC’s load balancer, LoadBalancer services and VirtualMachineServices cannot be deployed.
Add a VPC Connectivity Profile: the transit gateway, the service gateway and default outbound NAT.
Create and Configure a vSphere Namespace on a Supervisor with NSX VPC: placing a new namespace in an existing VPC.
Create a Virtual Private Cloud in VCF Automation: a tenant VPC, its connectivity profile and its load balancing setting.
Sample Blueprints in VCF Automation: a VPC, its VPCAttachment and a namespace that depends on them, as blueprint resources.

Previously in this series: nested ESXi inside an NSX VPC. Next: three datacenters, one IP plan, with identical isolated pods.

Lab environment; opinions my own. Output captured live, trimmed for length, never edited for outcome.

Three datacenters, one IP plan: identical isolated pods with NSX VPCs

Wed, 16 Sep 2026 08:00:00 +0100

Here are three hosts, all answering to vmk0 = 172.30.0.40:

ssh root@192.168.144.30  ->  [root@esx01-a:~]  vmk0  172.30.0.40
ssh root@192.168.144.32  ->  [root@esx01-b:~]  vmk0  172.30.0.40
ssh root@192.168.144.34  ->  [root@esx01-c:~]  vmk0  172.30.0.40

Same IP. Same VLAN. Same gateway. Same MAC address, as it turns out. And none of them can reach any of the others. This is the post where NSX VPCs stop being a workaround for nested labs, and become genuinely better than the physical alternative.

Why identical addressing matters

If you’ve ever built training pods, cert-study labs or per-team reproduction environments, you know the pain. Every copy needs a unique address plan. So every runbook, every screenshot and every “type this exact command” has to be parameterised per pod. Students in seat 7 see different numbers from the slides. Reproductions drift from the original.

The fix is obvious and normally impossible: give every pod the same addresses. On a physical fabric, that means VRFs, per-pod NAT and a great many favours from the network team. In an NSX VPC, it’s the default behaviour.

The mechanism: overlapping privateIPs

Each pod gets its own VPC, and every VPC declares the same private range:

apiVersion: vpc.nsx.vmware.com/v1alpha1
kind: VPC
metadata: {name: nested-vpc-a}      # then -b, then -c
spec:
  privateIPs: ["172.30.0.0/16"]      # identical in all three

A Private subnet is never advertised beyond its VPC, so NSX has no objection to three VPCs carving up the same /16. The pods aren’t “firewalled from each other”. There is simply no route between them: isolation by construction, not by policy.

NSX’s subnet view filtered to sn-mgmt: four rows, four VPCs, one CIDR.

The trick: deterministic realization

Identical ranges aren’t enough. I want identical subnets, so that the management gateway is .33 and the hosts are .40/.41 in every pod.

NSX allocates subnets from privateIPs in creation order, and a fresh VPC allocates deterministically. So the topology is applied in a fixed order (trunk, mgmt, vMotion, vSAN), and every pod realizes the same map:

Subnet	Realized	VLAN	Hosts
sn-trunk	172.30.0.0/27	—	(carries the tags)
sn-mgmt	172.30.0.32/27	1610	.40 / .41, gw .33
sn-vmotion	172.30.0.64/27	1611	.70 / .71, gw .65
sn-vsan	172.30.0.96/27	1612	.100 / .101, gw .97

In the catalog blueprint, dependsOn between the subnet resources enforces that order. It’s the one place a declarative tool needs to be told about sequence. Skip it and two pods can come out with mgmt and vMotion swapped, which works perfectly and confuses everyone.

The door: one VIP per host

Each pod is unreachable from outside by design. So each host gets a VirtualMachineService of type LoadBalancer, publishing SSH and HTTPS. The VIPs come from the org’s external block, and they’re the only addresses that differ between pods:

Pod	VPC	esx01 VIP	esx02 VIP
a	nested-vpc-a	192.168.144.30	.31
b	nested-vpc-b	192.168.144.32	.33
c	nested-vpc-c	192.168.144.34	.35

That’s how the opening transcript works: three VIPs, three hosts, one inside address.

Proving the isolation

Claims are cheap. Here’s the test matrix, from a VM in a fourth VPC (the org default):

ping 172.30.0.140 (own VPC)....... REACHABLE
ping 172.30.0.40  (pod space)..... unreachable
curl http://172.31.0.2/ (shared).. shared-svc repo01

Its own VPC’s 172.30.0.140: reachable. Then 172.30.0.40, an address that exists in three other VPCs at once: unreachable, because from here there is no such route. (The third line is the shared-services VPC, which is the next post.)

Inside each pod, east-west traffic is normal: esx01 → esx02 vmkping passes on all three VLANs, in all three pods. And here’s the detail I didn’t expect. The nested-ESXi appliance derives vmk0’s MAC deterministically from its config, so the three hosts share a MAC as well as an IP.

That’s harmless, because each VPC is its own L2 domain. On one shared physical segment, it would make for a rather more exciting afternoon. Here, it’s just a nice demonstration of how complete the separation is.

What this replaces

	Physical / VLAN-based pods	VPC pods
Identical addressing	VRF per pod + NAT, fabric change per pod	default behaviour
Adding a pod	switch config, IPAM, firewall rules	one API call for the VPC, one blueprint request
Isolation guarantee	policy (auditable, breakable)	topology (no route exists)
Tenant self-service	no	yes — the VPC is a tenant object

Why this matters outside the lab

“Identical environments” sounds like a lab nicety. It’s actually one of the most requested things in enterprise IT, usually asked for in other words:

Training at scale: every seat in the room sees the same addresses as the slides, so material is written once and never parameterised per pod.
Per-engineer or per-team replicas of a reference environment, for development and testing that behaves exactly like the original.
Regulatory or business-unit separation on shared infrastructure, without VRF sprawl or a bespoke firewall estate. Isolation is a property of the topology, which is the easiest kind to evidence to an auditor.
Blue/green copies of an environment for change rehearsal, then cut-over or discard.

On a physical network, each of these is a project. On VCF with NSX VPCs, it’s a template.

Rules learned

Overlapping privateIPs across VPCs is supported and intentional. Identical pods are a feature, not a hack.
Fresh VPCs realize subnets deterministically in creation order. Fix the order (dependsOn in a blueprint) and every pod gets the same map.
Pods are unreachable from outside by construction. Publish exactly what you mean to through LoadBalancer VIPs from the external block.
Prove isolation from a different VPC, with a positive control (own VPC reachable) beside the negative.
Expect duplicate MACs across pods from appliance images. It looks alarming, and it’s fine.

Broadcom documentation

Create a VPC: private CIDRs are local to a VPC and can overlap between VPCs.
Add a Subnet to a VPC: Private subnets, and subnet CIDRs auto-allocated from the VPC’s IP blocks by size.
Virtual Private Clouds Overview: the access modes, NAT, and external IPs as a way in to private workloads.
Creating bindings and dependencies between resources in blueprints in VCF Automation: dependsOn and the build order it sets.
Create a Virtual Private Cloud in VCF Automation: a VPC as an organization’s own object, with its own private CIDRs.

Previously: the LB that must exist first. Next: one WSUS for pods that can’t see each other.

Lab environment; opinions my own. Output captured live, trimmed for length, never edited for outcome.

Shared services for isolated tenants: PrivateTGW subnets

Wed, 16 Sep 2026 07:50:00 +0100

The pods from the last post are perfectly isolated. That’s the requirement, and straight away the problem. Every one of them needs Windows updates, a package repo, DNS, maybe a domain controller. Do I really run a WSUS (Windows Server Update Services) per pod?

No. There’s a third subnet access mode for exactly this. The design it enables is hub-and-spoke, with a very specific property: spokes reach the hub; the hub cannot reach the spokes; spokes never reach each other.

The three access modes

Everything in this series comes down to one field on a VPC subnet, which is either elegant or slightly deflating, depending on the day:

`accessMode`	Advertised to	Use
`Private`	nobody outside the VPC	workloads — isolation and overlapping CIDRs
`PrivateTGW`	every VPC attached to the org’s transit gateway	shared services
`Public`	the external network	internet/corp-facing endpoints

PrivateTGW (private transit gateway) subnets draw their addresses from a separate transit block (here 172.31.0.0/…), not from the VPC’s own privateIPs. That’s the key. The shared range can’t collide with the pods' 172.30.0.0/16, because it comes from a different pool that all VPCs agree on.

The build

One more VPC, shared-svc, follows the same ordering rules as any other: VPC, then VPCAttachment, then LoadBalancer, then namespace. Then a single subnet:

apiVersion: crd.nsx.vmware.com/v1alpha1
kind: Subnet
metadata: {name: sn-services}
spec: {accessMode: PrivateTGW, ipv4SubnetSize: 32}

It realized as 172.31.0.0/27. A VM on it, svc-repo01 at 172.31.0.2, serving HTTP, became the shared repo. Not the most glamorous job in the lab, but somebody has to do it.

One way by construction: the pods reach the hub, and nothing reaches a pod.

The test that matters is directional

Reachability to the service is the easy claim. From esx01 in each of the three pods (ESXi ships python3, so urllib is the test client):

pod-a esx01 -> http://172.31.0.2/   200  shared-svc repo01
pod-b esx01 -> http://172.31.0.2/   200  shared-svc repo01
pod-c esx01 -> http://172.31.0.2/   200  shared-svc repo01
default-vpc  -> http://172.31.0.2/   200  shared-svc repo01

Three pods with identical source addresses (172.30.0.40) all hit one service, and all get answers. So how does each reply find its way back to the right pod, when three of them claim .40?

Because pod traffic crosses the transit gateway SNAT’d to a per-VPC address from the external IP block, by the default outbound NAT in each VPC’s connectivity profile. The service never sees 172.30.0.40; it sees three distinct SNAT addresses. Ambiguity never arises.

Now the other direction, from svc-repo01 back towards a pod:

svc-repo01 -> 172.30.0.40   unreachable
svc-repo01 -> 172.30.0.41   unreachable

Not “blocked”: unroutable. Pod subnets are Private, so they were never advertised to the transit gateway. Even if they had been, 172.30.0.40 would be ambiguous across three VPCs. The hub literally cannot start a conversation with a spoke.

For a shared service that will one day be compromised, that’s the property you want. Gloomy, I know, but gloom is a perfectly good security posture.

What goes in the hub

Anything that pods consume and that is stateless about which pod is asking: WSUS and patch mirrors, OS and package repos, container registries, NTP, DNS forwarders, licence servers. Domain controllers work too, with the usual caveat: identical hostnames across pods need per-pod domains or a naming scheme.

What does not go in the hub: anything that needs to reach into a pod, such as monitoring pollers, backup agents that pull, and jump hosts. Those either live in the pod, or the pod publishes a LoadBalancer VIP for them: the deliberate door from part 0.

Tightening further

The transit gateway gives you reachability; policy gives you precision. A VPCGatewayFirewallPolicy on shared-svc can restrict inbound traffic to tcp/80,443 from the transit range and nothing else. Then the repo is a repo, and not a foothold.

I left it open for the test. You shouldn’t: do as I say, not as I did.

Why this matters outside the lab

This is the pattern that makes isolated tenants affordable. Without it, every isolated environment needs its own patch server, repository, DNS and directory. That’s cost and drift, and they quietly kill the idea.

With it, a customer runs one set of shared services for dozens of tenants, and keeps them patched in one place. They can still show a security reviewer that the shared service has no path back into any tenant.

The same hub serves well beyond patching: central logging and monitoring collectors, licence servers, artifact registries, build agents. Anything tenants consume, but shouldn’t be reachable by.

Rules learned

PrivateTGW is the shared-services mode: addresses from the transit block, advertised to every attached VPC, no collision with pod space.
Access is one-way by construction: pods to service works (SNAT’d per VPC), and service to pod has no route. Test both directions and write down both results.
Identical pod addressing and shared services coexist because of the SNAT. The hub sees per-VPC SNAT addresses, never the overlapping private ones.
The same ordering rules apply to the hub VPC: VPC, attachment, load balancer, namespace, subnets, wait for image sync, then workloads.
Add a gateway firewall policy on the hub. Reachability is not authorisation.

Broadcom documentation

Add a Subnet to a VPC: the Private Transit Gateway mode: addresses from the private transit gateway blocks, reachable from the project’s VPCs.
Add a VPC Connectivity Profile: private transit gateway IP blocks, and the SNAT rule default outbound NAT gives each VPC.
Transit Gateways: traffic between a tenant’s VPCs, and out to the external network.
Virtual Private Clouds Overview: Private, Private - Transit Gateway and Public subnets side by side.
Add GFW Rules for VPCs: north-south gateway firewall rules for a VPC’s ingress and egress traffic.

Previously: three datacenters, one IP plan. Next: the whole pod as a single catalog item.

Lab environment; opinions my own. Output captured live, trimmed for length, never edited for outcome.

A datacenter in a catalog tile: nested ESXi pods via VCF Automation All Apps

Wed, 16 Sep 2026 07:40:00 +0100

Everything in this series so far was built with kubectl and API calls. That proves the platform. It doesn’t make a product.

This post turns the pod into a catalog item. Fill in a name, pick a VPC, click Request, and a few minutes later there’s a datacenter in miniature with two SSH prompts waiting. Barely time to put the kettle on.

All Apps in one paragraph

VCF Automation 9.1 has two provisioning models side by side. VM Apps is the classic Aria Automation path: cloud templates run through an IaaS (infrastructure as a service) engine that drives vCenter. All Apps is the supervisor-native path. Its blueprint composes Kubernetes objects (a Supervisor Namespace, VM Service VMs, NSX subnets, vSphere Kubernetes Service clusters), and the vSphere Supervisor’s controllers reconcile them.

A blueprint is formatVersion: 2, and its resources are CCI.Supervisor.Namespace and CCI.Supervisor.Resource. The second is literally “here’s a manifest, apply it in that namespace.” So the blueprint is a composition of the manifests from the earlier posts, with two additions: inputs, and dependsOn.

The blueprint, section by section

Inputs — the form

inputs:
  podName:  {type: string, default: nested-pod, pattern: '^[a-z0-9]([-a-z0-9]*[a-z0-9])?$'}
  vpcName:  {type: string, description: Must exist and be Realized before deploying.}
  esxOva:   {type: string, default: vmi-61bb062ddfc506b79}   # Nested ESXi 9.1 appliance
  isoImage: {type: string, default: vmi-39f562e2ae9e9c501}   # the ISO to attach
  vmClass:  {type: string, default: best-effort-large, enum: [best-effort-large, best-effort-xlarge, best-effort-2xlarge]}

The namespace — with libraries attached

namespace:
  type: CCI.Supervisor.Namespace
  properties:
    generateName: ${input.podName}-        # NOT name — new namespaces get a suffix
    className: large
    regionName: f06
    vpcName: ${input.vpcName}              # pins the namespace to the pod's VPC
    storageClasses: [{name: vSAN Default Storage Policy, limit: 400000Mi}]
    zones: [{name: domain-c9, cpuLimit: 40000M, memoryLimit: 64000Mi, ...}]
    contentSources:
      - {name: ISO, type: ContentLibrary}
      - {name: f06-vks-lib01, type: ContentLibrary}

contentSources is the line that closes the gap a lot of first attempts hit. Our libraries were plain vCenter libraries, and a namespace created by VCF Automation got none of them. No libraries meant no VirtualMachineImages, so nothing could be deployed. An empty namespace is very tidy, and of no use to anyone.

Declaring the libraries here attaches them at creation. (The 9.1 docs say a namespace class is assigned a content library automatically, and provider libraries are shared with every namespace.)

The topology — ordered on purpose

snTrunk:   {type: CCI.Supervisor.Resource, properties: {context: ${resource.namespace.id}, manifest: }}
snMgmt:    {dependsOn: [snTrunk], ...  manifest: }
snVmotion: {dependsOn: [snMgmt],  ...  manifest: }
bmMgmt:    {... manifest:  sn-trunk, vlan 1610>}
bmVmotion: {... manifest:  sn-trunk, vlan 1611>}

The dependsOn chain is the whole reason every pod has identical CIDRs. A fresh VPC realizes its subnets in the order they were created, and the blueprint fixes that order.

The hosts — dual-NIC, ISO attached, bootstrapped by OVF

esx01:
  type: CCI.Supervisor.Resource
  dependsOn: [bmMgmt]                    # no point booting before VLAN 1610 exists
  properties:
    manifest:
      kind: VirtualMachine                # vmoperator.vmware.com/v1alpha5
      spec:
        hardware:
          cdrom: [ ... the ISO, declared, connected ... ]
        network:
          interfaces: [ eth0 -> sn-trunk, eth1 -> sn-trunk ]
        bootstrap:
          vAppConfig: [ guestinfo.hostname / ipaddress / vlan / ... ]
  wait:
    fields: [{path: status.powerState, value: PoweredOn}]

(Abridged: the full resource carries the image references, VM class, guest ID and the complete guestinfo set.)

Two vNICs, both on the trunk: the nested equivalent of a VCF host’s two pNICs. The ISO rides along as a declarative CD-ROM.

The wait block makes the deployment’s completion mean something: the request doesn’t finish until the host is powered on. Finishing any earlier would be optimism, not automation.

The doors — one VIP per host

Each host gets a VirtualMachineService of type LoadBalancer, which selects it by label and publishes 22 and 443. A blueprint output then reads the VIP back out of the service’s status:

outputs:
  esx01Ssh: {value: "ssh root@${resource.esx01Access.object.status.loadBalancer.ingress[0].ip}"}

The outputs show up in the deployment view, so the requester gets the SSH command rather than a scavenger hunt.

The request flow, end to end: request → deployment in progress → complete.

What the blueprint cannot express (yet)

Three cluster-scoped objects must exist before the request, in this order:

VPC: privateIPs: 172.30.0.0/16, the same in every pod.
VPCAttachment: the connectivity profile with the service gateway. Without it, creating the load balancer fails loudly.
LoadBalancer: silently, permanently required before the namespace.

VCF Automation 9.1 does have blueprint types for the first two: CCI.VPC, and CCI.VPC.Configuration with kind: VPCAttachment. One of its sample blueprints uses both. There is none for the third.

A later test confirmed it. CCI.VPC.Configuration rejects a LoadBalancer kind, and a VPC created that way comes up with load balancing off. Its namespace’s VIPs would sit pending, waiting for a load balancer that isn’t coming.

Today that’s a short script or a runbook step per pod. The honest framing: the blueprint is the pod, the VPC is the tenancy, and tenancy is still created one layer up. I’d expect the load balancer to become blueprintable too. Until then, keep the three calls next to the blueprint in version control.

Publishing: one version at a time

The order is blueprint, then BlueprintVersion, then release. Validation happens at version time, not at create time. The result lives in status.validationMessages rather than the HTTP code, so a 200 with ContentValid: False is a thing: the HTTP equivalent of “yes, but no”.

And only one version can be published. Unrelease 1.0.0 before releasing 1.1.0, or you get a 409. (The full list of sharp edges is its own post.)

Why this matters outside the lab

This is where platform engineering turns into a service. The gap between “we can build you an environment” and “request one from the catalog” is the gap between days and minutes. It’s also the gap between a bespoke build and one that is consistent, quota-controlled and recorded every time. For an organisation that means:

Time-to-environment measured in minutes, requested by the people who need it, without a queue.
Consistency by construction. Every environment comes from the same definition, so support, training material and runbooks all match.
Governance built in. Quotas, ownership, history and clean teardown are properties of the deployment record, not a spreadsheet.

The nested-ESXi pod is one catalog item. The same approach delivers any shape of environment: application stacks for developers, sandboxes for a proof of concept, demo kits for a sales team, isolated builds for a partner.

Rules learned

All Apps blueprints are compositions of manifests: CCI.Supervisor.Namespace plus CCI.Supervisor.Resource per object. If it works with kubectl, it works in a blueprint.
Use generateName, not name, for the namespace, and contentSources to attach libraries at creation. Keep zones/storageClasses flat, not wrapped.
dependsOn is how you get deterministic CIDRs: order the subnets.
wait.fields turns “request complete” into “host is powered on”.
VPC, VPCAttachment and LoadBalancer are prerequisites outside the blueprint, in that order, before every request.
One published version per blueprint. Validation results live in status, not in the HTTP response.

Broadcom documentation

Sample Blueprints in VCF Automation: CCI.Supervisor.Namespace and CCI.Supervisor.Resource examples, with generateName and context.
Creating bindings and dependencies between resources in blueprints in VCF Automation: dependsOn and property bindings, which set the build order.
Specifying formatVersion in Blueprints in VCF Automation: what formatVersion: 2 adds, outputs included.
Versioning Blueprints in VCF Automation: blueprint versions, and releasing one to the catalog.
Creating and Managing Content Libraries for Stand-Alone VMs in vSphere Supervisor: VM content libraries and the namespaces they are associated with.
Deploy VMs with Configurable OVF Properties in vSphere Supervisor: OVF properties set through the VM Service’s vAppConfig transport.

Previously: shared services for isolated tenants. This closes the Pod Papers’ core arc. The companion posts on dual-NIC, no-DHCP bootstrap and blueprint gotchas fill in the details.

Lab environment; opinions my own. Blueprint nested-esxi-pod 1.1.0 is live in the lab catalog; YAML above trimmed for length.

Dual-NIC nested hosts: what redundancy means when the fabric is virtual

Wed, 16 Sep 2026 07:30:00 +0100

“Naturally, a VCF host has at least two NICs. Are we testing that, or have you virtualised it away?”

Fair question, and like most fair questions it has an irritating two-part answer. In a nested lab, the outer host provides the physical redundancy: its vDS and its NSX uplinks. A second vNIC on the nested VM adds exactly none of that.

But VCF doesn’t know it’s nested. We expected bringup’s host validation, and the vDS uplink teaming it sets up, to want two vmnics. We never tried a host with one. For the record, Broadcom’s docs allow single-pNIC hosts, and the VCF Installer’s 9.1 known issue only bites single-pNIC hosts that use an NFS datastore.

So our nested hosts get two vNICs, both on the trunk subnet. Which leaves the fun question (for a given value of fun): does failover between them actually work inside a VPC?

The whole test on one page. The session I’m typing in rides the very path I break.

The setup

Both vNICs sit on the same sn-trunk subnet (the trunk from part 1), and ESXi sees two 10G vmnics:

vSwitch0 teams them active/active with the default originating-port-ID policy. Every port group inherits it: Management 1610, vMotion 1611 and vSAN 1612. Nothing you wouldn’t do on metal. Thrilling stuff, I know.

The test: pull a NIC while watching from inside

The interesting part isn’t whether pings carry on. It’s which session I’m watching from.

I’m SSH’d into esx01 through its public VIP. So my session runs through the NSX load balancer, the VPC, the trunk port and whichever vmnic happens to carry vmk0. If failover breaks anything, it breaks the terminal I’m typing in: the networking equivalent of sawing off the branch you’re sitting on. Low effort, high stakes: the best kind of test.

[root@esx01-a:~] esxcli network nic list
Name    ...  Admin Status  Link Status  Speed  MAC Address
vmnic0  ...  Up            Up           10000  04:50:56:00:5c:03
vmnic1  ...  Up            Up           10000  04:50:56:00:68:00

[root@esx01-a:~] esxcli network nic down -n vmnic0    # FAIL THE FIRST NIC
vmnic0  ...  Down          Down             0  04:50:56:00:5c:03
vmnic1  ...  Up            Up           10000  04:50:56:00:68:00

[root@esx01-a:~] vmkping -I vmk1 172.30.0.71          # vMotion VLAN, now over vmnic1
3 packets transmitted, 3 packets received, 0% packet loss

[root@esx01-a:~] vmkping -I vmk2 172.30.0.101         # vSAN VLAN, now over vmnic1
3 packets transmitted, 3 packets received, 0% packet loss

[root@esx01-a:~] esxcli network nic up -n vmnic0      # restore
# session never dropped.

Before and after: vmnic0 down, every VLAN still passing.

The full transcript, as captured

Our VPC subnets have no DHCP — and that's fine

Wed, 16 Sep 2026 07:20:00 +0100

The second trap from part 1 deserves its own short post, because it catches everything, not just ESXi: our VPC subnets have DHCP deactivated. NSX VPC subnets can run a DHCP server or relay (Add a Subnet to a VPC). But the subnets the Supervisor made for us, and every Subnet we created with the defaults, came up DHCP_DEACTIVATED with static IP allocation.

Drop a stock appliance onto one and it will boot, sit at “waiting for DHCP”, and wait politely until the heat death of the universe.

This isn’t a gap. It’s the model. NSX allocates the address at the port and pins it there with address bindings, and the guest has to be told what it was given. The VM Service does that telling through bootstrap providers. Once you know the three of them, static addressing stops being a chore and starts being a feature.

Three providers, three guest types

Guest	Provider	Carries
Linux	`cloudInit`	user-data (users, `write_files`, `runcmd`) + network config
Windows	`sysprep`	unattend XML / sysprep spec, identity, network
Appliances (OVF)	`vAppConfig`	OVF properties (`guestinfo.*`) the appliance reads on boot

All three are typed fields on the VirtualMachine object, not bolt-on customisation specs. The platform already knows the network side: the VM Service knows which subnet each interface landed on, and what NSX allocated. So for cloud-init and sysprep, the addressing is injected for you. Appliances are the exception, because each one has its own idea of which properties it wants.

Appliances: vAppConfig

The nested-ESXi appliance reads guestinfo.* OVF properties. In the VM Service spec, that’s:

bootstrap:
  vAppConfig:
    properties:
      - {key: guestinfo.hostname,  value: {value: "esx01.pod-a.res.lab"}}
      - {key: guestinfo.ipaddress, value: {value: "172.30.0.40"}}
      - {key: guestinfo.netmask,   value: {value: "255.255.255.224"}}
      - {key: guestinfo.gateway,   value: {value: "172.30.0.33"}}
      - {key: guestinfo.vlan,      value: {value: "1610"}}
      - {key: guestinfo.dns,       value: {value: "10.20.52.1"}}
      - {key: guestinfo.ssh,       value: {value: "True"}}

The one that trips people up: the address you give must be the one NSX allocated to the port. In a standard subnet, SpoofGuard enforces that, and it doesn’t negotiate. On a trunk subnet, the VLAN subnets have their own allocations, and you’re choosing addresses within them.

Either way, pick from the realized range. And remember that recreating a VM reallocates its addresses. So the fixed .40/.41 in a deterministic pod is a design choice, not luck.

Linux: cloud-init

A Secret holding user-data, referenced from the VM:

bootstrap:
  cloudInit:
    cloudConfig:
      users: [ ... a local user with a key ... ]
      write_files:
        - path: /var/www/html/index.html
          content: "shared-svc repo01\n"
      runcmd:
        - [systemctl, enable, --now, nginx]

Networking arrives through the platform’s own network-config, so you don’t write it. Every line of YAML I don’t write is a line I can’t get wrong.

The svc-repo01 VM from the shared-services post was exactly this: write_files plus runcmd, with its web page verified from three pods.

Windows: sysprep

bootstrap:
  sysprep:
    sysprep:
      guiUnattended: {autoLogon: true, autoLogonCount: 1, timeZone: 85}
      identification: {joinWorkgroup: WORKGROUP}
      userData: {fullName: Lab, orgName: Lab, computerName: {name: win01}}

Or use rawSysprep with an unattend XML in a Secret, if you already have one. The ISO from the blueprint post can ride along as a declarative hardware.cdrom. That’s handy for tools and agents on first boot.

The ESXi footnote: one gateway, many vmks

Once the appliance is up and you add vMotion and vSAN vmks on their own subnets, you hit a detail that makes the Host Client look wrong:

ESXi’s default TCP/IP stack has one default gateway (vmk0’s .33), and the UI repeats it on every vmk row with total confidence. Same-subnet vMotion never uses a gateway, so nothing breaks. But each NSX subnet does have its own gateway, and cross-subnet traffic from vmk1 and vmk2 would take the wrong exit. Set per-vmk override gateways:

esxcli network ip interface ipv4 set -i vmk1 -t static -I 172.30.0.70  -N 255.255.255.224 -g 172.30.0.65
esxcli network ip interface ipv4 set -i vmk2 -t static -I 172.30.0.100 -N 255.255.255.224 -g 172.30.0.97

Now the display is truthful and the routing is correct. (The full-realism alternative is a dedicated vmotion netstack for vmk1. I kept the default stack so the service tags stay visible in the Host Client.)

Why this matters outside the lab

For the business, “no DHCP” translates into something security and operations teams both want: predictable addressing. Every environment has a known address plan, firewall rules can be written once, and nothing turns up on the network with an address nobody expected.

Bootstrap providers deliver the second benefit. Images stay generic and configuration is injected at deploy time, so there are fewer golden images to maintain and far less drift between environments.

Rules learned

No DHCP in our VPC subnets: that’s the Supervisor’s default, not a VPC limit. NSX allocates at the port, and the guest is told through a bootstrap provider.
cloudInit (Linux), sysprep (Windows) and vAppConfig (appliances) are typed fields on the VM, not customisation specs.
Appliance addresses must match the realized subnet. Fix the order of subnet creation if you want fixed addresses across pods.
ESXi has one default gateway per stack. Set -g per vmk, or the Host Client lies to you and cross-subnet traffic exits wrong.
A static IP plan is a feature in a lab: it’s what makes screenshots, runbooks and pods identical.

Broadcom documentation

Add a Subnet to a VPC: a subnet’s DHCP setting: none for static addresses, a DHCP server, or DHCP relay.
Understanding SpoofGuard Segment Profile: the port address bindings SpoofGuard enforces.
Provision a VM Using Self-Service: the four bootstrap methods, cloud-init, Sysprep, Linuxprep and vAppConfig, and static IP allocation.
Deploy VMs with Configurable OVF Properties in vSphere Supervisor: OVF properties set through the VM Service’s vAppConfig transport.
Configure the VMkernel Adapter Gateway by Using esxcli Commands: a gateway per VMkernel adapter, set with esxcli.

Companion to nested ESXi inside an NSX VPC.

Lab environment; opinions my own.

NSX on The Nested Lab

Every resource in the VCF Automation 9.1 blueprint designer: the complete guide

The shape of a blueprint

How the palette maps to YAML

Where the designer and the platform disagree

What every resource shares

Properties on every type

wait: when a resource is finished

Reading a resource’s live state

Checking a manifest before you deploy it

Supervisor Namespace

What's a VPC? Let Pac-Man explain

Why this matters outside the lab

Rules learned

Broadcom documentation

Nested ESXi inside an NSX VPC: the trunk-subnet design

The setup

Failure #1: the silent blackhole

The design that works: a trunk subnet + binding maps

Does it actually work? The receipts

Why this matters outside the lab

Rules learned

Broadcom documentation

The load balancer that must exist before the namespace

The setup

Digging: what NCP actually wants

The actual bug-shaped behaviour: a snapshot, not a lookup

The ordering rule

Why this matters outside the lab

Rules learned

Broadcom documentation

Three datacenters, one IP plan: identical isolated pods with NSX VPCs

Why identical addressing matters

The mechanism: overlapping privateIPs

The trick: deterministic realization

The door: one VIP per host

Proving the isolation

What this replaces

Why this matters outside the lab

Rules learned

Broadcom documentation

Shared services for isolated tenants: PrivateTGW subnets

The three access modes

The build

The test that matters is directional

What goes in the hub

Tightening further

Why this matters outside the lab

Rules learned

Broadcom documentation

A datacenter in a catalog tile: nested ESXi pods via VCF Automation All Apps

All Apps in one paragraph

The blueprint, section by section

Inputs — the form

The namespace — with libraries attached

The topology — ordered on purpose

The hosts — dual-NIC, ISO attached, bootstrapped by OVF

The doors — one VIP per host

What the blueprint cannot express (yet)

Publishing: one version at a time

Why this matters outside the lab

Rules learned

Broadcom documentation

Dual-NIC nested hosts: what redundancy means when the fabric is virtual

The setup

The test: pull a NIC while watching from inside

Our VPC subnets have no DHCP — and that's fine

Three providers, three guest types

Appliances: vAppConfig

Linux: cloud-init

Windows: sysprep

The ESXi footnote: one gateway, many vmks

Why this matters outside the lab

Rules learned

Broadcom documentation

`wait`: when a resource is finished