The lab has Pac-Man running twice on VCF 9.1 — once on a VKS cluster I built by hand with kubectl, once on a cluster deployed through VCF Automation’s catalog. Both live inside VPCs. Both were reachable when I started:

http://192.168.144.15/  ->  <title>Pacman in HTML 5 Canvas
http://192.168.144.23/  ->  <title>Pacman in HTML 5 Canvas

A VPC is a private universe with a door policy

Think of an NSX VPC as a tenant’s own routed network space: its own subnets, its own gateway, its own address plan — carved out by the tenant, not filed as a ticket with the network team. Three rules define it:

1. Private by default. A Private subnet is reachable only from inside the same VPC. Nobody outside can route to it — not other tenants, not other VPCs in the same org, not the corporate network.
2. Your addresses are your business. Because private subnets aren’t advertised anywhere, two VPCs can use identical CIDRs. (This is the superpower the rest of the series is built on.)
3. Every door out is deliberate. Traffic leaves via the transit gateway — SNAT’d — or arrives via a LoadBalancer VIP from an external block the provider allocated. Nothing is exposed by accident.

Pac-Man’s pods sit on a private subnet. The only reason 192.168.144.23 answers is a Kubernetes Service of type LoadBalancer, which NSX turns into a VIP on the VPC’s load balancer. So let’s remove the door.

Before: close the door

$ kubectl patch svc pacman -n pacman -p '{"spec":{"type":"ClusterIP"}}'
service/pacman patched
NAME     TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
pacman   ClusterIP   10.106.219.213   <none>        80/TCP    4d13h

$ curl -m 5 http://192.168.144.23/
curl: timed out / unreachable

The pods are running. The service exists. The game is fine — for anything inside the VPC. From my desk it’s simply gone. That’s the whole VPC model in one curl: the boundary isn’t a firewall rule somebody wrote, it’s the absence of a route.

After: open it again

$ kubectl patch svc pacman -n pacman -p '{"spec":{"type":"LoadBalancer"}}'
service/pacman patched
NAME     TYPE           CLUSTER-IP       EXTERNAL-IP      PORT(S)        AGE
pacman   LoadBalancer   10.106.219.213   192.168.144.23   80:31467/TCP   4d13h

Same VIP handed straight back. NSX programmed a virtual server and pool on the VPC LB; the supervisor stitched it to the cluster’s NodePort. Door open.

And then the game didn’t load.

The outage I gave myself (this is the useful bit)

Everything was green:

• kubectl get svc — LoadBalancer, VIP assigned
• kubectl get endpoints — pod IPs present
• NSX — virtual server up, pool members healthy
• iptables on the node — NodePort rules identical to a working neighbour service

Five layers, all green, and curl hung. The control experiment was the manually-built cluster’s Pac-Man at .15, untouched throughout, still playing.

The cause: my “harmless” ClusterIP patch earlier had included a ports list. kubectl patch with a merge patch replaces arrays, it doesn’t merge them — and my array said targetPort: 80. Pac-Man listens on 8080. Every layer above was faithfully forwarding traffic to a port nothing was listening on, and every layer reported success because its job was done.

$ kubectl patch svc pacman -n pacman --type=json \
    -p '[{"op":"replace","path":"/spec/ports/0/targetPort","value":8080}]'
service/pacman patched
$ curl -s http://192.168.144.23/ | grep -o '<title>.*</title>'
<title>Pacman in HTML 5 Canvas</title>

Instant recovery. The diagnosis walked the entire paravirtual chain — VIP → supervisor VirtualMachineService → NSX VS/pool → NodePort iptables → pod — and it’s exactly the walk you’ll need one day:

Bonus find on the way: kube-proxy and Antrea on that cluster had dropped their API watches days earlier (http2: client connection lost) and never re-established informers until restarted. It didn’t cause this outage, but it’s the kind of thing you only find when you’re forced to look.

Why this matters outside the lab

If you run a platform for more than one team, this is the feature you’ve been asking the network team for. A VPC gives each team, project or customer its own private network space — created by them, in minutes, with nothing reachable from outside until they publish it. Security teams like it for the same reason developers do: exposure is a deliberate, auditable act, not a side effect of plugging something in.

What organisations do with it once they have it:

• Per-team sandboxes that can’t see each other, provisioned without a ticket.
• Partner or supplier environments isolated from the corporate estate but hosted on the same platform.
• Multi-tenant hosting — service providers and internal IT alike — with isolation enforced by topology rather than a growing pile of firewall rules.

Rules learned

• A VPC’s boundary is the absence of a route, not a rule. Private subnets are unreachable from outside by construction — which is also why identical CIDRs across VPCs just work.
• A LoadBalancer service is the deliberate door: NSX VIP from the external block, programmed per service. Flip the type and the door closes with nothing else to clean up.
• kubectl patch (merge) replaces spec.ports, it doesn’t merge it. Patch a single field with --type=json, or don’t include the array.
• Five green layers can hide one wrong integer. Keep a working control (here: the untouched .15 instance) and compare layer by layer.
• Read kubectl get endpoints as IP:targetPort — the port is the part people skim.

Next in the Pod Papers: nested ESXi inside a VPC — where “private by default” meets a host that fakes its own MAC address.

Lab environment; opinions my own. Output captured live, trimmed for length, never edited for outcome — including the outage.

That’s how my first attempt at running nested ESXi inside an NSX VPC ended, and the failure mode is nasty precisely because nothing looks wrong. If you’re trying to build nested vSphere labs on VCF 9 with VPC networking, this post is the map of the minefield — and the design that gets you across it, verified live.

The setup

VCF 9.1, vSphere Supervisor with NSX VPC networking. The goal: deploy nested ESXi hosts as ordinary VM Service VMs inside a tenant’s VPC — no physical fabric changes, no provider tickets, no special treatment. The kind of thing you want for training pods, cert-study labs, or reproducing customer issues.

Nested ESXi needs what physical ESXi needs: a management network, vMotion, vSAN — traditionally VLANs trunked to every host. But a VPC is an overlay world. There are no VLANs to trunk. So what happens if you just attach the nested host’s vNIC to a normal VPC subnet?

Failure #1: the silent blackhole

Here’s the trap. A standard VPC subnet port gets address bindings: NSX pins the exact IP + MAC it allocated to that vNIC, and SpoofGuard drops everything else.

ESXi’s vmk0 doesn’t use the vNIC’s MAC. It synthesises its own:

vmk0
   MAC Address: 00:50:ac:1e:00:8c     <- NOT the vNIC MAC (04:50:56:...)

So every frame the management interface sends carries a MAC the port doesn’t own. NSX drops it all — ARP, ping, everything — while the host itself boots green and reports healthy. There is no error anywhere. You just can’t reach it, ever.

(There’s a second trap stacked on top: VPC subnets run with DHCP deactivated, so the appliance also sits at “waiting for DHCP” unless you inject static addressing via OVF guestinfo.* properties. More on that below.)

The design that works: a trunk subnet + binding maps

The fix isn’t a hack — it’s a first-class NSX VPC construct that’s barely documented in the wild: SubnetConnectionBindingMap.

The idea:

1. Create one ordinary VPC subnet to act as a trunk (sn-trunk). The nested host’s vNICs attach only here.
2. Create a normal VPC subnet per traditional network — sn-mgmt, sn-vmotion, sn-vsan.
3. Bind each of those to the trunk with a binding map carrying a VLAN tag. The nested host’s vSwitch tags frames exactly as it would on metal; the binding map strips the tag and delivers the frame into the right subnet.

Pure L2 demultiplexing. One vNIC carries N VLANs, the VPC never routes on a tag, and the physical fabric never sees any of it (the 802.1Q header rides inside the Geneve overlay).

All of it is tenant-creatable through the supervisor as Kubernetes objects:

# sn-trunk and sn-mgmt are ordinary Private Subnets; the interesting object:
apiVersion: crd.nsx.vmware.com/v1alpha1
kind: SubnetConnectionBindingMap
metadata: {name: bm-mgmt}
spec:
  subnetName: sn-mgmt          # the map is a child of the VLAN subnet...
  targetSubnetName: sn-trunk   # ...and points AT the trunk
  vlanTrafficTag: 1610

That direction is easy to invert, so it’s worth saying twice: the binding map belongs to the VLAN subnet and points at the trunk, not the other way round.

On the nested host, nothing exotic — plain VST, like physical:

Name                Virtual Switch  Active Clients  VLAN ID
------------------  --------------  --------------  -------
Management Network  vSwitch0                     1     1610
vMotion             vSwitch0                     1     1611
vSAN                vSwitch0                     1     1612

The same three VLANs as the nested host sees them.

And because there’s no DHCP in a VPC subnet, the nested-ESXi appliance gets its identity through OVF properties in the VM Service spec:

bootstrap:
  vAppConfig:
    properties:
      - {key: guestinfo.ipaddress, value: {value: "172.30.0.40"}}
      - {key: guestinfo.netmask,   value: {value: "255.255.255.224"}}
      - {key: guestinfo.gateway,   value: {value: "172.30.0.33"}}
      - {key: guestinfo.vlan,     value: {value: "1610"}}

Does it actually work? The receipts

Two nested hosts, vNICs on sn-trunk, three VLANs. From host one:

[root@esx01:~] vmkping -c2 172.30.0.41            # mgmt, VLAN 1610
3 packets transmitted, 3 packets received, 0% packet loss
[root@esx01:~] vmkping -I vmk1 -c3 172.30.0.71    # vMotion, VLAN 1611
3 packets transmitted, 3 packets received, 0% packet loss
[root@esx01:~] vmkping -I vmk2 -c3 172.30.0.101   # vSAN, VLAN 1612
3 packets transmitted, 3 packets received, 0% packet loss

The transcript that matters: fail the first NIC, and every VLAN keeps flowing on the second — captured live.

Two more results worth knowing before you design around this:

Untagged frames are dropped. I put a probe vmk on the untagged portgroup using the address NSX itself had allocated to the trunk port: 100% loss, empty ARP table, while tagged traffic flowed happily beside it. Every network your nested host uses needs a VLAN and a binding map — there is no untagged fallback.

Failover behaves like real hardware. With two vNICs on the trunk teamed active/active, esxcli network nic down -n vmnic0 moved every VLAN onto vmnic1 with zero loss — and the SSH session I was watching from never dropped. The vmk MAC migrating between trunk ports mid-flow is exactly the scenario that MAC-pinned standard ports would blackhole; the trunk carries it fine.

Why this matters outside the lab

Running whole vSphere environments inside a VPC turns the platform into something most customers never had: a way to stand up complete, isolated copies of infrastructure on demand, without a physical fabric change and without waiting for anyone. That’s what makes it commercially interesting:

• Training and certification labs where every learner gets a real vSphere environment, not a shared one.
• Reproducing a customer problem on a like-for-like copy instead of on the customer’s estate.
• Rehearsing upgrades and migrations end to end before the change window, then throwing the copy away.
• Vendor and feature evaluations with real behaviour, at zero risk to production.

This is the design Comms-care uses to give every consultant a dedicated environment, and the same pattern scales to a classroom or a proof-of-concept factory.

Rules learned

• A nested ESXi vNIC on a standard VPC subnet is dead on arrival: vmk0’s synthesised MAC loses to SpoofGuard, silently.
• Attach nested-host vNICs only to a trunk subnet; one binding map per VLAN; the map lives under the VLAN subnet and points at the trunk.
• No DHCP in VPC subnets — bootstrap addressing via guestinfo.* (appliances) or cloud-init (Linux). Static IP plans are a feature in a lab anyway.
• ESXi’s default TCP/IP stack has one gateway — set per-vmk override gateways (esxcli ... ipv4 set -g) so vMotion/vSAN carry their own subnet’s gateway.
• Recreating a VM reallocates its NSX addresses. Pin what you depend on.
• MTU: everything here ran at 1500. Raise the trunk and the nested vDS before you do vSAN at any real scale.

Next in this series: what happens when you want ten of these labs — with byte-identical IP plans, firewalled from each other by construction. That’s where NSX VPCs go from “workaround” to genuinely better than physical.

Lab environment; opinions my own. Everything above was captured from a live VCF 9.1 environment — output trimmed for length, never edited for outcome.

This is the story of the least helpful error message in my recent memory, what it actually means, and the one-line ordering rule that would have saved an afternoon. If you’re doing self-service NSX VPCs on VCF 9 with the vSphere Supervisor, you will hit this. Bookmark accordingly.

The setup

Tenant-created VPC (via the VCF Automation CCI API), a supervisor namespace pinned to it, and a couple of VirtualMachineService objects of type LoadBalancer to publish SSH and HTTPS for the workloads inside. Standard stuff — the exact pattern that works out of the box in the org’s default VPC.

The k8s side looked perfect:

$ kubectl get endpoints -n pod-a
NAME           ENDPOINTS                        AGE
esx01-access   172.30.0.40:443,172.30.0.40:22   6m36s
esx02-access   172.30.0.41:443,172.30.0.41:22   6m35s

Endpoints resolved. VIPs: nothing. The only clue, a recurring event:

Warning  FailedRealizeNSXResource  service/esx01-access
Generic error occurred during realizing network for Service

“Generic error.” Wonderful.

Digging: what NCP actually wants

The supervisor’s network container plugin (NCP) logs told the real story:

nsx_ujo.ncp.nsx.policy.lb_layer4_service Lb Service not Found for Namespace pod-a
NCP00270 Failed to process virtual ip for service ...: Lbs pod-a is not found
Encountered retryable error ... : Lbs pod-a is not found

NCP wants an NSX LBService in the namespace’s VPC. In the org’s default VPC, one exists — the platform created it when the VPC was born. In my self-service VPC? Nobody had created one. Fair enough — that’s actually documented behaviour once you know where to look: a fresh VPC needs a LoadBalancer object (and before that, a VPCAttachment to a connectivity profile with the service gateway enabled, or the LB creation itself fails with a much better error message).

So I created the attachment, then the LBService. NSX: Realized=True. Problem solved?

Warning  FailedRealizeNSXResource  service/esx01-access
Generic error occurred during realizing network for Service

No.

The actual bug-shaped behaviour: a snapshot, not a lookup

Here’s the part that costs you the afternoon. That “retryable error” retries the lookup in NCP’s cache — not the discovery. NCP snapshots the VPC’s LB inventory when the namespace is created. An LBService that appears afterwards is never discovered, no matter how long you wait:

• Recreating the k8s services: no effect.
• Tagging the LBService with the nsx-op/* ownership tags the working ones carry: no effect — the cache doesn’t re-read NSX.
• Restarting NCP would force a full resync — but supervisor system pods are protected; even Administrator@vsphere.local gets a Forbidden.
• Mutating the namespace to nudge a re-sync: also blocked, by the supervisor’s namespace validation webhook.

As a tenant, there is exactly one fix: delete and recreate the namespace, now that its VPC has an LB. Fifteen minutes of rebuild for want of one ordering rule.

And the control experiment proves the rule: a namespace created after its VPC already had an LBService got its VIPs assigned without any drama —

esx01-access   VIP=192.168.144.34   22 OPEN · 443 OPEN
esx02-access   VIP=192.168.144.35   22 OPEN · 443 OPEN

And once the ordering is right, this is what “working” looks like — the pod’s state a couple of minutes after a correctly-ordered deployment:

What the requester sees once the order is right.

The ordering rule

For every self-service VPC that will publish LoadBalancer services, create — in this order, before the namespace:

1. VPC                                   (vpc.nsx.vmware.com/v1alpha1)
2. VPCAttachment                         (connectivity profile w/ service gateway
                                          — LB creation errors without it)
3. LoadBalancer   {regionName, vpcName}  (the step everyone misses)
4. ...and only THEN the Supervisor Namespace

Encode it in whatever provisions your VPCs — a script, a pipeline, an operator. It’s four API calls and it turns a silent, undiagnosable <pending> into a platform that just works.

Why this matters outside the lab

Nobody buys a platform for its ordering rules — but this is exactly the kind of edge that decides whether self-service provisioning feels reliable or flaky to the people using it. In a customer deployment the answer isn’t a blog post; it’s that the provisioning automation already does the four steps in the right order, every time, so a tenant never sees a VIP stuck at <pending>. Knowing where the sharp edges are — because you’ve been cut by them in a lab — is most of what an experienced delivery partner is for.

Rules learned

• In a self-service NSX VPC, the LBService must predate the namespace. NCP discovers LBs at namespace-add and never again.
• FailedRealizeNSXResource: Generic error on a Service = go read the NCP logs; the real message (Lbs <ns> is not found, NCP00270) is there.
• VPCAttachment (service gateway) is the prerequisite for the LB itself — that one at least fails loudly.
• Retro-tagging NSX objects to look “owned” doesn’t help a cache that never re-reads. Recreating the namespace is the only tenant-level fix.
• While you’re at it: new namespaces also reject VM creation until image status.disks syncs (~1–3 minutes after content library attach). Build the wait into your automation and both sharp edges disappear.

Previously in this series: nested ESXi inside an NSX VPC. Next: three datacenters, one IP plan — identical isolated pods.

Lab environment; opinions my own. Output captured live, trimmed for length, never edited for outcome.

ssh root@192.168.144.30  ->  [root@esx01-a:~]  vmk0  172.30.0.40
ssh root@192.168.144.32  ->  [root@esx01-b:~]  vmk0  172.30.0.40
ssh root@192.168.144.34  ->  [root@esx01-c:~]  vmk0  172.30.0.40

Same IP. Same VLAN. Same gateway. Same MAC address, as it turns out. And none of them can reach any of the others. This is the post where NSX VPCs stop being a workaround for nested labs and become genuinely better than the physical alternative.

Why identical addressing matters

If you’ve ever built training pods, cert-study labs, or per-team reproduction environments, you know the pain: every copy needs a unique address plan, so every runbook, every screenshot, every “type this exact command” has to be parameterised per pod. Students in seat 7 see different numbers from the slides. Reproductions drift from the original.

The fix is obvious and normally impossible: give every pod the same addresses. On a physical fabric that means VRFs, per-pod NAT, and a network team that stops answering your emails. In an NSX VPC it’s the default behaviour.

The mechanism: overlapping privateIPs

Each pod gets its own VPC, and every VPC declares the same private range:

apiVersion: vpc.nsx.vmware.com/v1alpha1
kind: VPC
metadata: {name: nested-vpc-a}      # then -b, then -c
spec:
  privateIPs: ["172.30.0.0/16"]      # identical in all three

A Private subnet is never advertised beyond its VPC, so NSX has no objection to three VPCs carving up the same /16. The pods aren’t “firewalled from each other” — there is simply no route between them. Isolation by construction, not by policy.

NSX’s subnet view filtered to sn-mgmt: four rows, four VPCs, one CIDR.

The trick: deterministic realization

Identical ranges aren’t enough — I want identical subnets, so the management gateway is .33 and the hosts are .40/.41 in every pod. NSX allocates subnets from privateIPs in creation order, and a fresh VPC allocates deterministically. So the topology is applied in a fixed order — trunk, mgmt, vMotion, vSAN — and every pod realizes the same map:

In the catalog blueprint that order is enforced with dependsOn between the subnet resources — the one place a declarative tool needs to be told about sequence. Skip it and two pods can come out with mgmt and vMotion swapped, which works perfectly and confuses everyone.

The door: one VIP per host

Each pod is unreachable from outside by design, so each host gets a VirtualMachineService of type LoadBalancer publishing SSH and HTTPS. The VIPs come from the org’s external block, and they’re the only addresses that differ between pods:

Which is how the opening transcript works: three VIPs, three hosts, one inside address.

Proving the isolation

Claims are cheap. The test matrix, from a VM in a fourth VPC (the org default):

ping 172.30.0.140 (own VPC)....... REACHABLE
ping 172.30.0.40  (pod space)..... unreachable
curl http://172.31.0.2/ (shared).. shared-svc repo01

Its own VPC’s 172.30.0.140: reachable. 172.30.0.40 — an address that exists in three other VPCs simultaneously: unreachable, because from here there is no such route. (The third line is the shared-services VPC, which is the next post.)

Inside each pod, east-west is normal: esx01 → esx02 vmkping passes on all three VLANs, in all three pods. And the detail I didn’t expect: the nested-ESXi appliance derives vmk0’s MAC deterministically from its config, so the three hosts share a MAC as well as an IP. Harmless — each VPC is its own L2 domain — but a nice demonstration of how complete the separation is.

What this replaces

Why this matters outside the lab

“Identical environments” sounds like a lab nicety. It’s actually one of the most requested things in enterprise IT, usually asked for in other words:

• Training at scale — every seat in the room sees the same addresses as the slides, so material is written once and never parameterised per pod.
• Per-engineer or per-team replicas of a reference environment, for development and testing that behaves exactly like the original.
• Regulatory or business-unit separation on shared infrastructure without VRF sprawl or a bespoke firewall estate — isolation is a property of the topology, which is the easiest kind to evidence to an auditor.
• Blue/green copies of an environment for change rehearsal, then cut-over or discard.

On a physical network each of these is a project. On VCF with NSX VPCs it’s a template.

Rules learned

• Overlapping privateIPs across VPCs is supported and intentional. Identical pods are a feature, not a hack.
• Fresh VPCs realize subnets deterministically in creation order — fix the order (dependsOn in a blueprint) and every pod gets the same map.
• Pods are unreachable from outside by construction; publish exactly what you mean to via LoadBalancer VIPs from the external block.
• Prove isolation from a different VPC, with a positive control (own VPC reachable) beside the negative.
• Expect duplicate MACs across pods from appliance images. It’s fine.

Previously: the LB that must exist first. Next: one WSUS for pods that can’t see each other.

Lab environment; opinions my own. Output captured live, trimmed for length, never edited for outcome.

No. There’s a third subnet access mode for exactly this, and the design it enables is hub-and-spoke with a very specific property: spokes reach the hub; the hub cannot reach the spokes; spokes never reach each other.

The three access modes

Everything in this series comes down to one field on a VPC subnet:

PrivateTGW subnets draw their addresses from a separate transit block (here 172.31.0.0/…), not from the VPC’s own privateIPs. That’s the key: the shared range can’t collide with the pods’ 172.30.0.0/16 because it comes from a different pool that all VPCs agree on.

The build

One more VPC, shared-svc, with the same ordering rules as any other (VPC → VPCAttachment → LoadBalancer → namespace), then a single subnet:

apiVersion: crd.nsx.vmware.com/v1alpha1
kind: Subnet
metadata: {name: sn-services}
spec: {accessMode: PrivateTGW, ipv4SubnetSize: 32}

It realized as 172.31.0.0/27, and a VM on it — svc-repo01, 172.31.0.2, serving HTTP — became the shared repo.

The test that matters is directional

Reachability to the service is the easy claim. From esx01 in each of the three pods (ESXi ships python3, so urllib is the test client):

pod-a esx01 -> http://172.31.0.2/   200  shared-svc repo01
pod-b esx01 -> http://172.31.0.2/   200  shared-svc repo01
pod-c esx01 -> http://172.31.0.2/   200  shared-svc repo01
default-vpc  -> http://172.31.0.2/   200  shared-svc repo01

Three pods with identical source addresses (172.30.0.40) all hit one service and all get answers. How does the reply find its way back to the right pod when three of them claim .40? Because pod traffic crosses the transit gateway SNAT’d to a per-VPC transit address. The service never sees 172.30.0.40; it sees three distinct transit IPs. Ambiguity never arises.

Now the other direction — from svc-repo01 back toward a pod:

svc-repo01 -> 172.30.0.40   unreachable
svc-repo01 -> 172.30.0.41   unreachable

Not “blocked” — unroutable. Pod subnets are Private, so they were never advertised to the transit gateway, and even if they had been, 172.30.0.40 would be ambiguous across three VPCs. The hub literally cannot initiate into a spoke. For a shared service that will one day be compromised, that’s the property you want.

What goes in the hub

Anything that’s consumed by pods and stateless about which pod is asking: WSUS/patch mirrors, OS and package repos, container registries, NTP, DNS forwarders, license servers. Domain controllers work too, with the usual caveat that identical hostnames across pods need per-pod domains or a naming scheme.

What does not go in the hub: anything that needs to reach into a pod (monitoring pollers, backup agents pulling, jump hosts). Those either live in the pod, or the pod publishes a LoadBalancer VIP for them — the deliberate door from part 0.

Tightening further

The transit gateway gives you reachability; policy gives you precision. A VPCGatewayFirewallPolicy on shared-svc can restrict inbound to tcp/80,443 from the transit range and nothing else, so the repo is a repo and not a foothold. I left it open for the test; you shouldn’t.

Why this matters outside the lab

This is the pattern that makes isolated tenants affordable. Without it, every isolated environment needs its own patch server, repository, DNS and directory — cost and drift that quietly kill the idea. With it, a customer runs one set of shared services for dozens of tenants, keeps them patched in one place, and can still show a security reviewer that the shared service has no path back into any tenant.

The same hub serves well beyond patching: central logging and monitoring collectors, licence servers, artifact registries, build agents — anything tenants consume but shouldn’t be able to be reached by.

Rules learned

• PrivateTGW is the shared-services mode: addresses from the transit block, advertised to every attached VPC, no collision with pod space.
• Access is one-way by construction: pods → service works (SNAT’d per VPC), service → pod has no route. Test both directions and write down both results.
• Identical pod addressing and shared services coexist because of the SNAT — the hub sees per-VPC transit addresses, never the overlapping private ones.
• Same ordering rules apply to the hub VPC (VPC → attachment → LB → namespace → subnets → wait for image sync → workloads).
• Add a gateway firewall policy on the hub. Reachability is not authorisation.

Previously: three datacenters, one IP plan. Next: the whole pod as a single catalog item.

Lab environment; opinions my own. Output captured live, trimmed for length, never edited for outcome.

All Apps in one paragraph

VCF Automation 9.1 has two provisioning models side by side. VM Apps is the classic Aria Automation path — cloud templates through an IaaS engine that drives vCenter. All Apps is the supervisor-native path: the blueprint composes Kubernetes objects (a Supervisor Namespace, VM Service VMs, NSX subnets, VKS clusters) and the vSphere Supervisor’s controllers reconcile them. A blueprint is formatVersion: 2; its resources are CCI.Supervisor.Namespace and CCI.Supervisor.Resource — the latter is literally “here’s a manifest, apply it in that namespace.”

That makes the blueprint a composition of the manifests from the earlier posts, with two additions: inputs, and dependsOn.

The blueprint, section by section

Inputs — the form

inputs:
  podName:  {type: string, default: nested-pod, pattern: '^[a-z0-9]([-a-z0-9]*[a-z0-9])?$'}
  vpcName:  {type: string, description: Must exist and be Realized before deploying.}
  esxOva:   {type: string, default: vmi-61bb062ddfc506b79}   # Nested ESXi 9.1 appliance
  isoImage: {type: string, default: vmi-39f562e2ae9e9c501}   # the ISO to attach
  vmClass:  {type: string, default: best-effort-large, enum: [best-effort-large, best-effort-xlarge, best-effort-2xlarge]}

The namespace — with libraries attached

namespace:
  type: CCI.Supervisor.Namespace
  properties:
    generateName: ${input.podName}-        # NOT name — new namespaces get a suffix
    className: large
    regionName: f06
    vpcName: ${input.vpcName}              # pins the namespace to the pod's VPC
    storageClasses: [{name: vSAN Default Storage Policy, limit: 400000Mi}]
    zones: [{name: domain-c9, cpuLimit: 40000M, memoryLimit: 64000Mi, ...}]
    contentSources:
      - {name: ISO, type: ContentLibrary}
      - {name: f06-vks-lib01, type: ContentLibrary}

contentSources is the line that closes the gap a lot of first attempts hit: a VCFA-created namespace has no content library, so there are no VirtualMachineImages and nothing can be deployed. Declaring the libraries here attaches them at creation.

The topology — ordered on purpose

snTrunk:   {type: CCI.Supervisor.Resource, properties: {context: ${resource.namespace.id}, manifest: <Subnet sn-trunk>}}
snMgmt:    {dependsOn: [snTrunk], ...  manifest: <Subnet sn-mgmt>}
snVmotion: {dependsOn: [snMgmt],  ...  manifest: <Subnet sn-vmotion>}
bmMgmt:    {... manifest: <SubnetConnectionBindingMap sn-mgmt -> sn-trunk, vlan 1610>}
bmVmotion: {... manifest: <SubnetConnectionBindingMap sn-vmotion -> sn-trunk, vlan 1611>}

The dependsOn chain is the whole reason every pod has identical CIDRs: fresh VPCs realize subnets in creation order, and the blueprint fixes that order.

The hosts — dual-NIC, ISO attached, bootstrapped by OVF

esx01:
  type: CCI.Supervisor.Resource
  dependsOn: [bmMgmt]                    # no point booting before VLAN 1610 exists
  properties:
    manifest:
      kind: VirtualMachine                # vmoperator.vmware.com/v1alpha5
      spec:
        hardware:
          cdrom: [ ... the ISO, declared, connected ... ]
        network:
          interfaces: [ eth0 -> sn-trunk, eth1 -> sn-trunk ]
        bootstrap:
          vAppConfig: [ guestinfo.hostname / ipaddress / vlan / ... ]
  wait:
    fields: [{path: status.powerState, value: PoweredOn}]

(Abridged — the full resource carries the image references, VM class, guest ID and the complete guestinfo set.)

Two vNICs, both on the trunk — the nested equivalent of a VCF host’s two pNICs. The ISO rides along as a declarative CD-ROM. And the wait block makes the deployment’s completion mean something: the request doesn’t finish until the host is powered on.

The doors — one VIP per host

A VirtualMachineService of type LoadBalancer per host, selecting it by label and publishing 22 and 443, and a blueprint output that reads the VIP back out of the service’s status:

outputs:
  esx01Ssh: {value: "ssh root@${resource.esx01Access.object.status.loadBalancer.ingress[0].ip}"}

The outputs surface in the deployment view — the requester gets the SSH command, not a scavenger hunt.

The request flow, end to end.

What the blueprint cannot express (yet)

Three cluster-scoped objects have no blueprint resource type, and they must exist before the request — in this order:

1. VPC — privateIPs: 172.30.0.0/16, same in every pod
2. VPCAttachment — connectivity profile with the service gateway; the LB creation fails loudly without it
3. LoadBalancer — silently, permanently required before the namespace

Today that’s a short script or a runbook step per pod. The honest framing: the blueprint is the pod; the VPC is the tenancy, and tenancy is still created one layer up. I’d expect that layer to become blueprintable; until then, keep the three calls next to the blueprint in version control.

Publishing: one version at a time

Blueprint → BlueprintVersion → release. Validation happens at version time, not create time, and the result lives in status.validationMessages rather than the HTTP code — a 200 with ContentValid: False is a thing. And only one version can be published: unrelease 1.0.0 before releasing 1.1.0, or you get a 409. (The full list of sharp edges is its own post.)

Why this matters outside the lab

This is where platform engineering turns into a service. The difference between “we can build you an environment” and “request one from the catalog” is the difference between days and minutes — and between a bespoke build and one that is consistent, quota-controlled and recorded every time. For an organisation that means:

• Time-to-environment measured in minutes, requested by the people who need it, without a queue.
• Consistency by construction — every environment comes from the same definition, so support, training material and runbooks all match.
• Governance built in — quotas, ownership, history and clean teardown are properties of the deployment record, not a spreadsheet.

The nested-ESXi pod is one catalog item. The same approach delivers any environment shape: application stacks for developers, sandboxes for a proof of concept, demo kits for a sales team, isolated builds for a partner.

Rules learned

• All Apps blueprints are compositions of manifests: CCI.Supervisor.Namespace plus CCI.Supervisor.Resource per object. If it works with kubectl, it works in a blueprint.
• generateName, not name, for the namespace; contentSources to attach libraries at creation; zones/storageClasses flat, not wrapped.
• dependsOn is how you get deterministic CIDRs — order the subnets.
• wait.fields turns “request complete” into “host is powered on”.
• VPC / VPCAttachment / LoadBalancer are prerequisites outside the blueprint, in that order, before every request.
• One published version per blueprint; validation in status, not the HTTP response.

Previously: shared services for isolated tenants. This closes the Pod Papers’ core arc — the companion posts on dual-NIC, no-DHCP bootstrap and blueprint gotchas fill in the details.

Lab environment; opinions my own. Blueprint nested-esxi-pod 1.1.0 is live in the lab catalog; YAML above trimmed for length.

Fair question, and the honest answer has two halves. In a nested lab the physical redundancy is provided by the outer host — its vDS, its NSX uplinks — and a second vNIC on the nested VM adds precisely none. But VCF doesn’t know it’s nested. Bringup’s host validation and the vDS uplink teaming it configures expect two vmnics, and a host with one gets flagged. So the nested hosts get two vNICs, both on the trunk subnet, and the question becomes: does failover between them actually work inside a VPC?

The setup

Both vNICs attach to the same sn-trunk subnet — the trunk from part 1 — and ESXi sees them as two 10G vmnics:

vSwitch0 teams them active/active with the default originating-port-ID policy; every portgroup (Management 1610, vMotion 1611, vSAN 1612) inherits it. Nothing you wouldn’t do on metal.

The test: pull a NIC while watching from inside

The interesting bit isn’t whether pings continue — it’s which session I’m watching from. I’m SSH’d to esx01 through its public VIP, which means my session traverses the NSX LB → the VPC → the trunk port → whichever vmnic happens to carry vmk0. If failover breaks anything, it breaks the terminal I’m typing in.

[root@esx01-a:~] esxcli network nic list
Name    ...  Admin Status  Link Status  Speed  MAC Address
vmnic0  ...  Up            Up           10000  04:50:56:00:5c:03
vmnic1  ...  Up            Up           10000  04:50:56:00:68:00

[root@esx01-a:~] esxcli network nic down -n vmnic0    # FAIL THE FIRST NIC
vmnic0  ...  Down          Down             0  04:50:56:00:5c:03
vmnic1  ...  Up            Up           10000  04:50:56:00:68:00

[root@esx01-a:~] vmkping -I vmk1 172.30.0.71          # vMotion VLAN, now over vmnic1
3 packets transmitted, 3 packets received, 0% packet loss

[root@esx01-a:~] vmkping -I vmk2 172.30.0.101         # vSAN VLAN, now over vmnic1
3 packets transmitted, 3 packets received, 0% packet loss

[root@esx01-a:~] esxcli network nic up -n vmnic0      # restore
# session never dropped.

Every VLAN moved to vmnic1. Zero loss on vMotion and vSAN. And the management session — the one most likely to notice — never blinked.

Why this is a real result, not a party trick

Think about what just happened at the NSX layer. vmk0’s MAC — a MAC ESXi synthesised, not the vNIC’s — was being learned on trunk port A. When vmnic0 went down, the same MAC appeared on trunk port B mid-flow, with an established TCP session riding on it.

On a standard VPC subnet port that is exactly the scenario SpoofGuard exists to stop: the port’s address bindings pin one MAC, and a frame from a different MAC — or the same MAC arriving on a different port — is dropped. Part 1 showed that killing the host on a standard subnet before it ever spoke. This test shows the trunk subnet tolerating the live migration of a foreign MAC between two of its ports, which is the property nested vSphere (and anything else with a vSwitch inside a VM) fundamentally needs.

So the second vNIC buys three things, none of them physical redundancy:

1. Bringup and vLCM stop complaining about a single-uplink host.
2. The teaming policy you’ll configure in production gets exercised — uplink failover, active/standby for vSAN, whatever you’re rehearsing.
3. A live proof that the trunk carries MAC mobility, which is the real assurance that the design isn’t relying on a quiet network.

What it does not buy, and how to say so

If someone asks “is this host redundant?”, the answer is “the nested host believes it is; actual redundancy lives one layer down.” In a training pod that’s the correct and useful answer — students configure and test failover exactly as they would on metal, and the outer platform does the real work. In a reproduction lab for a customer NIC-teaming issue, it’s usually enough too: most teaming bugs are in ESXi’s policy handling, not in the copper.

Where it’s genuinely insufficient: anything about physical link behaviour — LACP negotiation, LLDP, flapping, MTU mismatch on one uplink. The virtual fabric never fails asymmetrically, so it can’t reproduce those.

Why this matters outside the lab

The practical value here is knowing what a nested environment can and can’t prove — which is what lets you decide when a virtual lab is enough and when it isn’t. For training, upgrade rehearsals, configuration and policy testing and the vast majority of “how does it behave when…” questions, nested is enough and dramatically cheaper. For physical link behaviour — LACP, optics, asymmetric faults — you still want metal. Being able to make that call confidently is worth more than the test itself.

Rules learned

• Give nested VCF hosts two vNICs on the same trunk subnet. Bringup, vLCM and vDS teaming expect ≥ 2 vmnics; humouring them costs nothing.
• Test failover from a session that depends on it (SSH via the VIP). Pings passing while your terminal dies is not success.
• The trunk subnet tolerates a vmk MAC moving between ports mid-flow — that’s the property standard subnets lack and nested vSphere needs.
• Be precise in the write-up: nested dual-NIC gives policy realism, not physical redundancy. Physical link faults can’t be reproduced here.
• Rebuilds re-run the vmk config: the appliance creates vmk0 only; vmk1/vmk2, their VLANs and override gateways are applied post-boot.

Companion to nested ESXi inside an NSX VPC.

Lab environment; opinions my own. Output captured live, trimmed for length, never edited for outcome.

This isn’t a gap. It’s the model: NSX allocates the address at the port and pins it there with address bindings; the guest has to be told what it was given. The VM Service does that telling through bootstrap providers, and once you know the three of them, static addressing stops being a chore and starts being a feature.

Three providers, three guest types

All three are typed fields on the VirtualMachine object, not bolt-on customisation specs. The network side is already known to the platform — the VM Service knows which subnet each interface landed on and what NSX allocated — so for cloud-init and sysprep the addressing is injected for you. Appliances are the exception, because each one has its own idea of which properties it wants.

Appliances: vAppConfig

The nested-ESXi appliance reads guestinfo.* OVF properties. In the VM Service spec that’s:

bootstrap:
  vAppConfig:
    properties:
      - {key: guestinfo.hostname,  value: {value: "esx01.pod-a.res.lab"}}
      - {key: guestinfo.ipaddress, value: {value: "172.30.0.40"}}
      - {key: guestinfo.netmask,   value: {value: "255.255.255.224"}}
      - {key: guestinfo.gateway,   value: {value: "172.30.0.33"}}
      - {key: guestinfo.vlan,      value: {value: "1610"}}
      - {key: guestinfo.dns,       value: {value: "10.20.52.1"}}
      - {key: guestinfo.ssh,       value: {value: "True"}}

The one that trips people: the address you give must be the one NSX allocated to the port. In a standard subnet that’s enforced by SpoofGuard; on a trunk subnet the VLAN subnets have their own allocations and you’re choosing addresses within them. Either way, pick from the realized range — and remember recreating a VM reallocates its addresses, so the fixed .40/.41 in a deterministic pod is a design choice, not luck.

Linux: cloud-init

A Secret holding user-data, referenced from the VM:

bootstrap:
  cloudInit:
    cloudConfig:
      users: [ ... a local user with a key ... ]
      write_files:
        - path: /var/www/html/index.html
          content: "shared-svc repo01\n"
      runcmd:
        - [systemctl, enable, --now, nginx]

Networking arrives via the platform’s own network-config; you don’t write it. The svc-repo01 VM from the shared-services post was exactly this — write_files + runcmd, web page verified from three pods.

Windows: sysprep

bootstrap:
  sysprep:
    sysprep:
      guiUnattended: {autoLogon: true, autoLogonCount: 1, timeZone: 85}
      identification: {joinWorkgroup: WORKGROUP}
      userData: {fullName: Lab, orgName: Lab, computerName: {name: win01}}

Or rawSysprep with an unattend XML in a Secret if you already have one. The ISO from the blueprint post can ride along as a declarative hardware.cdrom — handy for tools and agents on first boot.

The ESXi footnote: one gateway, many vmks

Once the appliance is up and you add vMotion and vSAN vmks on their own subnets, you hit a detail that makes the Host Client look wrong:

ESXi’s default TCP/IP stack has one default gateway — vmk0’s .33 — and the UI repeats it on every vmk row. Same-subnet vMotion never uses a gateway so nothing breaks, but each NSX subnet does have its own gateway, and cross-subnet traffic from vmk1/vmk2 would take the wrong exit. Set per-vmk override gateways:

esxcli network ip interface ipv4 set -i vmk1 -t static -I 172.30.0.70  -N 255.255.255.224 -g 172.30.0.65
esxcli network ip interface ipv4 set -i vmk2 -t static -I 172.30.0.100 -N 255.255.255.224 -g 172.30.0.97

Now the display is truthful and the routing is correct. (The full-realism alternative is a dedicated vmotion netstack for vmk1; I kept the default stack so the service tags stay visible in the Host Client.)

Why this matters outside the lab

For the business, “no DHCP” translates into something security and operations teams both want: predictable addressing. Every environment has a known address plan, firewall rules can be written once, and nothing turns up on the network with an address nobody expected. Bootstrap providers deliver the second benefit — images stay generic and configuration is injected at deploy time, so there are fewer golden images to maintain and far less drift between environments.

Rules learned

• No DHCP in VPC subnets — by design. NSX allocates at the port; the guest is told via a bootstrap provider.
• cloudInit (Linux), sysprep (Windows), vAppConfig (appliances) — typed fields on the VM, not customisation specs.
• Appliance addresses must match the realized subnet; fix the order of subnet creation if you want fixed addresses across pods.
• ESXi has one default gateway per stack. Set -g per vmk, or the Host Client lies to you and cross-subnet traffic exits wrong.
• A static IP plan is a feature in a lab: it’s what makes screenshots, runbooks and pods identical.

Companion to nested ESXi inside an NSX VPC.

Lab environment; opinions my own.