That’s how my first attempt at running nested ESXi inside an NSX VPC ended. The failure is nasty precisely because nothing looks wrong. If you’re building nested vSphere labs on VCF 9 with VPC networking, this post is the map of the minefield. It’s also the design that gets you across it, verified live.

The setup

VCF 9.1, with the vSphere Supervisor on NSX VPC networking. The goal was to deploy nested ESXi hosts as ordinary VM Service VMs inside a tenant’s VPC. No physical fabric changes, no provider tickets, no special treatment. It’s the kind of thing you want for training pods, cert-study labs, or reproducing customer issues.

Nested ESXi needs what physical ESXi needs: a management network, vMotion and vSAN. Traditionally, those are VLANs trunked to every host. But a VPC is an overlay world, and there are no VLANs to trunk. So what happens if you just attach the nested host’s vNIC to a normal VPC subnet?

Failure #1: the silent blackhole

Here’s the trap. A standard VPC subnet port gets address bindings. NSX pins the exact IP and MAC it allocated to that vNIC, and SpoofGuard drops everything else.

ESXi’s vmk0 doesn’t use the vNIC’s MAC. It makes up its own:

vmk0
   MAC Address: 00:50:ac:1e:00:8c     <- NOT the vNIC MAC (04:50:56:...)

So every frame the management interface sends carries a MAC the port doesn’t own. NSX drops all of it (ARP, ping, everything), while the host itself boots green and reports healthy. There’s no error anywhere, which is somehow worse than a bad one. You just can’t reach it, ever.

(There’s a second trap stacked on top. Our VPC subnets run with DHCP deactivated, so the appliance also sits at “waiting for DHCP” unless you inject static addressing through OVF guestinfo.* properties. NSX can give a VPC subnet a DHCP server or relay (Add a Subnet to a VPC); ours had neither. More on that below.)

The design that works: a trunk subnet + binding maps

The fix isn’t a hack. It’s a first-class NSX VPC construct that’s barely documented in the wild: SubnetConnectionBindingMap.

The idea:

1. Create one ordinary VPC subnet to act as a trunk (sn-trunk). The nested host’s vNICs attach only here.
2. Create a normal VPC subnet for each traditional network: sn-mgmt, sn-vmotion and sn-vsan.
3. Bind each of those to the trunk with a binding map carrying a VLAN tag. The nested host’s vSwitch tags frames exactly as it would on metal. The binding map strips the tag and delivers the frame into the right subnet.

It’s pure L2 demultiplexing. One vNIC carries N VLANs, and the VPC never routes on a tag. The physical fabric never sees any of it, because the 802.1Q header rides inside the Geneve overlay.

A tenant can create all of it through the supervisor, as Kubernetes objects:

# sn-trunk and sn-mgmt are ordinary Private Subnets; the interesting object:
apiVersion: crd.nsx.vmware.com/v1alpha1
kind: SubnetConnectionBindingMap
metadata: {name: bm-mgmt}
spec:
  subnetName: sn-mgmt          # the map is a child of the VLAN subnet...
  targetSubnetName: sn-trunk   # ...and points AT the trunk
  vlanTrafficTag: 1610

That direction is easy to get backwards, so it’s worth saying twice: the binding map belongs to the VLAN subnet and points at the trunk, not the other way round.

On the nested host, nothing exotic: plain virtual switch tagging (VST), just like physical. It never suspects a thing.

Name                Virtual Switch  Active Clients  VLAN ID
------------------  --------------  --------------  -------
Management Network  vSwitch0                     1     1610
vMotion             vSwitch0                     1     1611
vSAN                vSwitch0                     1     1612

The same three VLANs as the nested host sees them.

And because there’s no DHCP in our VPC subnets, the nested-ESXi appliance gets its identity through OVF properties in the VM Service spec:

bootstrap:
  vAppConfig:
    properties:
      - {key: guestinfo.ipaddress, value: {value: "172.30.0.40"}}
      - {key: guestinfo.netmask,   value: {value: "255.255.255.224"}}
      - {key: guestinfo.gateway,   value: {value: "172.30.0.33"}}
      - {key: guestinfo.vlan,     value: {value: "1610"}}

Does it actually work? The receipts

Two nested hosts, vNICs on sn-trunk, three VLANs. From host one:

[root@esx01:~] vmkping -c2 172.30.0.41            # mgmt, VLAN 1610
3 packets transmitted, 3 packets received, 0% packet loss
[root@esx01:~] vmkping -I vmk1 -c3 172.30.0.71    # vMotion, VLAN 1611
3 packets transmitted, 3 packets received, 0% packet loss
[root@esx01:~] vmkping -I vmk2 -c3 172.30.0.101   # vSAN, VLAN 1612
3 packets transmitted, 3 packets received, 0% packet loss

The transcript that matters: fail the first NIC, and every VLAN keeps flowing on the second — captured live.

Two more results are worth knowing before you design around this.

Untagged frames are dropped. I put a probe vmk on the untagged port group, using the address NSX itself had allocated to the trunk port. The result was 100% loss and an empty ARP table, while tagged traffic flowed happily beside it, as if to make a point. Every network your nested host uses needs a VLAN and a binding map. There is no untagged fallback.

Failover behaves like real hardware. With two vNICs on the trunk, teamed active/active, esxcli network nic down -n vmnic0 moved every VLAN onto vmnic1 with zero loss. The SSH session I was watching from never dropped. A vmk MAC moving between trunk ports mid-flow is exactly what MAC-pinned standard ports would blackhole. The trunk carries it fine.

Why this matters outside the lab

Running whole vSphere environments inside a VPC turns the platform into something most customers never had. It’s a way to stand up complete, isolated copies of infrastructure on demand, with no physical fabric change and no waiting for anyone. That’s what makes it commercially interesting:

• Training and certification labs where every learner gets a real vSphere environment, not a shared one.
• Reproducing a customer problem on a like-for-like copy, rather than on their own estate.
• Rehearsing upgrades and migrations end to end before the change window, then throwing the copy away.
• Vendor and feature evaluations with real behaviour, at zero risk to production.

This is the design Comms-care uses to give every consultant a dedicated environment. The same pattern scales to a classroom or a proof-of-concept factory.

Rules learned

• A nested ESXi vNIC on a standard VPC subnet is dead on arrival: vmk0’s made-up MAC loses to SpoofGuard, silently.
• Attach nested-host vNICs only to a trunk subnet, with one binding map per VLAN. The map lives under the VLAN subnet and points at the trunk.
• No DHCP in our VPC subnets, so bootstrap addressing goes through guestinfo.* (appliances) or cloud-init (Linux). Static IP plans are a feature in a lab anyway.
• ESXi’s default TCP/IP stack has one gateway. Set per-vmk override gateways (esxcli ... ipv4 set -g) so vMotion and vSAN use their own subnet’s gateway.
• Recreating a VM reallocates its NSX addresses. Pin what you depend on.
• MTU: everything here ran at 1500. Raise the trunk and the nested vDS before you do vSAN at any real scale.

Broadcom documentation

• Create a SubnetConnectionBindingMap CR on the Supervisor: the binding map CR and its vlanTrafficTag, shown for VKS egress subnets.
• Creating a Child Segment: the NSX mechanism underneath: a binding map on the child segment, pointing at its parent with a VLAN ID.
• Understanding SpoofGuard Segment Profile: port address bindings, and traffic dropped when its MAC or IP does not match them.
• Add a VPC Service Profile: the DHCP settings and segment profiles, SpoofGuard included, that a VPC’s subnets inherit.
• Deploy VMs with Configurable OVF Properties in vSphere Supervisor: OVF properties set through the VM Service’s vAppConfig transport.
• Configure the VMkernel Adapter Gateway by Using esxcli Commands: a gateway per VMkernel adapter, set with esxcli.

Next in this series: what happens when you want ten of these labs, with byte-identical IP plans, firewalled from each other by construction. That’s where NSX VPCs go from “workaround” to genuinely better than physical.

Lab environment; opinions my own. Everything above was captured from a live VCF 9.1 environment — output trimmed for length, never edited for outcome.

This is the story of the least helpful error message in my recent memory, what it actually means, and the one-line ordering rule that would have saved an afternoon. If you’re doing self-service NSX VPCs on VCF 9 with the vSphere Supervisor, you will hit this. Bookmark accordingly.

The setup

The VPC was tenant-created, through VCF Automation’s Cloud Consumption Interface (CCI) API. A supervisor namespace was pinned to it, with a couple of VirtualMachineService objects of type LoadBalancer to publish SSH and HTTPS for the workloads inside. Standard stuff: the exact pattern that works out of the box in the org’s default VPC.

The Kubernetes side looked perfect:

$ kubectl get endpoints -n pod-a
NAME           ENDPOINTS                        AGE
esx01-access   172.30.0.40:443,172.30.0.40:22   6m36s
esx02-access   172.30.0.41:443,172.30.0.41:22   6m35s

Endpoints resolved. VIPs: nothing. The only clue was a recurring event:

Warning  FailedRealizeNSXResource  service/esx01-access
Generic error occurred during realizing network for Service

“Generic error.” Wonderful.

Digging: what NCP actually wants

The supervisor’s network container plugin (NCP) told the real story in its logs:

nsx_ujo.ncp.nsx.policy.lb_layer4_service Lb Service not Found for Namespace pod-a
NCP00270 Failed to process virtual ip for service ...: Lbs pod-a is not found
Encountered retryable error ... : Lbs pod-a is not found

NCP wants an NSX LBService in the namespace’s VPC. In the org’s default VPC, one exists, because the platform created it when the VPC was born. In my self-service VPC? Nobody had created one.

Fair enough. That’s actually documented behaviour, once you know where to look. A fresh VPC needs a LoadBalancer object. Before that, it needs a VPCAttachment to a connectivity profile with the service gateway enabled. Without the attachment, the load balancer creation itself fails, with a much better error message.

So I created the attachment, then the LBService. NSX: Realized=True. Problem solved?

Warning  FailedRealizeNSXResource  service/esx01-access
Generic error occurred during realizing network for Service

No.

The actual bug-shaped behaviour: a snapshot, not a lookup

Here’s the part that costs you the afternoon. That “retryable error” retries the lookup in NCP’s cache, not the discovery. NCP snapshots the VPC’s load balancer inventory when the namespace is created. An LBService that appears afterwards is never discovered, however long you wait:

• Recreating the Kubernetes services: no effect.
• Tagging the LBService with the nsx-op/* ownership tags the working ones carry: no effect, because the cache doesn’t re-read NSX.
• Restarting NCP would force a full resync, but supervisor system pods are protected. Even Administrator@vsphere.local gets a Forbidden, which is a humbling thing to read.
• Mutating the namespace to nudge a resync: also blocked, by the supervisor’s namespace validation webhook.

As a tenant, there is exactly one fix: delete and recreate the namespace, now that its VPC has a load balancer. Fifteen minutes of rebuild, for want of one ordering rule.

And the control experiment proves the rule. A namespace created after its VPC already had an LBService got its VIPs assigned without any drama:

esx01-access   VIP=192.168.144.34   22 OPEN · 443 OPEN
esx02-access   VIP=192.168.144.35   22 OPEN · 443 OPEN

Once the ordering is right, this is what “working” looks like. It’s the pod’s state a couple of minutes after a correctly ordered deployment:

What the requester sees once the order is right.

The ordering rule

For every self-service VPC that will publish LoadBalancer services, create these in order, before the namespace:

1. VPC                                   (vpc.nsx.vmware.com/v1alpha1)
2. VPCAttachment                         (connectivity profile w/ service gateway
                                          — LB creation errors without it)
3. LoadBalancer   {regionName, vpcName}  (the step everyone misses)
4. ...and only THEN the Supervisor Namespace

Encode it in whatever provisions your VPCs: a script, a pipeline, an operator. It’s four API calls, which is a good deal cheaper than an afternoon. And it turns a silent, undiagnosable <pending> into a platform that just works.

Why this matters outside the lab

Nobody buys a platform for its ordering rules. But this is exactly the kind of edge that decides whether self-service provisioning feels reliable or flaky to the people using it.

In a customer deployment, the answer isn’t a blog post. It’s provisioning automation that already does the four steps in the right order, every time, so a tenant never sees a VIP stuck at <pending>. Knowing where the sharp edges are, because you’ve been cut by them in a lab, is most of what an experienced delivery partner is for.

Rules learned

• In a self-service NSX VPC, the LBService must predate the namespace. NCP discovers load balancers when the namespace is added, and never again.
• FailedRealizeNSXResource: Generic error on a Service means: go and read the NCP logs. The real message (Lbs <ns> is not found, NCP00270) is there.
• VPCAttachment (service gateway) is the prerequisite for the load balancer itself. That one, at least, fails loudly.
• Retro-tagging NSX objects to look “owned” doesn’t help a cache that never re-reads. Recreating the namespace is the only tenant-level fix.
• While you’re at it: new namespaces also reject VM creation until image status.disks syncs (~1–3 minutes after content library attach). Build the wait into your automation and both sharp edges disappear.

Broadcom documentation

• Deploying Supervisor with VCF Networking with VPC: what NCP creates for a namespace given no VPC: a VPC with its load balancer and SNAT IP.
• Create vSphere Namespaces on VPCs without SNAT and Load Balancer: without the VPC’s load balancer, LoadBalancer services and VirtualMachineServices cannot be deployed.
• Add a VPC Connectivity Profile: the transit gateway, the service gateway and default outbound NAT.
• Create and Configure a vSphere Namespace on a Supervisor with NSX VPC: placing a new namespace in an existing VPC.
• Create a Virtual Private Cloud in VCF Automation: a tenant VPC, its connectivity profile and its load balancing setting.
• Sample Blueprints in VCF Automation: a VPC, its VPCAttachment and a namespace that depends on them, as blueprint resources.

Previously in this series: nested ESXi inside an NSX VPC. Next: three datacenters, one IP plan, with identical isolated pods.

Lab environment; opinions my own. Output captured live, trimmed for length, never edited for outcome.

Fair question, and like most fair questions it has an irritating two-part answer. In a nested lab, the outer host provides the physical redundancy: its vDS and its NSX uplinks. A second vNIC on the nested VM adds exactly none of that.

But VCF doesn’t know it’s nested. We expected bringup’s host validation, and the vDS uplink teaming it sets up, to want two vmnics. We never tried a host with one. For the record, Broadcom’s docs allow single-pNIC hosts, and the VCF Installer’s 9.1 known issue only bites single-pNIC hosts that use an NFS datastore.

So our nested hosts get two vNICs, both on the trunk subnet. Which leaves the fun question (for a given value of fun): does failover between them actually work inside a VPC?

The whole test on one page. The session I’m typing in rides the very path I break.

The setup

Both vNICs sit on the same sn-trunk subnet (the trunk from part 1), and ESXi sees two 10G vmnics:

vSwitch0 teams them active/active with the default originating-port-ID policy. Every port group inherits it: Management 1610, vMotion 1611 and vSAN 1612. Nothing you wouldn’t do on metal. Thrilling stuff, I know.

The test: pull a NIC while watching from inside

The interesting part isn’t whether pings carry on. It’s which session I’m watching from.

I’m SSH’d into esx01 through its public VIP. So my session runs through the NSX load balancer, the VPC, the trunk port and whichever vmnic happens to carry vmk0. If failover breaks anything, it breaks the terminal I’m typing in: the networking equivalent of sawing off the branch you’re sitting on. Low effort, high stakes: the best kind of test.

[root@esx01-a:~] esxcli network nic list
Name    ...  Admin Status  Link Status  Speed  MAC Address
vmnic0  ...  Up            Up           10000  04:50:56:00:5c:03
vmnic1  ...  Up            Up           10000  04:50:56:00:68:00

[root@esx01-a:~] esxcli network nic down -n vmnic0    # FAIL THE FIRST NIC
vmnic0  ...  Down          Down             0  04:50:56:00:5c:03
vmnic1  ...  Up            Up           10000  04:50:56:00:68:00

[root@esx01-a:~] vmkping -I vmk1 172.30.0.71          # vMotion VLAN, now over vmnic1
3 packets transmitted, 3 packets received, 0% packet loss

[root@esx01-a:~] vmkping -I vmk2 172.30.0.101         # vSAN VLAN, now over vmnic1
3 packets transmitted, 3 packets received, 0% packet loss

[root@esx01-a:~] esxcli network nic up -n vmnic0      # restore
# session never dropped.

Every VLAN moved to vmnic1. Zero loss on vMotion and vSAN. And the management session, the one most likely to notice, never blinked. Slightly disappointing, if I’m honest; I’d prepared a dramatic paragraph.

Why this is a real result, not a party trick

Think about what just happened at the NSX layer. vmk0 has a MAC of its own, one ESXi made up rather than the vNIC’s. NSX had learned it on trunk port A. When vmnic0 went down, the same MAC turned up on trunk port B, mid-flow, with a live TCP session riding on it.

On a standard VPC subnet port, that is exactly what SpoofGuard exists to stop. The port’s address bindings pin one MAC. A frame from a different MAC, or the same MAC arriving on a different port, gets dropped. Part 1 showed that rule silencing a host on a standard subnet before it ever spoke.

This test shows the trunk subnet calmly accepting a foreign MAC that moves between two of its ports. Nested vSphere needs exactly that property, and so does anything else with a vSwitch inside a VM.

So the second vNIC buys three things, none of them physical redundancy:

1. Bringup and vDS teaming get the two uplinks we expected them to want.
2. The teaming policy you’ll configure in production gets exercised: uplink failover, active/standby for vSAN, whatever you’re rehearsing.
3. A live proof that the trunk carries MAC mobility. That’s the real reassurance: the design isn’t relying on a quiet network.

One caution if the host is heading into a VCF Installer bringup. The 9.x installer’s validation wants exactly one physical NIC on vSwitch0, and stops with “has 2 Physical NICs connected to vSphere Standard Switch vSwitch0 (Expecting 1)” (KB 415469). Give such a host its second vNIC, but leave vmnic1 unclaimed until bringup takes it. The test above teams both on vSwitch0 because it tests the trunk, not a bringup.

What it does not buy, and how to say so

If someone asks “is this host redundant?”, the honest answer is “the nested host believes it is, bless it; the actual redundancy lives one layer down.”

In a training pod, that’s the right answer. Students configure and test failover exactly as they would on metal, while the outer platform does the real work. It’s usually enough to reproduce a NIC-teaming issue from the field too: most teaming bugs live in ESXi’s policy handling, not in the copper.

Where it genuinely falls short: anything about physical link behaviour. LACP negotiation, LLDP, a flapping link, an MTU mismatch on one uplink. The virtual fabric never fails lopsidedly, so it can’t reproduce any of those.

Why this matters outside the lab

The real value is knowing what a nested environment can and can’t prove, so you can tell when a virtual lab is enough. For training, upgrade rehearsals, configuration and policy testing, and most “how does it behave when…” questions, nested is enough and far cheaper. For physical link behaviour (LACP, optics, lopsided faults) you still want metal. Making that call with confidence is worth more than the test itself.

Rules learned

• Give nested VCF hosts two vNICs on the same trunk subnet. We expected bringup and vDS teaming to want at least two vmnics, and humouring them costs nothing. Before a VCF Installer bringup, leave vmnic1 off vSwitch0 (KB 415469).
• Test failover from a session that depends on it (SSH through the VIP). Pings passing while your terminal dies is not success, however much the change ticket would like it to be.
• The trunk subnet tolerates a vmk MAC moving between ports mid-flow. Standard subnets lack that property, and nested vSphere needs it.
• Be precise in the write-up: nested dual-NIC gives policy realism, not physical redundancy. Physical link faults can’t be reproduced here.
• Rebuilds re-run the vmk config: the appliance creates vmk0 only, so vmk1, vmk2, their VLANs and override gateways are applied after boot.

Broadcom documentation

• Configure NIC Teaming, Failover, and Load Balancing on a vSphere Standard Switch or Standard Port Group: the default originating-port policy, failover order, and port groups inheriting the switch’s policy.
• Understanding SpoofGuard Segment Profile: the address bindings a standard port enforces.
• Understanding MAC Discovery Segment Profile: MAC learning for nested hypervisors, with many MACs behind one vNIC.
• VCF Installer: the VCF 9.1 known issue where single-pNIC hosts fail NFS datastore validation, and the fix of two or more pNICs.
• Support for running ESXi as a nested virtualization solution: nested ESXi is not supported in production, and is encouraged for learning, training and testing.
• VCF 9.0 Installer validation fails at ESX Host Configuration (KB 415469): one physical NIC on vSwitch0 before deployment, the other left unclaimed.

Companion to nested ESXi inside an NSX VPC.

Lab environment; opinions my own. Output captured live, trimmed for length, never edited for outcome.

Drop a stock appliance onto one and it will boot, sit at “waiting for DHCP”, and wait politely until the heat death of the universe.

This isn’t a gap. It’s the model. NSX allocates the address at the port and pins it there with address bindings, and the guest has to be told what it was given. The VM Service does that telling through bootstrap providers. Once you know the three of them, static addressing stops being a chore and starts being a feature.

Three providers, three guest types

All three are typed fields on the VirtualMachine object, not bolt-on customisation specs. The platform already knows the network side: the VM Service knows which subnet each interface landed on, and what NSX allocated. So for cloud-init and sysprep, the addressing is injected for you. Appliances are the exception, because each one has its own idea of which properties it wants.

Appliances: vAppConfig

The nested-ESXi appliance reads guestinfo.* OVF properties. In the VM Service spec, that’s:

bootstrap:
  vAppConfig:
    properties:
      - {key: guestinfo.hostname,  value: {value: "esx01.pod-a.res.lab"}}
      - {key: guestinfo.ipaddress, value: {value: "172.30.0.40"}}
      - {key: guestinfo.netmask,   value: {value: "255.255.255.224"}}
      - {key: guestinfo.gateway,   value: {value: "172.30.0.33"}}
      - {key: guestinfo.vlan,      value: {value: "1610"}}
      - {key: guestinfo.dns,       value: {value: "10.20.52.1"}}
      - {key: guestinfo.ssh,       value: {value: "True"}}

The one that trips people up: the address you give must be the one NSX allocated to the port. In a standard subnet, SpoofGuard enforces that, and it doesn’t negotiate. On a trunk subnet, the VLAN subnets have their own allocations, and you’re choosing addresses within them.

Either way, pick from the realized range. And remember that recreating a VM reallocates its addresses. So the fixed .40/.41 in a deterministic pod is a design choice, not luck.

Linux: cloud-init

A Secret holding user-data, referenced from the VM:

bootstrap:
  cloudInit:
    cloudConfig:
      users: [ ... a local user with a key ... ]
      write_files:
        - path: /var/www/html/index.html
          content: "shared-svc repo01\n"
      runcmd:
        - [systemctl, enable, --now, nginx]

Networking arrives through the platform’s own network-config, so you don’t write it. Every line of YAML I don’t write is a line I can’t get wrong.

The svc-repo01 VM from the shared-services post was exactly this: write_files plus runcmd, with its web page verified from three pods.

Windows: sysprep

bootstrap:
  sysprep:
    sysprep:
      guiUnattended: {autoLogon: true, autoLogonCount: 1, timeZone: 85}
      identification: {joinWorkgroup: WORKGROUP}
      userData: {fullName: Lab, orgName: Lab, computerName: {name: win01}}

Or use rawSysprep with an unattend XML in a Secret, if you already have one. The ISO from the blueprint post can ride along as a declarative hardware.cdrom. That’s handy for tools and agents on first boot.

The ESXi footnote: one gateway, many vmks

Once the appliance is up and you add vMotion and vSAN vmks on their own subnets, you hit a detail that makes the Host Client look wrong:

ESXi’s default TCP/IP stack has one default gateway (vmk0’s .33), and the UI repeats it on every vmk row with total confidence. Same-subnet vMotion never uses a gateway, so nothing breaks. But each NSX subnet does have its own gateway, and cross-subnet traffic from vmk1 and vmk2 would take the wrong exit. Set per-vmk override gateways:

esxcli network ip interface ipv4 set -i vmk1 -t static -I 172.30.0.70  -N 255.255.255.224 -g 172.30.0.65
esxcli network ip interface ipv4 set -i vmk2 -t static -I 172.30.0.100 -N 255.255.255.224 -g 172.30.0.97

Now the display is truthful and the routing is correct. (The full-realism alternative is a dedicated vmotion netstack for vmk1. I kept the default stack so the service tags stay visible in the Host Client.)

Why this matters outside the lab

For the business, “no DHCP” translates into something security and operations teams both want: predictable addressing. Every environment has a known address plan, firewall rules can be written once, and nothing turns up on the network with an address nobody expected.

Bootstrap providers deliver the second benefit. Images stay generic and configuration is injected at deploy time, so there are fewer golden images to maintain and far less drift between environments.

Rules learned

• No DHCP in our VPC subnets: that’s the Supervisor’s default, not a VPC limit. NSX allocates at the port, and the guest is told through a bootstrap provider.
• cloudInit (Linux), sysprep (Windows) and vAppConfig (appliances) are typed fields on the VM, not customisation specs.
• Appliance addresses must match the realized subnet. Fix the order of subnet creation if you want fixed addresses across pods.
• ESXi has one default gateway per stack. Set -g per vmk, or the Host Client lies to you and cross-subnet traffic exits wrong.
• A static IP plan is a feature in a lab: it’s what makes screenshots, runbooks and pods identical.

Broadcom documentation

• Add a Subnet to a VPC: a subnet’s DHCP setting: none for static addresses, a DHCP server, or DHCP relay.
• Understanding SpoofGuard Segment Profile: the port address bindings SpoofGuard enforces.
• Provision a VM Using Self-Service: the four bootstrap methods, cloud-init, Sysprep, Linuxprep and vAppConfig, and static IP allocation.
• Deploy VMs with Configurable OVF Properties in vSphere Supervisor: OVF properties set through the VM Service’s vAppConfig transport.
• Configure the VMkernel Adapter Gateway by Using esxcli Commands: a gateway per VMkernel adapter, set with esxcli.

Companion to nested ESXi inside an NSX VPC.

Lab environment; opinions my own.