That’s how my first attempt at running nested ESXi inside an NSX VPC ended, and the failure mode is nasty precisely because nothing looks wrong. If you’re trying to build nested vSphere labs on VCF 9 with VPC networking, this post is the map of the minefield — and the design that gets you across it, verified live.

The setup

VCF 9.1, vSphere Supervisor with NSX VPC networking. The goal: deploy nested ESXi hosts as ordinary VM Service VMs inside a tenant’s VPC — no physical fabric changes, no provider tickets, no special treatment. The kind of thing you want for training pods, cert-study labs, or reproducing customer issues.

Nested ESXi needs what physical ESXi needs: a management network, vMotion, vSAN — traditionally VLANs trunked to every host. But a VPC is an overlay world. There are no VLANs to trunk. So what happens if you just attach the nested host’s vNIC to a normal VPC subnet?

Failure #1: the silent blackhole

Here’s the trap. A standard VPC subnet port gets address bindings: NSX pins the exact IP + MAC it allocated to that vNIC, and SpoofGuard drops everything else.

ESXi’s vmk0 doesn’t use the vNIC’s MAC. It synthesises its own:

vmk0
   MAC Address: 00:50:ac:1e:00:8c     <- NOT the vNIC MAC (04:50:56:...)

So every frame the management interface sends carries a MAC the port doesn’t own. NSX drops it all — ARP, ping, everything — while the host itself boots green and reports healthy. There is no error anywhere. You just can’t reach it, ever.

(There’s a second trap stacked on top: VPC subnets run with DHCP deactivated, so the appliance also sits at “waiting for DHCP” unless you inject static addressing via OVF guestinfo.* properties. More on that below.)

The design that works: a trunk subnet + binding maps

The fix isn’t a hack — it’s a first-class NSX VPC construct that’s barely documented in the wild: SubnetConnectionBindingMap.

The idea:

1. Create one ordinary VPC subnet to act as a trunk (sn-trunk). The nested host’s vNICs attach only here.
2. Create a normal VPC subnet per traditional network — sn-mgmt, sn-vmotion, sn-vsan.
3. Bind each of those to the trunk with a binding map carrying a VLAN tag. The nested host’s vSwitch tags frames exactly as it would on metal; the binding map strips the tag and delivers the frame into the right subnet.

Pure L2 demultiplexing. One vNIC carries N VLANs, the VPC never routes on a tag, and the physical fabric never sees any of it (the 802.1Q header rides inside the Geneve overlay).

All of it is tenant-creatable through the supervisor as Kubernetes objects:

# sn-trunk and sn-mgmt are ordinary Private Subnets; the interesting object:
apiVersion: crd.nsx.vmware.com/v1alpha1
kind: SubnetConnectionBindingMap
metadata: {name: bm-mgmt}
spec:
  subnetName: sn-mgmt          # the map is a child of the VLAN subnet...
  targetSubnetName: sn-trunk   # ...and points AT the trunk
  vlanTrafficTag: 1610

That direction is easy to invert, so it’s worth saying twice: the binding map belongs to the VLAN subnet and points at the trunk, not the other way round.

On the nested host, nothing exotic — plain VST, like physical:

Name                Virtual Switch  Active Clients  VLAN ID
------------------  --------------  --------------  -------
Management Network  vSwitch0                     1     1610
vMotion             vSwitch0                     1     1611
vSAN                vSwitch0                     1     1612

The same three VLANs as the nested host sees them.

And because there’s no DHCP in a VPC subnet, the nested-ESXi appliance gets its identity through OVF properties in the VM Service spec:

bootstrap:
  vAppConfig:
    properties:
      - {key: guestinfo.ipaddress, value: {value: "172.30.0.40"}}
      - {key: guestinfo.netmask,   value: {value: "255.255.255.224"}}
      - {key: guestinfo.gateway,   value: {value: "172.30.0.33"}}
      - {key: guestinfo.vlan,     value: {value: "1610"}}

Does it actually work? The receipts

Two nested hosts, vNICs on sn-trunk, three VLANs. From host one:

[root@esx01:~] vmkping -c2 172.30.0.41            # mgmt, VLAN 1610
3 packets transmitted, 3 packets received, 0% packet loss
[root@esx01:~] vmkping -I vmk1 -c3 172.30.0.71    # vMotion, VLAN 1611
3 packets transmitted, 3 packets received, 0% packet loss
[root@esx01:~] vmkping -I vmk2 -c3 172.30.0.101   # vSAN, VLAN 1612
3 packets transmitted, 3 packets received, 0% packet loss

The transcript that matters: fail the first NIC, and every VLAN keeps flowing on the second — captured live.

Two more results worth knowing before you design around this:

Untagged frames are dropped. I put a probe vmk on the untagged portgroup using the address NSX itself had allocated to the trunk port: 100% loss, empty ARP table, while tagged traffic flowed happily beside it. Every network your nested host uses needs a VLAN and a binding map — there is no untagged fallback.

Failover behaves like real hardware. With two vNICs on the trunk teamed active/active, esxcli network nic down -n vmnic0 moved every VLAN onto vmnic1 with zero loss — and the SSH session I was watching from never dropped. The vmk MAC migrating between trunk ports mid-flow is exactly the scenario that MAC-pinned standard ports would blackhole; the trunk carries it fine.

Why this matters outside the lab

Running whole vSphere environments inside a VPC turns the platform into something most customers never had: a way to stand up complete, isolated copies of infrastructure on demand, without a physical fabric change and without waiting for anyone. That’s what makes it commercially interesting:

• Training and certification labs where every learner gets a real vSphere environment, not a shared one.
• Reproducing a customer problem on a like-for-like copy instead of on the customer’s estate.
• Rehearsing upgrades and migrations end to end before the change window, then throwing the copy away.
• Vendor and feature evaluations with real behaviour, at zero risk to production.

This is the design Comms-care uses to give every consultant a dedicated environment, and the same pattern scales to a classroom or a proof-of-concept factory.

Rules learned

• A nested ESXi vNIC on a standard VPC subnet is dead on arrival: vmk0’s synthesised MAC loses to SpoofGuard, silently.
• Attach nested-host vNICs only to a trunk subnet; one binding map per VLAN; the map lives under the VLAN subnet and points at the trunk.
• No DHCP in VPC subnets — bootstrap addressing via guestinfo.* (appliances) or cloud-init (Linux). Static IP plans are a feature in a lab anyway.
• ESXi’s default TCP/IP stack has one gateway — set per-vmk override gateways (esxcli ... ipv4 set -g) so vMotion/vSAN carry their own subnet’s gateway.
• Recreating a VM reallocates its NSX addresses. Pin what you depend on.
• MTU: everything here ran at 1500. Raise the trunk and the nested vDS before you do vSAN at any real scale.

Next in this series: what happens when you want ten of these labs — with byte-identical IP plans, firewalled from each other by construction. That’s where NSX VPCs go from “workaround” to genuinely better than physical.

Lab environment; opinions my own. Everything above was captured from a live VCF 9.1 environment — output trimmed for length, never edited for outcome.

ssh root@192.168.144.30  ->  [root@esx01-a:~]  vmk0  172.30.0.40
ssh root@192.168.144.32  ->  [root@esx01-b:~]  vmk0  172.30.0.40
ssh root@192.168.144.34  ->  [root@esx01-c:~]  vmk0  172.30.0.40

Same IP. Same VLAN. Same gateway. Same MAC address, as it turns out. And none of them can reach any of the others. This is the post where NSX VPCs stop being a workaround for nested labs and become genuinely better than the physical alternative.

Why identical addressing matters

If you’ve ever built training pods, cert-study labs, or per-team reproduction environments, you know the pain: every copy needs a unique address plan, so every runbook, every screenshot, every “type this exact command” has to be parameterised per pod. Students in seat 7 see different numbers from the slides. Reproductions drift from the original.

The fix is obvious and normally impossible: give every pod the same addresses. On a physical fabric that means VRFs, per-pod NAT, and a network team that stops answering your emails. In an NSX VPC it’s the default behaviour.

The mechanism: overlapping privateIPs

Each pod gets its own VPC, and every VPC declares the same private range:

apiVersion: vpc.nsx.vmware.com/v1alpha1
kind: VPC
metadata: {name: nested-vpc-a}      # then -b, then -c
spec:
  privateIPs: ["172.30.0.0/16"]      # identical in all three

A Private subnet is never advertised beyond its VPC, so NSX has no objection to three VPCs carving up the same /16. The pods aren’t “firewalled from each other” — there is simply no route between them. Isolation by construction, not by policy.

NSX’s subnet view filtered to sn-mgmt: four rows, four VPCs, one CIDR.

The trick: deterministic realization

Identical ranges aren’t enough — I want identical subnets, so the management gateway is .33 and the hosts are .40/.41 in every pod. NSX allocates subnets from privateIPs in creation order, and a fresh VPC allocates deterministically. So the topology is applied in a fixed order — trunk, mgmt, vMotion, vSAN — and every pod realizes the same map:

In the catalog blueprint that order is enforced with dependsOn between the subnet resources — the one place a declarative tool needs to be told about sequence. Skip it and two pods can come out with mgmt and vMotion swapped, which works perfectly and confuses everyone.

The door: one VIP per host

Each pod is unreachable from outside by design, so each host gets a VirtualMachineService of type LoadBalancer publishing SSH and HTTPS. The VIPs come from the org’s external block, and they’re the only addresses that differ between pods:

Which is how the opening transcript works: three VIPs, three hosts, one inside address.

Proving the isolation

Claims are cheap. The test matrix, from a VM in a fourth VPC (the org default):

ping 172.30.0.140 (own VPC)....... REACHABLE
ping 172.30.0.40  (pod space)..... unreachable
curl http://172.31.0.2/ (shared).. shared-svc repo01

Its own VPC’s 172.30.0.140: reachable. 172.30.0.40 — an address that exists in three other VPCs simultaneously: unreachable, because from here there is no such route. (The third line is the shared-services VPC, which is the next post.)

Inside each pod, east-west is normal: esx01 → esx02 vmkping passes on all three VLANs, in all three pods. And the detail I didn’t expect: the nested-ESXi appliance derives vmk0’s MAC deterministically from its config, so the three hosts share a MAC as well as an IP. Harmless — each VPC is its own L2 domain — but a nice demonstration of how complete the separation is.

What this replaces

Why this matters outside the lab

“Identical environments” sounds like a lab nicety. It’s actually one of the most requested things in enterprise IT, usually asked for in other words:

• Training at scale — every seat in the room sees the same addresses as the slides, so material is written once and never parameterised per pod.
• Per-engineer or per-team replicas of a reference environment, for development and testing that behaves exactly like the original.
• Regulatory or business-unit separation on shared infrastructure without VRF sprawl or a bespoke firewall estate — isolation is a property of the topology, which is the easiest kind to evidence to an auditor.
• Blue/green copies of an environment for change rehearsal, then cut-over or discard.

On a physical network each of these is a project. On VCF with NSX VPCs it’s a template.

Rules learned

• Overlapping privateIPs across VPCs is supported and intentional. Identical pods are a feature, not a hack.
• Fresh VPCs realize subnets deterministically in creation order — fix the order (dependsOn in a blueprint) and every pod gets the same map.
• Pods are unreachable from outside by construction; publish exactly what you mean to via LoadBalancer VIPs from the external block.
• Prove isolation from a different VPC, with a positive control (own VPC reachable) beside the negative.
• Expect duplicate MACs across pods from appliance images. It’s fine.

Previously: the LB that must exist first. Next: one WSUS for pods that can’t see each other.

Lab environment; opinions my own. Output captured live, trimmed for length, never edited for outcome.

All Apps in one paragraph

VCF Automation 9.1 has two provisioning models side by side. VM Apps is the classic Aria Automation path — cloud templates through an IaaS engine that drives vCenter. All Apps is the supervisor-native path: the blueprint composes Kubernetes objects (a Supervisor Namespace, VM Service VMs, NSX subnets, VKS clusters) and the vSphere Supervisor’s controllers reconcile them. A blueprint is formatVersion: 2; its resources are CCI.Supervisor.Namespace and CCI.Supervisor.Resource — the latter is literally “here’s a manifest, apply it in that namespace.”

That makes the blueprint a composition of the manifests from the earlier posts, with two additions: inputs, and dependsOn.

The blueprint, section by section

Inputs — the form

inputs:
  podName:  {type: string, default: nested-pod, pattern: '^[a-z0-9]([-a-z0-9]*[a-z0-9])?$'}
  vpcName:  {type: string, description: Must exist and be Realized before deploying.}
  esxOva:   {type: string, default: vmi-61bb062ddfc506b79}   # Nested ESXi 9.1 appliance
  isoImage: {type: string, default: vmi-39f562e2ae9e9c501}   # the ISO to attach
  vmClass:  {type: string, default: best-effort-large, enum: [best-effort-large, best-effort-xlarge, best-effort-2xlarge]}

The namespace — with libraries attached

namespace:
  type: CCI.Supervisor.Namespace
  properties:
    generateName: ${input.podName}-        # NOT name — new namespaces get a suffix
    className: large
    regionName: f06
    vpcName: ${input.vpcName}              # pins the namespace to the pod's VPC
    storageClasses: [{name: vSAN Default Storage Policy, limit: 400000Mi}]
    zones: [{name: domain-c9, cpuLimit: 40000M, memoryLimit: 64000Mi, ...}]
    contentSources:
      - {name: ISO, type: ContentLibrary}
      - {name: f06-vks-lib01, type: ContentLibrary}

contentSources is the line that closes the gap a lot of first attempts hit: a VCFA-created namespace has no content library, so there are no VirtualMachineImages and nothing can be deployed. Declaring the libraries here attaches them at creation.

The topology — ordered on purpose

snTrunk:   {type: CCI.Supervisor.Resource, properties: {context: ${resource.namespace.id}, manifest: <Subnet sn-trunk>}}
snMgmt:    {dependsOn: [snTrunk], ...  manifest: <Subnet sn-mgmt>}
snVmotion: {dependsOn: [snMgmt],  ...  manifest: <Subnet sn-vmotion>}
bmMgmt:    {... manifest: <SubnetConnectionBindingMap sn-mgmt -> sn-trunk, vlan 1610>}
bmVmotion: {... manifest: <SubnetConnectionBindingMap sn-vmotion -> sn-trunk, vlan 1611>}

The dependsOn chain is the whole reason every pod has identical CIDRs: fresh VPCs realize subnets in creation order, and the blueprint fixes that order.

The hosts — dual-NIC, ISO attached, bootstrapped by OVF

esx01:
  type: CCI.Supervisor.Resource
  dependsOn: [bmMgmt]                    # no point booting before VLAN 1610 exists
  properties:
    manifest:
      kind: VirtualMachine                # vmoperator.vmware.com/v1alpha5
      spec:
        hardware:
          cdrom: [ ... the ISO, declared, connected ... ]
        network:
          interfaces: [ eth0 -> sn-trunk, eth1 -> sn-trunk ]
        bootstrap:
          vAppConfig: [ guestinfo.hostname / ipaddress / vlan / ... ]
  wait:
    fields: [{path: status.powerState, value: PoweredOn}]

(Abridged — the full resource carries the image references, VM class, guest ID and the complete guestinfo set.)

Two vNICs, both on the trunk — the nested equivalent of a VCF host’s two pNICs. The ISO rides along as a declarative CD-ROM. And the wait block makes the deployment’s completion mean something: the request doesn’t finish until the host is powered on.

The doors — one VIP per host

A VirtualMachineService of type LoadBalancer per host, selecting it by label and publishing 22 and 443, and a blueprint output that reads the VIP back out of the service’s status:

outputs:
  esx01Ssh: {value: "ssh root@${resource.esx01Access.object.status.loadBalancer.ingress[0].ip}"}

The outputs surface in the deployment view — the requester gets the SSH command, not a scavenger hunt.

The request flow, end to end.

What the blueprint cannot express (yet)

Three cluster-scoped objects have no blueprint resource type, and they must exist before the request — in this order:

1. VPC — privateIPs: 172.30.0.0/16, same in every pod
2. VPCAttachment — connectivity profile with the service gateway; the LB creation fails loudly without it
3. LoadBalancer — silently, permanently required before the namespace

Today that’s a short script or a runbook step per pod. The honest framing: the blueprint is the pod; the VPC is the tenancy, and tenancy is still created one layer up. I’d expect that layer to become blueprintable; until then, keep the three calls next to the blueprint in version control.

Publishing: one version at a time

Blueprint → BlueprintVersion → release. Validation happens at version time, not create time, and the result lives in status.validationMessages rather than the HTTP code — a 200 with ContentValid: False is a thing. And only one version can be published: unrelease 1.0.0 before releasing 1.1.0, or you get a 409. (The full list of sharp edges is its own post.)

Why this matters outside the lab

This is where platform engineering turns into a service. The difference between “we can build you an environment” and “request one from the catalog” is the difference between days and minutes — and between a bespoke build and one that is consistent, quota-controlled and recorded every time. For an organisation that means:

• Time-to-environment measured in minutes, requested by the people who need it, without a queue.
• Consistency by construction — every environment comes from the same definition, so support, training material and runbooks all match.
• Governance built in — quotas, ownership, history and clean teardown are properties of the deployment record, not a spreadsheet.

The nested-ESXi pod is one catalog item. The same approach delivers any environment shape: application stacks for developers, sandboxes for a proof of concept, demo kits for a sales team, isolated builds for a partner.

Rules learned

• All Apps blueprints are compositions of manifests: CCI.Supervisor.Namespace plus CCI.Supervisor.Resource per object. If it works with kubectl, it works in a blueprint.
• generateName, not name, for the namespace; contentSources to attach libraries at creation; zones/storageClasses flat, not wrapped.
• dependsOn is how you get deterministic CIDRs — order the subnets.
• wait.fields turns “request complete” into “host is powered on”.
• VPC / VPCAttachment / LoadBalancer are prerequisites outside the blueprint, in that order, before every request.
• One published version per blueprint; validation in status, not the HTTP response.

Previously: shared services for isolated tenants. This closes the Pod Papers’ core arc — the companion posts on dual-NIC, no-DHCP bootstrap and blueprint gotchas fill in the details.

Lab environment; opinions my own. Blueprint nested-esxi-pod 1.1.0 is live in the lab catalog; YAML above trimmed for length.

Fair question, and the honest answer has two halves. In a nested lab the physical redundancy is provided by the outer host — its vDS, its NSX uplinks — and a second vNIC on the nested VM adds precisely none. But VCF doesn’t know it’s nested. Bringup’s host validation and the vDS uplink teaming it configures expect two vmnics, and a host with one gets flagged. So the nested hosts get two vNICs, both on the trunk subnet, and the question becomes: does failover between them actually work inside a VPC?

The setup

Both vNICs attach to the same sn-trunk subnet — the trunk from part 1 — and ESXi sees them as two 10G vmnics:

vSwitch0 teams them active/active with the default originating-port-ID policy; every portgroup (Management 1610, vMotion 1611, vSAN 1612) inherits it. Nothing you wouldn’t do on metal.

The test: pull a NIC while watching from inside

The interesting bit isn’t whether pings continue — it’s which session I’m watching from. I’m SSH’d to esx01 through its public VIP, which means my session traverses the NSX LB → the VPC → the trunk port → whichever vmnic happens to carry vmk0. If failover breaks anything, it breaks the terminal I’m typing in.

[root@esx01-a:~] esxcli network nic list
Name    ...  Admin Status  Link Status  Speed  MAC Address
vmnic0  ...  Up            Up           10000  04:50:56:00:5c:03
vmnic1  ...  Up            Up           10000  04:50:56:00:68:00

[root@esx01-a:~] esxcli network nic down -n vmnic0    # FAIL THE FIRST NIC
vmnic0  ...  Down          Down             0  04:50:56:00:5c:03
vmnic1  ...  Up            Up           10000  04:50:56:00:68:00

[root@esx01-a:~] vmkping -I vmk1 172.30.0.71          # vMotion VLAN, now over vmnic1
3 packets transmitted, 3 packets received, 0% packet loss

[root@esx01-a:~] vmkping -I vmk2 172.30.0.101         # vSAN VLAN, now over vmnic1
3 packets transmitted, 3 packets received, 0% packet loss

[root@esx01-a:~] esxcli network nic up -n vmnic0      # restore
# session never dropped.

Every VLAN moved to vmnic1. Zero loss on vMotion and vSAN. And the management session — the one most likely to notice — never blinked.

Why this is a real result, not a party trick

Think about what just happened at the NSX layer. vmk0’s MAC — a MAC ESXi synthesised, not the vNIC’s — was being learned on trunk port A. When vmnic0 went down, the same MAC appeared on trunk port B mid-flow, with an established TCP session riding on it.

On a standard VPC subnet port that is exactly the scenario SpoofGuard exists to stop: the port’s address bindings pin one MAC, and a frame from a different MAC — or the same MAC arriving on a different port — is dropped. Part 1 showed that killing the host on a standard subnet before it ever spoke. This test shows the trunk subnet tolerating the live migration of a foreign MAC between two of its ports, which is the property nested vSphere (and anything else with a vSwitch inside a VM) fundamentally needs.

So the second vNIC buys three things, none of them physical redundancy:

1. Bringup and vLCM stop complaining about a single-uplink host.
2. The teaming policy you’ll configure in production gets exercised — uplink failover, active/standby for vSAN, whatever you’re rehearsing.
3. A live proof that the trunk carries MAC mobility, which is the real assurance that the design isn’t relying on a quiet network.

What it does not buy, and how to say so

If someone asks “is this host redundant?”, the answer is “the nested host believes it is; actual redundancy lives one layer down.” In a training pod that’s the correct and useful answer — students configure and test failover exactly as they would on metal, and the outer platform does the real work. In a reproduction lab for a customer NIC-teaming issue, it’s usually enough too: most teaming bugs are in ESXi’s policy handling, not in the copper.

Where it’s genuinely insufficient: anything about physical link behaviour — LACP negotiation, LLDP, flapping, MTU mismatch on one uplink. The virtual fabric never fails asymmetrically, so it can’t reproduce those.

Why this matters outside the lab

The practical value here is knowing what a nested environment can and can’t prove — which is what lets you decide when a virtual lab is enough and when it isn’t. For training, upgrade rehearsals, configuration and policy testing and the vast majority of “how does it behave when…” questions, nested is enough and dramatically cheaper. For physical link behaviour — LACP, optics, asymmetric faults — you still want metal. Being able to make that call confidently is worth more than the test itself.

Rules learned

• Give nested VCF hosts two vNICs on the same trunk subnet. Bringup, vLCM and vDS teaming expect ≥ 2 vmnics; humouring them costs nothing.
• Test failover from a session that depends on it (SSH via the VIP). Pings passing while your terminal dies is not success.
• The trunk subnet tolerates a vmk MAC moving between ports mid-flow — that’s the property standard subnets lack and nested vSphere needs.
• Be precise in the write-up: nested dual-NIC gives policy realism, not physical redundancy. Physical link faults can’t be reproduced here.
• Rebuilds re-run the vmk config: the appliance creates vmk0 only; vmk1/vmk2, their VLANs and override gateways are applied post-boot.

Companion to nested ESXi inside an NSX VPC.

Lab environment; opinions my own. Output captured live, trimmed for length, never edited for outcome.

This post is about what it became: a set of VCF Automation catalog items where requesting one form produces a complete nested VCF 9.1 instance — ESXi hosts, bringup (vCenter, NSX, SDDC Manager), a vSphere Supervisor, VCF Automation, Operations, identity — with the environment number as practically the only real input.

The shape of the factory

Three stages, each a catalog item, plus a wrapper that chains them:

Stage 1  Nested ESX Hosts        VM Apps template + vRO actions
         (the old script, reborn declaratively)
Stage 2  Deploy VCF 9.1 Instance vRO workflow driving the VCF Installer API
         (spec generated, validated, bringup started)
Day-N    Supervisor · NSX Edge · VCF Automation · Ops Logs/Networks/RTM ·
         Identity (AD)           one catalog item each
Wrapper  "Deploy VCF Stack"      one form, checkbox per component

The wrapper’s form has a checkbox per component; ticking one reveals that component’s tab with every field pre-populated. One lab password feeds every credential. Tick everything, click request, and go get coffee.

Rule 1: derive everything from one number

Each lab environment is f0X, and everything scales from X by formula:

The bringup spec — hundreds of lines of JSON the VCF Installer wants — is generated by a vRO action from a known-good reference spec plus X. Nobody edits a deployment spec by hand, which means nobody typo-breaks a bringup at 2am. When the old script did this, the formulas lived in string concatenation; now they live in one action with the reference spec beside it.

The same philosophy carried into stage 1: the script’s “next free esxNN index” scan became a vRO action bound to the request form, its VLAN/IP arithmetic became template expressions, its --prop:guestinfo.* flags became ovfProperties in the template. Porting a script isn’t rewriting it — it’s finding the declarative home for each behaviour.

Rule 2: never wait for anything you can watch instead

The hard constraint that shaped the whole design: a VCF Automation request gets about two hours before the platform gives up on it. A full VCF bringup takes longer than that. So the wrapper never waits:

• Bringup is fire-and-forget — the workflow authenticates to the installer, validates the spec, starts the task, and hands back a watchTaskId. Re-attach any time to check on it.
• Fleet deployments (VCF Automation, Ops for Logs/Networks, metrics) are server-side tasks; the items submit with waitForCompletion=false.
• The supervisor item submits enablement and returns; vCenter carries on.
• Only fast, deterministic steps (identity configuration, minutes) run to completion inside the request.

A whole VCF instance as one deployment record — the request finished in under an hour while the build ran on for twelve.

Result: a full-stack kick-off completes as a request in well under an hour (53 minutes on the run pictured), while the actual multi-hour build continues as watchable server-side tasks. The request’s job isn’t to do the work — it’s to start the work correctly and tell you where to watch it.

The corollary: a failed component is recorded and the remaining components still run. You fix one thing and re-run one item, not the world.

Rule 3: plan mode for infrastructure

Every item in the chain supports validateOnly — and the wrapper cascades it. Tick everything, set validateOnly, and the entire stack is planned against the live environment with zero changes: specs generated, prerequisites checked, name/IP collisions caught. A smoke runner exercises exactly this on every change to the automation itself.

If you build nothing else into your lab automation, build this. The number of 2am bringups saved by a five-minute dry run is not small.

The traps that shaped the rules

Some of the design above exists because of scars:

• Hardware validation hates virtual NVMe. Bringup’s HCL check will block nested hosts; the spec generator has to account for it, or you discover it two hours in — twice, if you’re us.
• Small disks, surprising layouts. Nested hosts with 64 GB disks ship ESX-OSDATA at essentially the whole disk; a post-provision step relocates scratch or stage-2 fills the disk with logs.
• DNS is a prerequisite, not a step. The installer’s pre-flight wants every record resolvable before it starts; a one-shot script creates the per-environment records ahead of the request.
• vRO’s content-source lag. A new or changed workflow takes 15–20 minutes of data-collection before the catalog sees it. Publish, wait, then test — or you’ll debug a ghost.
• Wrapper inputs are duplicated by necessity. vRO requires every sub-workflow input to be passed explicitly, so adding an input to a component means updating the wrapper’s call too. Null-guards in each component turn a forgotten field into a loud failure instead of a silent default.

Why bother?

Because the payoff compounds. Once a full VCF instance is a catalog request, everything downstream changes character: upgrade rehearsals happen on freshly-built instances instead of precious pets; a broken environment is redeployed, not repaired; and the lab stops being a snowflake collection and becomes a product — versioned, validated, reproducible.

Where this is heading: the factory’s output feeding per-student pods with identical addressing.

The factory’s next customers, funnily enough, are the isolated VPC pods from the other series on this blog — same philosophy, one layer further down.

Why this matters outside the lab

A complete VCF instance from one form changes what an environment costs to have. Environments that used to be precious — because building one took a week — become disposable, and a lot follows from that:

• Proofs of concept run on an environment built for the customer’s scenario, not on whatever happens to be free.
• Upgrade and migration rehearsals happen on a fresh instance of the right version, then it’s deleted.
• Training and enablement get a real VCF per person or per team.
• Reference builds exist for every supported release, on demand.

This is how Comms-care provides a dedicated instance to every consultant. The same factory, pointed at a customer’s requirements, is a repeatable way to deliver environments rather than a one-off project each time.

Rules learned

• Derive names, subnets and VLANs from a single environment number; generate specs, never hand-edit them.
• Respect the request-duration ceiling: start long work, return a task handle, re-attach to watch. Never block a wrapper on an hours-long task.
• validateOnly on every item, cascaded by the wrapper — dry-run the whole stack before touching anything.
• Componentise failure: one broken step re-runs alone.
• Pre-create DNS; expect HCL friction on virtual hardware; budget for vRO’s content-source lag.

Lab environment; opinions my own. The automation described builds nested VCF 9.1 instances for lab and rehearsal use — patterns transfer, specifics are ours.

Porting it to a VCF Automation catalog item (VM Apps — a cloud template plus vRO) is not a rewrite. It’s a sorting exercise: every behaviour in the script has a natural home in the declarative model, and the skill is finding it. Here’s the whole table, then the four rows that took thought.

The mapping

Five homes, in decreasing order of preference: input, template expression, platform abstraction (image mapping, network profile, cloud account), form action (read-only lookup at request time), subscription (imperative work at a lifecycle stage). Push each behaviour as far up that list as it will go.

The four decisions that weren’t obvious

1. Drop auto-detect; the platform already remembers

The script inspected an existing host to infer size. That was a workaround for having no record. The catalog is the record — deployment history shows what size every existing host was requested at — so the input simply asks. Fewer moving parts, and the requester sees the choice.

2. The index scan is a form action, not a workflow step

“Next free esx07” has to be known at request time so the requester sees the names they’ll get. That’s a vRO action bound to the custom form (getNextEsxHostIndexes(environment, role, count) → array of strings), not a step inside provisioning. Forms can call actions; use it for anything that’s a lookup.

3. Rename and folder go in Compute Allocation, hardware goes in Post Provision

Two blocking subscriptions, filtered by a custom property on the template:

The rename must be at allocation — it’s the only stage where a workflow output named resourceNames is applied to the machine. Disk growth and nested-HV need a VM that exists, so they wait for post-provision. Both are blocking: the deployment doesn’t proceed until they return.

4. nestedEsx: 'yes' — not true

The subscription condition is event.data.customProperties.nestedEsx == "yes". Why not "true"? Because boolean-looking strings can arrive in the event payload as typed booleans, and true == "true" is false in the condition evaluator and in the vRO code. It fails silently — the subscription just never fires. yes can’t be coerced. Small thing; two hours.

What stayed exactly the same

The formulas. 10.(20+X).<sub>.0/24, VLAN 2X0n, gateway .254 — they were string concatenation in PowerShell and they’re template expressions now, character for character. The guestinfo.* property names — identical, because the OVA didn’t change. Porting a script well means most of it survives; only the plumbing moves.

The bits that still bite

• Network profile without IP ranges. Addressing is injected via guestinfo, not the platform’s IPAM. Tag the trunk portgroup, add no ranges, or IPAM and guestinfo will disagree.
• Two template revisions in the repo. v1 is what the guide documents; v2 grew later. Both kept deliberately, both labelled. Check which one the org has imported before editing either.
• Content-source lag. A new or changed vRO action needs ~15–20 minutes of data collection before the form sees it. Publish, wait, then test.
• Small disks and OSDATA. Nested hosts with 64 GB disks ship ESX-OSDATA at essentially the whole disk. Templates now provision 128 GB; a relocate-scratch script mitigates existing hosts.

Every menu prompt from the script is now a field; Host Indexes is the form action’s answer to “next free esxNN”.

Why this matters outside the lab

Almost every organisation has these scripts: valuable, trusted, and stuck on one person’s machine. The message of this post for them is that modernising doesn’t mean rewriting. The logic survives; what changes is where it lives — behind a request form with access control, an audit trail, consistent inputs and a deployment record. That’s how a team turns tribal knowledge into a service without losing the years of edge cases the script already handles.

Rules learned

• Porting is sorting: input → expression → platform abstraction → form action → subscription. Push each behaviour as far up as it goes.
• Lookups the requester needs to see are form actions.
• Rename at allocation (resourceNames output); hardware at post-provision. Both blocking.
• Filter subscriptions on a custom property, and make its value a word that can’t be coerced to a boolean.
• Drop workarounds for missing state; the catalog is the state.
• Keep the formulas. Move the plumbing.

Part of The Lab Factory. Next: driving the VCF Installer API from vRO.

Lab environment; opinions my own.

Unlike stage 1 this isn’t VM provisioning, so it’s not a cloud template. It’s a vRO workflow published directly as a catalog item through an Orchestrator content source.

The spec is generated, never edited

A vRO action, buildVcfDeploymentSpec(environment, hostFqdns, labPassword, …), returns the whole spec as a string. Its structure was reconciled against a validated export from a real bringup — the installer UI lets you export the spec it accepted — and everything variable derives from the environment number X:

var n   = parseInt(environment.substring(1), 10);   // "f03" -> 3
var pfx = environment + "-m01";
var net = "10." + (20 + n);
function vlan(o) { return 2000 + (n * 100) + o; }
function gw(sub) { return net + "." + sub + ".254"; }

Static across environments: DNS, NTP, subdomain, component sizes, vSAN ESA FTT=1, and the component build versions pinned to the installer binaries. One lab password feeds every credential field (the UI export scrubs them; the action puts them back per the API schema).

Two spec-level decisions worth stealing:

• skipEsxThumbprintValidation: true instead of carrying per-host sslThumbprint. Supported, and the right trade-off for a lab.
• Ops and Automation are checkboxes that add their blocks to the spec — and the installer only accepts a licenseServerSpec when Ops is present, so the action adds them together or not at all.

The workflow: authenticate → validate → start → return

1. POST /v1/tokens                 installer login
2. POST /v1/sddcs/validations      the installer's OWN pre-flight on the spec
   (poll until COMPLETED; fail on any FAILED check)
3. if validateOnly -> return the validation report; touch nothing
4. POST /v1/sddcs                  start bringup -> sddcTaskId
5. return { sddcTaskId, installerUrl }

Step 2 is the validateOnly story made concrete: the installer will tell you, in seconds, that f03-m01-nsx01.res.lab doesn’t resolve, that an IP is in use, that a host isn’t reachable. Two hours into a bringup is a bad time to learn that. Every one of the ~40 DNS records the pre-flight wants is created ahead of time by a one-shot PowerShell script (New-LabEnvDnsRecords.ps1) — DNS is a prerequisite, not a step.

The two-hour leash, and how to slip it

A request from the catalog carries a token with a roughly two-hour lifetime, and a bringup takes around eight. So by default the workflow is fire-and-forget: waitForCompletion=false, return the task id, watch progress in the installer UI. A watchTaskId input lets you re-attach later and poll an already-running bringup from a new request.

The wrapper that chains everything — hosts, bringup, then the day-N components that need bringup to be finished — has a neater trick. The catalog-bound parent deploys the hosts, submits bringup, and then re-executes itself as a plain vRO run (Orchestrator → Run, no catalog token, no two-hour kill) carrying the hidden bringupWatchTaskId. That continuation polls the installer task to completion — eight hours, fine — and then runs certificates, fleet items, edge, supervisor and identity. Watch it under Orchestrator → Activity → Runs. The catalog request itself completes in under an hour — 47 minutes on the run pictured below — having started the work correctly and handed off.

Two rows, one build. The parent returns inside the catalog’s window; the continuation waits out the bringup and does the day-N work.

Nested-host frictions

Three things a physical bringup never meets:

• HCL validation vs virtual NVMe. The installer’s hardware check blocks the virtual NVMe controller. Fix at the vLCM layer: enforce_hcl_validation=false on the image policy. The vSAN health test nvmeonhcl also complains; silenced via the vSAN API, best-effort with manual fallback.
• DVS compatibility appears late. After bringup, NSX takes 1–2 hours to settle before the supervisor’s zones endpoint stops returning 500. If the supervisor stage fails “No compatible DVS” on a fresh instance, wait and re-run just that item.
• TSM-SSH. Bringup wants SSH on the hosts; the wrapper enables it host-direct via SOAP before submitting.

Stale schema: the failure that looks like a bug and isn’t

Add an input to the vRO workflow after the catalog item exists and the form will show the new field, the request will record its value, and the workflow will receive null — Service Broker keeps the old request schema until the content source re-imports. The workflow null-guards every boolean and aborts with “inputs not mapped” rather than running with silently-wrong options. Fix: re-import the content source, confirm the schema, submit a new request (resubmitting an old one reuses the old payload).

There are actually three async layers between “publish” and “mappable request” — vRO processing the import, the catalog schema after re-import, and the form service still enforcing the previous custom form for a minute or two. Same symptom for all three. Check timing before assuming a bug.

Why this matters outside the lab

Repeatable, generated VCF deployments matter well beyond a lab: a second site, a disaster-recovery instance, a new business unit, an environment per supported release. Generating the specification from a validated reference removes the class of errors that comes from editing hundreds of lines of JSON by hand, and running the installer’s own validation first turns “find out in hour two” into “find out in minute one”. It’s the difference between a VCF deployment being a project and being a procedure.

Rules learned

• Generate the spec from a validated export plus one number. Nobody hand-edits JSON at 2am.
• Run the installer’s own validation first, and make it a mode you can request on its own.
• Pre-create DNS. Enable SSH. Disable HCL enforcement on virtual NVMe.
• Respect the request lifetime: start, return a task id, re-attach. For a long chain, have the workflow re-run itself outside the catalog.
• Null-guard every input and fail loud; stale schemas are a fact of life after adding inputs.
• On a fresh instance, give NSX an hour before you expect DVS compatibility.

Part of The Lab Factory. Previously: porting the host script.

Lab environment; opinions my own. Bringup verified end-to-end on a rebuilt environment: 305/305 tasks, COMPLETED_WITH_SUCCESS.