[{"content":"Every platform needs a demo stack — something that looks alive on a projector and quietly exercises every layer underneath. This is the lab\u0026rsquo;s: seven apps on a VKS cluster that VCF Automation provisioned, in a tenant VPC, each app behind its own load-balancer VIP.\nkubedoom 192.168.144.20:5900 (VNC — yes, it kills pods) kubeinvaders 192.168.144.21 kube-ops-view 192.168.144.22 pacman 192.168.144.23 (+ MongoDB on a PVC — persistent high scores) podinfo 192.168.144.24 goldpinger 192.168.144.25 (DaemonSet incl. control plane) prometheus (in-cluster: KSM + node-exporter, 11/11 targets up) Captured from the VIPs the platform handed out. KubeDoom is VNC-only and podinfo is an API, so they sit this one out; the Ops topology tile shows all seven by name.\nThe path that matters: tenanted, not shortcut The first version of this stack was deployed against the supervisor — an admin kubeconfig, a vSphere namespace, kubectl apply. It worked and it was wrong, for the reason the previous post spells out: nothing about it was provided to anyone. So it was torn down (seven apps, cluster, VPC and VIPs gone in about seven minutes) and rebuilt the proper way:\ndev-01 org → default-project → SupervisorNamespace (class large, region f06, VPC default-f06) → VKS cluster vks-demo01 → seven apps All created through VCF Automation — the CCI API for the namespace and cluster, then the apps via the cluster\u0026rsquo;s own kubeconfig. Two auth facts worth writing down, because they cost a cycle each:\nA provider service account reaches the cloud API only. CCI (/cci/kubernetes/apis/...) rejects provider tokens with 401 — it needs an org-scoped service account. Device-flow login uses the service account\u0026rsquo;s own UUID as client_id (not its software ID), against the tenant endpoint /oauth/tenant/\u0026lt;org\u0026gt;/device_authorization. What the platform does for free Each app is an ordinary Deployment + Service of type LoadBalancer. The supervisor turns each service into an NSX VPC LB virtual server and hands back a VIP from the org\u0026rsquo;s external block. No ingress controller, no MetalLB, no port-forwarding — seven services, seven VIPs, done. Pac-Man\u0026rsquo;s MongoDB asks for a PVC and gets a vSAN-backed volume through the CSI the supervisor already installed. Goldpinger runs as a DaemonSet across every node including the control plane and draws the node-to-node mesh live.\nThat\u0026rsquo;s three platform services (LB, storage, networking) exercised by apps that know nothing about VCF.\nThe traps Supervisor refuses cluster-scoped RBAC — even to admin. Demo apps that need ClusterRoles (kube-ops-view, Goldpinger, KubeDoom) must live on a guest cluster. You cannot run them as vSphere Pods on the supervisor.\nPodSecurity restricted is the VKS 1.35 default. Half the demo set runs as root. Symptom: Deployment shows 0 UP-TO-DATE, ReplicaSet exists, zero pods, events say FailedCreate. Fix: label the namespace pod-security.kubernetes.io/enforce=privileged — and mention in the demo that you did, because it\u0026rsquo;s a teaching moment.\nnode-exporter without hostNetwork. The VPC fabric blocks pod → node IP scrapes, so the stock DaemonSet\u0026rsquo;s hostNetwork: true doesn\u0026rsquo;t help; run it as a normal pod and let Prometheus scrape it in-cluster.\nDocker Hub is flaky from behind a proxy. TLS handshake timeouts put pods into kubelet\u0026rsquo;s image-pull backoff. Deleting the stuck pods bypasses the backoff; a Harbor proxy-cache project fixes it properly.\nPod CIDR shadowing. The stock 192.168.0.0/16 pod range hid the org\u0026rsquo;s 192.168.144.0/21 external block from inside the cluster — apps couldn\u0026rsquo;t reach their neighbours\u0026rsquo; VIPs. Pod CIDR is now 172.16.0.0/16.\nAutomation notes The whole stack is a checkbox on the lab\u0026rsquo;s catalog item that deploys a supervisor: installDemoApps creates the cluster (newest compatible Kubernetes release, newest built-in ClusterClass, auto-detected), applies the seven apps, and reports their URLs in the deployment summary. Run integrated on a fresh environment: 16.7 minutes to CREATE_SUCCESSFUL. An immediate re-run: 3.3 minutes, all steps idempotent — which is the number I actually care about, because it means a broken demo is a re-run, not a rebuild.\nWhy this matters outside the lab A demo stack sounds like a toy. It\u0026rsquo;s actually the fastest way to make a platform legible to people who don\u0026rsquo;t read YAML: a customer watches a request become a cluster, watches seven services get their own addresses, opens one and plays it. Everything underneath — self-service Kubernetes, load balancing, persistent storage, isolation — is being exercised in a way a non-technical stakeholder can see working. The same stack is what we put in front of a new team on day one, and the same idempotent deploy is what makes it safe to demonstrate live: if it breaks on stage, it re-runs in three minutes.\nRules learned Demo apps that need cluster-scoped RBAC must run on a guest cluster; the supervisor won\u0026rsquo;t grant it, even to admin. Build the stack through the tenanted path (org SA → CCI → namespace → cluster → apps). Same apps, but now they\u0026rsquo;re provided, quota\u0026rsquo;d and visible in Ops. VKS 1.35: restricted PodSecurity by default. Label the namespace and say why. LoadBalancer per app = NSX VIP per app. No ingress needed for a demo. Pick a pod CIDR that doesn\u0026rsquo;t overlap the VPC external block. Make the deploy idempotent; a demo that re-runs in 3 minutes is one you can afford to break on stage. Previously: one VKS cluster, two ways. The Pac-Man instance here is the one from What\u0026rsquo;s a VPC?.\nLab environment; opinions my own.\n","permalink":"https://thenestedlab.com/posts/demo-apps-via-vcfa/","summary":"KubeDoom, KubeInvaders, kube-ops-view, Pac-Man with persistent MongoDB, podinfo, Goldpinger and a Prometheus stack — deployed onto a VCFA-provisioned VKS cluster in a tenant VPC, each behind its own NSX VIP. The proper tenanted path (not the supervisor shortcut), what the platform does for free, and the traps.","title":"Seven demo apps, one request: deploying a showcase stack via VCF Automation"},{"content":"Before this series gets into trunk subnets and binding maps, it\u0026rsquo;s worth spending ten minutes on the thing everything else stands on: what an NSX VPC actually is from the tenant\u0026rsquo;s chair. No slides. A game of Pac-Man.\nThe lab has Pac-Man running twice on VCF 9.1 — once on a VKS cluster I built by hand with kubectl, once on a cluster deployed through VCF Automation\u0026rsquo;s catalog. Both live inside VPCs. Both were reachable when I started:\nhttp://192.168.144.15/ -\u0026gt; \u0026lt;title\u0026gt;Pacman in HTML 5 Canvas http://192.168.144.23/ -\u0026gt; \u0026lt;title\u0026gt;Pacman in HTML 5 Canvas Pac-Man, live at 192.168.144.23 — a VIP on the VPC load balancer. The only door in. A VPC is a private universe with a door policy Think of an NSX VPC as a tenant\u0026rsquo;s own routed network space: its own subnets, its own gateway, its own address plan — carved out by the tenant, not filed as a ticket with the network team. Three rules define it:\nPrivate by default. A Private subnet is reachable only from inside the same VPC. Nobody outside can route to it — not other tenants, not other VPCs in the same org, not the corporate network. Your addresses are your business. Because private subnets aren\u0026rsquo;t advertised anywhere, two VPCs can use identical CIDRs. (This is the superpower the rest of the series is built on.) Every door out is deliberate. Traffic leaves via the transit gateway — SNAT\u0026rsquo;d — or arrives via a LoadBalancer VIP from an external block the provider allocated. Nothing is exposed by accident. Pac-Man\u0026rsquo;s pods sit on a private subnet. The only reason 192.168.144.23 answers is a Kubernetes Service of type LoadBalancer, which NSX turns into a VIP on the VPC\u0026rsquo;s load balancer. So let\u0026rsquo;s remove the door.\nBefore: close the door $ kubectl patch svc pacman -n pacman -p \u0026#39;{\u0026#34;spec\u0026#34;:{\u0026#34;type\u0026#34;:\u0026#34;ClusterIP\u0026#34;}}\u0026#39; service/pacman patched NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE pacman ClusterIP 10.106.219.213 \u0026lt;none\u0026gt; 80/TCP 4d13h $ curl -m 5 http://192.168.144.23/ curl: timed out / unreachable The pods are running. The service exists. The game is fine — for anything inside the VPC. From my desk it\u0026rsquo;s simply gone. That\u0026rsquo;s the whole VPC model in one curl: the boundary isn\u0026rsquo;t a firewall rule somebody wrote, it\u0026rsquo;s the absence of a route.\nAfter: open it again $ kubectl patch svc pacman -n pacman -p \u0026#39;{\u0026#34;spec\u0026#34;:{\u0026#34;type\u0026#34;:\u0026#34;LoadBalancer\u0026#34;}}\u0026#39; service/pacman patched NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE pacman LoadBalancer 10.106.219.213 192.168.144.23 80:31467/TCP 4d13h Same VIP handed straight back. NSX programmed a virtual server and pool on the VPC LB; the supervisor stitched it to the cluster\u0026rsquo;s NodePort. Door open.\nAnd then the game didn\u0026rsquo;t load.\nThe outage I gave myself (this is the useful bit) Everything was green:\nkubectl get svc — LoadBalancer, VIP assigned kubectl get endpoints — pod IPs present NSX — virtual server up, pool members healthy iptables on the node — NodePort rules identical to a working neighbour service Five layers, all green, and curl hung. The control experiment was the manually-built cluster\u0026rsquo;s Pac-Man at .15, untouched throughout, still playing.\nThe cause: my \u0026ldquo;harmless\u0026rdquo; ClusterIP patch earlier had included a ports list. kubectl patch with a merge patch replaces arrays, it doesn\u0026rsquo;t merge them — and my array said targetPort: 80. Pac-Man listens on 8080. Every layer above was faithfully forwarding traffic to a port nothing was listening on, and every layer reported success because its job was done.\n$ kubectl patch svc pacman -n pacman --type=json \\ -p \u0026#39;[{\u0026#34;op\u0026#34;:\u0026#34;replace\u0026#34;,\u0026#34;path\u0026#34;:\u0026#34;/spec/ports/0/targetPort\u0026#34;,\u0026#34;value\u0026#34;:8080}]\u0026#39; service/pacman patched $ curl -s http://192.168.144.23/ | grep -o \u0026#39;\u0026lt;title\u0026gt;.*\u0026lt;/title\u0026gt;\u0026#39; \u0026lt;title\u0026gt;Pacman in HTML 5 Canvas\u0026lt;/title\u0026gt; Instant recovery. The diagnosis walked the entire paravirtual chain — VIP → supervisor VirtualMachineService → NSX VS/pool → NodePort iptables → pod — and it\u0026rsquo;s exactly the walk you\u0026rsquo;ll need one day:\nLayer Check What \u0026ldquo;green\u0026rdquo; hides VIP kubectl get svc EXTERNAL-IP nothing about the backend NSX LB virtual server + pool status pool health is TCP to the NodePort, not the pod Endpoints kubectl get endpoints it lists pod IP:targetPort — read the number Node iptables -t nat -L KUBE-SERVICES rules can be perfect and point at the wrong port Pod kubectl exec ... ss -ltn the only place the truth lives Bonus find on the way: kube-proxy and Antrea on that cluster had dropped their API watches days earlier (http2: client connection lost) and never re-established informers until restarted. It didn\u0026rsquo;t cause this outage, but it\u0026rsquo;s the kind of thing you only find when you\u0026rsquo;re forced to look.\nWhy this matters outside the lab If you run a platform for more than one team, this is the feature you\u0026rsquo;ve been asking the network team for. A VPC gives each team, project or customer its own private network space — created by them, in minutes, with nothing reachable from outside until they publish it. Security teams like it for the same reason developers do: exposure is a deliberate, auditable act, not a side effect of plugging something in.\nWhat organisations do with it once they have it:\nPer-team sandboxes that can\u0026rsquo;t see each other, provisioned without a ticket. Partner or supplier environments isolated from the corporate estate but hosted on the same platform. Multi-tenant hosting — service providers and internal IT alike — with isolation enforced by topology rather than a growing pile of firewall rules. Rules learned A VPC\u0026rsquo;s boundary is the absence of a route, not a rule. Private subnets are unreachable from outside by construction — which is also why identical CIDRs across VPCs just work. A LoadBalancer service is the deliberate door: NSX VIP from the external block, programmed per service. Flip the type and the door closes with nothing else to clean up. kubectl patch (merge) replaces spec.ports, it doesn\u0026rsquo;t merge it. Patch a single field with --type=json, or don\u0026rsquo;t include the array. Five green layers can hide one wrong integer. Keep a working control (here: the untouched .15 instance) and compare layer by layer. Read kubectl get endpoints as IP:targetPort — the port is the part people skim. Next in the Pod Papers: nested ESXi inside a VPC — where \u0026ldquo;private by default\u0026rdquo; meets a host that fakes its own MAC address.\nLab environment; opinions my own. Output captured live, trimmed for length, never edited for outcome — including the outage.\n","permalink":"https://thenestedlab.com/posts/whats-a-vpc-with-pacman/","summary":"Part 0 of the Pod Papers: an NSX VPC explained with a running game. Private by default, one deliberate door out — and a self-inflicted outage that taught me five green layers can hide one wrong integer.","title":"What's a VPC? Let Pac-Man explain"},{"content":"The host booted clean. Management IP configured, services up, DCUI happy. And every single packet it sent — ARP included — died silently.\nThat\u0026rsquo;s how my first attempt at running nested ESXi inside an NSX VPC ended, and the failure mode is nasty precisely because nothing looks wrong. If you\u0026rsquo;re trying to build nested vSphere labs on VCF 9 with VPC networking, this post is the map of the minefield — and the design that gets you across it, verified live.\nThe setup VCF 9.1, vSphere Supervisor with NSX VPC networking. The goal: deploy nested ESXi hosts as ordinary VM Service VMs inside a tenant\u0026rsquo;s VPC — no physical fabric changes, no provider tickets, no special treatment. The kind of thing you want for training pods, cert-study labs, or reproducing customer issues.\nNested ESXi needs what physical ESXi needs: a management network, vMotion, vSAN — traditionally VLANs trunked to every host. But a VPC is an overlay world. There are no VLANs to trunk. So what happens if you just attach the nested host\u0026rsquo;s vNIC to a normal VPC subnet?\nFailure #1: the silent blackhole Here\u0026rsquo;s the trap. A standard VPC subnet port gets address bindings: NSX pins the exact IP + MAC it allocated to that vNIC, and SpoofGuard drops everything else.\nESXi\u0026rsquo;s vmk0 doesn\u0026rsquo;t use the vNIC\u0026rsquo;s MAC. It synthesises its own:\nvmk0 MAC Address: 00:50:ac:1e:00:8c \u0026lt;- NOT the vNIC MAC (04:50:56:...) So every frame the management interface sends carries a MAC the port doesn\u0026rsquo;t own. NSX drops it all — ARP, ping, everything — while the host itself boots green and reports healthy. There is no error anywhere. You just can\u0026rsquo;t reach it, ever.\n(There\u0026rsquo;s a second trap stacked on top: VPC subnets run with DHCP deactivated, so the appliance also sits at \u0026ldquo;waiting for DHCP\u0026rdquo; unless you inject static addressing via OVF guestinfo.* properties. More on that below.)\nThe design that works: a trunk subnet + binding maps The fix isn\u0026rsquo;t a hack — it\u0026rsquo;s a first-class NSX VPC construct that\u0026rsquo;s barely documented in the wild: SubnetConnectionBindingMap.\nThe idea:\nCreate one ordinary VPC subnet to act as a trunk (sn-trunk). The nested host\u0026rsquo;s vNICs attach only here. Create a normal VPC subnet per traditional network — sn-mgmt, sn-vmotion, sn-vsan. Bind each of those to the trunk with a binding map carrying a VLAN tag. The nested host\u0026rsquo;s vSwitch tags frames exactly as it would on metal; the binding map strips the tag and delivers the frame into the right subnet. Pure L2 demultiplexing. One vNIC carries N VLANs, the VPC never routes on a tag, and the physical fabric never sees any of it (the 802.1Q header rides inside the Geneve overlay).\nAll of it is tenant-creatable through the supervisor as Kubernetes objects:\n# sn-trunk and sn-mgmt are ordinary Private Subnets; the interesting object: apiVersion: crd.nsx.vmware.com/v1alpha1 kind: SubnetConnectionBindingMap metadata: {name: bm-mgmt} spec: subnetName: sn-mgmt # the map is a child of the VLAN subnet... targetSubnetName: sn-trunk # ...and points AT the trunk vlanTrafficTag: 1610 That direction is easy to invert, so it\u0026rsquo;s worth saying twice: the binding map belongs to the VLAN subnet and points at the trunk, not the other way round.\nOn the nested host, nothing exotic — plain VST, like physical:\nName Virtual Switch Active Clients VLAN ID ------------------ -------------- -------------- ------- Management Network vSwitch0 1 1610 vMotion vSwitch0 1 1611 vSAN vSwitch0 1 1612 The same three VLANs as the nested host sees them.\nAnd because there\u0026rsquo;s no DHCP in a VPC subnet, the nested-ESXi appliance gets its identity through OVF properties in the VM Service spec:\nbootstrap: vAppConfig: properties: - {key: guestinfo.ipaddress, value: {value: \u0026#34;172.30.0.40\u0026#34;}} - {key: guestinfo.netmask, value: {value: \u0026#34;255.255.255.224\u0026#34;}} - {key: guestinfo.gateway, value: {value: \u0026#34;172.30.0.33\u0026#34;}} - {key: guestinfo.vlan, value: {value: \u0026#34;1610\u0026#34;}} Does it actually work? The receipts Two nested hosts, vNICs on sn-trunk, three VLANs. From host one:\n[root@esx01:~] vmkping -c2 172.30.0.41 # mgmt, VLAN 1610 3 packets transmitted, 3 packets received, 0% packet loss [root@esx01:~] vmkping -I vmk1 -c3 172.30.0.71 # vMotion, VLAN 1611 3 packets transmitted, 3 packets received, 0% packet loss [root@esx01:~] vmkping -I vmk2 -c3 172.30.0.101 # vSAN, VLAN 1612 3 packets transmitted, 3 packets received, 0% packet loss The transcript that matters: fail the first NIC, and every VLAN keeps flowing on the second — captured live.\nTwo more results worth knowing before you design around this:\nUntagged frames are dropped. I put a probe vmk on the untagged portgroup using the address NSX itself had allocated to the trunk port: 100% loss, empty ARP table, while tagged traffic flowed happily beside it. Every network your nested host uses needs a VLAN and a binding map — there is no untagged fallback.\nFailover behaves like real hardware. With two vNICs on the trunk teamed active/active, esxcli network nic down -n vmnic0 moved every VLAN onto vmnic1 with zero loss — and the SSH session I was watching from never dropped. The vmk MAC migrating between trunk ports mid-flow is exactly the scenario that MAC-pinned standard ports would blackhole; the trunk carries it fine.\nWhy this matters outside the lab Running whole vSphere environments inside a VPC turns the platform into something most customers never had: a way to stand up complete, isolated copies of infrastructure on demand, without a physical fabric change and without waiting for anyone. That\u0026rsquo;s what makes it commercially interesting:\nTraining and certification labs where every learner gets a real vSphere environment, not a shared one. Reproducing a customer problem on a like-for-like copy instead of on the customer\u0026rsquo;s estate. Rehearsing upgrades and migrations end to end before the change window, then throwing the copy away. Vendor and feature evaluations with real behaviour, at zero risk to production. This is the design Comms-care uses to give every consultant a dedicated environment, and the same pattern scales to a classroom or a proof-of-concept factory.\nRules learned A nested ESXi vNIC on a standard VPC subnet is dead on arrival: vmk0\u0026rsquo;s synthesised MAC loses to SpoofGuard, silently. Attach nested-host vNICs only to a trunk subnet; one binding map per VLAN; the map lives under the VLAN subnet and points at the trunk. No DHCP in VPC subnets — bootstrap addressing via guestinfo.* (appliances) or cloud-init (Linux). Static IP plans are a feature in a lab anyway. ESXi\u0026rsquo;s default TCP/IP stack has one gateway — set per-vmk override gateways (esxcli ... ipv4 set -g) so vMotion/vSAN carry their own subnet\u0026rsquo;s gateway. Recreating a VM reallocates its NSX addresses. Pin what you depend on. MTU: everything here ran at 1500. Raise the trunk and the nested vDS before you do vSAN at any real scale. Next in this series: what happens when you want ten of these labs — with byte-identical IP plans, firewalled from each other by construction. That\u0026rsquo;s where NSX VPCs go from \u0026ldquo;workaround\u0026rdquo; to genuinely better than physical.\nLab environment; opinions my own. Everything above was captured from a live VCF 9.1 environment — output trimmed for length, never edited for outcome.\n","permalink":"https://thenestedlab.com/posts/nested-esxi-nsx-vpc/","summary":"Plain VPC subnets silently blackhole a nested ESXi host. Here\u0026rsquo;s why — and the trunk subnet + binding map design that makes nested labs work as an ordinary NSX VPC tenant, verified end to end.","title":"Nested ESXi inside an NSX VPC: the trunk-subnet design"},{"content":"Everything was green. The VPC: realized. The namespace: ready. The VMs: powered on, endpoints populated, ports listening. And the LoadBalancer services sat at \u0026lt;pending\u0026gt; — for an hour.\nThis is the story of the least helpful error message in my recent memory, what it actually means, and the one-line ordering rule that would have saved an afternoon. If you\u0026rsquo;re doing self-service NSX VPCs on VCF 9 with the vSphere Supervisor, you will hit this. Bookmark accordingly.\nThe setup Tenant-created VPC (via the VCF Automation CCI API), a supervisor namespace pinned to it, and a couple of VirtualMachineService objects of type LoadBalancer to publish SSH and HTTPS for the workloads inside. Standard stuff — the exact pattern that works out of the box in the org\u0026rsquo;s default VPC.\nThe k8s side looked perfect:\n$ kubectl get endpoints -n pod-a NAME ENDPOINTS AGE esx01-access 172.30.0.40:443,172.30.0.40:22 6m36s esx02-access 172.30.0.41:443,172.30.0.41:22 6m35s Endpoints resolved. VIPs: nothing. The only clue, a recurring event:\nWarning FailedRealizeNSXResource service/esx01-access Generic error occurred during realizing network for Service \u0026ldquo;Generic error.\u0026rdquo; Wonderful.\nDigging: what NCP actually wants The supervisor\u0026rsquo;s network container plugin (NCP) logs told the real story:\nnsx_ujo.ncp.nsx.policy.lb_layer4_service Lb Service not Found for Namespace pod-a NCP00270 Failed to process virtual ip for service ...: Lbs pod-a is not found Encountered retryable error ... : Lbs pod-a is not found NCP wants an NSX LBService in the namespace\u0026rsquo;s VPC. In the org\u0026rsquo;s default VPC, one exists — the platform created it when the VPC was born. In my self-service VPC? Nobody had created one. Fair enough — that\u0026rsquo;s actually documented behaviour once you know where to look: a fresh VPC needs a LoadBalancer object (and before that, a VPCAttachment to a connectivity profile with the service gateway enabled, or the LB creation itself fails with a much better error message).\nSo I created the attachment, then the LBService. NSX: Realized=True. Problem solved?\nWarning FailedRealizeNSXResource service/esx01-access Generic error occurred during realizing network for Service No.\nThe actual bug-shaped behaviour: a snapshot, not a lookup Here\u0026rsquo;s the part that costs you the afternoon. That \u0026ldquo;retryable error\u0026rdquo; retries the lookup in NCP\u0026rsquo;s cache — not the discovery. NCP snapshots the VPC\u0026rsquo;s LB inventory when the namespace is created. An LBService that appears afterwards is never discovered, no matter how long you wait:\nRecreating the k8s services: no effect. Tagging the LBService with the nsx-op/* ownership tags the working ones carry: no effect — the cache doesn\u0026rsquo;t re-read NSX. Restarting NCP would force a full resync — but supervisor system pods are protected; even Administrator@vsphere.local gets a Forbidden. Mutating the namespace to nudge a re-sync: also blocked, by the supervisor\u0026rsquo;s namespace validation webhook. As a tenant, there is exactly one fix: delete and recreate the namespace, now that its VPC has an LB. Fifteen minutes of rebuild for want of one ordering rule.\nAnd the control experiment proves the rule: a namespace created after its VPC already had an LBService got its VIPs assigned without any drama —\nesx01-access VIP=192.168.144.34 22 OPEN · 443 OPEN esx02-access VIP=192.168.144.35 22 OPEN · 443 OPEN And once the ordering is right, this is what \u0026ldquo;working\u0026rdquo; looks like — the pod\u0026rsquo;s state a couple of minutes after a correctly-ordered deployment:\nWhat the requester sees once the order is right.\nThe ordering rule For every self-service VPC that will publish LoadBalancer services, create — in this order, before the namespace:\n1. VPC (vpc.nsx.vmware.com/v1alpha1) 2. VPCAttachment (connectivity profile w/ service gateway — LB creation errors without it) 3. LoadBalancer {regionName, vpcName} (the step everyone misses) 4. ...and only THEN the Supervisor Namespace Encode it in whatever provisions your VPCs — a script, a pipeline, an operator. It\u0026rsquo;s four API calls and it turns a silent, undiagnosable \u0026lt;pending\u0026gt; into a platform that just works.\nWhy this matters outside the lab Nobody buys a platform for its ordering rules — but this is exactly the kind of edge that decides whether self-service provisioning feels reliable or flaky to the people using it. In a customer deployment the answer isn\u0026rsquo;t a blog post; it\u0026rsquo;s that the provisioning automation already does the four steps in the right order, every time, so a tenant never sees a VIP stuck at \u0026lt;pending\u0026gt;. Knowing where the sharp edges are — because you\u0026rsquo;ve been cut by them in a lab — is most of what an experienced delivery partner is for.\nRules learned In a self-service NSX VPC, the LBService must predate the namespace. NCP discovers LBs at namespace-add and never again. FailedRealizeNSXResource: Generic error on a Service = go read the NCP logs; the real message (Lbs \u0026lt;ns\u0026gt; is not found, NCP00270) is there. VPCAttachment (service gateway) is the prerequisite for the LB itself — that one at least fails loudly. Retro-tagging NSX objects to look \u0026ldquo;owned\u0026rdquo; doesn\u0026rsquo;t help a cache that never re-reads. Recreating the namespace is the only tenant-level fix. While you\u0026rsquo;re at it: new namespaces also reject VM creation until image status.disks syncs (~1–3 minutes after content library attach). Build the wait into your automation and both sharp edges disappear. Previously in this series: nested ESXi inside an NSX VPC. Next: three datacenters, one IP plan — identical isolated pods.\nLab environment; opinions my own. Output captured live, trimmed for length, never edited for outcome.\n","permalink":"https://thenestedlab.com/posts/the-lb-that-must-exist-first/","summary":"VIPs pending forever, a retryable error that never stops retrying, and an ordering rule the docs don\u0026rsquo;t tell you: in a self-service NSX VPC, the LBService must exist before the namespace that will use it.","title":"The load balancer that must exist before the namespace"},{"content":"Here are three hosts, all answering to vmk0 = 172.30.0.40:\nssh root@192.168.144.30 -\u0026gt; [root@esx01-a:~] vmk0 172.30.0.40 ssh root@192.168.144.32 -\u0026gt; [root@esx01-b:~] vmk0 172.30.0.40 ssh root@192.168.144.34 -\u0026gt; [root@esx01-c:~] vmk0 172.30.0.40 Same IP. Same VLAN. Same gateway. Same MAC address, as it turns out. And none of them can reach any of the others. This is the post where NSX VPCs stop being a workaround for nested labs and become genuinely better than the physical alternative.\nWhy identical addressing matters If you\u0026rsquo;ve ever built training pods, cert-study labs, or per-team reproduction environments, you know the pain: every copy needs a unique address plan, so every runbook, every screenshot, every \u0026ldquo;type this exact command\u0026rdquo; has to be parameterised per pod. Students in seat 7 see different numbers from the slides. Reproductions drift from the original.\nThe fix is obvious and normally impossible: give every pod the same addresses. On a physical fabric that means VRFs, per-pod NAT, and a network team that stops answering your emails. In an NSX VPC it\u0026rsquo;s the default behaviour.\nThe mechanism: overlapping privateIPs Each pod gets its own VPC, and every VPC declares the same private range:\napiVersion: vpc.nsx.vmware.com/v1alpha1 kind: VPC metadata: {name: nested-vpc-a} # then -b, then -c spec: privateIPs: [\u0026#34;172.30.0.0/16\u0026#34;] # identical in all three A Private subnet is never advertised beyond its VPC, so NSX has no objection to three VPCs carving up the same /16. The pods aren\u0026rsquo;t \u0026ldquo;firewalled from each other\u0026rdquo; — there is simply no route between them. Isolation by construction, not by policy.\nNSX\u0026rsquo;s subnet view filtered to sn-mgmt: four rows, four VPCs, one CIDR.\nThe trick: deterministic realization Identical ranges aren\u0026rsquo;t enough — I want identical subnets, so the management gateway is .33 and the hosts are .40/.41 in every pod. NSX allocates subnets from privateIPs in creation order, and a fresh VPC allocates deterministically. So the topology is applied in a fixed order — trunk, mgmt, vMotion, vSAN — and every pod realizes the same map:\nSubnet Realized VLAN Hosts sn-trunk 172.30.0.0/27 — (carries the tags) sn-mgmt 172.30.0.32/27 1610 .40 / .41, gw .33 sn-vmotion 172.30.0.64/27 1611 .70 / .71, gw .65 sn-vsan 172.30.0.96/27 1612 .100 / .101, gw .97 In the catalog blueprint that order is enforced with dependsOn between the subnet resources — the one place a declarative tool needs to be told about sequence. Skip it and two pods can come out with mgmt and vMotion swapped, which works perfectly and confuses everyone.\nThe door: one VIP per host Each pod is unreachable from outside by design, so each host gets a VirtualMachineService of type LoadBalancer publishing SSH and HTTPS. The VIPs come from the org\u0026rsquo;s external block, and they\u0026rsquo;re the only addresses that differ between pods:\nPod VPC esx01 VIP esx02 VIP a nested-vpc-a 192.168.144.30 .31 b nested-vpc-b 192.168.144.32 .33 c nested-vpc-c 192.168.144.34 .35 Which is how the opening transcript works: three VIPs, three hosts, one inside address.\nProving the isolation Claims are cheap. The test matrix, from a VM in a fourth VPC (the org default):\nping 172.30.0.140 (own VPC)....... REACHABLE ping 172.30.0.40 (pod space)..... unreachable curl http://172.31.0.2/ (shared).. shared-svc repo01 Its own VPC\u0026rsquo;s 172.30.0.140: reachable. 172.30.0.40 — an address that exists in three other VPCs simultaneously: unreachable, because from here there is no such route. (The third line is the shared-services VPC, which is the next post.)\nInside each pod, east-west is normal: esx01 → esx02 vmkping passes on all three VLANs, in all three pods. And the detail I didn\u0026rsquo;t expect: the nested-ESXi appliance derives vmk0\u0026rsquo;s MAC deterministically from its config, so the three hosts share a MAC as well as an IP. Harmless — each VPC is its own L2 domain — but a nice demonstration of how complete the separation is.\nWhat this replaces Physical / VLAN-based pods VPC pods Identical addressing VRF per pod + NAT, fabric change per pod default behaviour Adding a pod switch config, IPAM, firewall rules one API call for the VPC, one blueprint request Isolation guarantee policy (auditable, breakable) topology (no route exists) Tenant self-service no yes — the VPC is a tenant object Why this matters outside the lab \u0026ldquo;Identical environments\u0026rdquo; sounds like a lab nicety. It\u0026rsquo;s actually one of the most requested things in enterprise IT, usually asked for in other words:\nTraining at scale — every seat in the room sees the same addresses as the slides, so material is written once and never parameterised per pod. Per-engineer or per-team replicas of a reference environment, for development and testing that behaves exactly like the original. Regulatory or business-unit separation on shared infrastructure without VRF sprawl or a bespoke firewall estate — isolation is a property of the topology, which is the easiest kind to evidence to an auditor. Blue/green copies of an environment for change rehearsal, then cut-over or discard. On a physical network each of these is a project. On VCF with NSX VPCs it\u0026rsquo;s a template.\nRules learned Overlapping privateIPs across VPCs is supported and intentional. Identical pods are a feature, not a hack. Fresh VPCs realize subnets deterministically in creation order — fix the order (dependsOn in a blueprint) and every pod gets the same map. Pods are unreachable from outside by construction; publish exactly what you mean to via LoadBalancer VIPs from the external block. Prove isolation from a different VPC, with a positive control (own VPC reachable) beside the negative. Expect duplicate MACs across pods from appliance images. It\u0026rsquo;s fine. Previously: the LB that must exist first. Next: one WSUS for pods that can\u0026rsquo;t see each other.\nLab environment; opinions my own. Output captured live, trimmed for length, never edited for outcome.\n","permalink":"https://thenestedlab.com/posts/three-datacenters-one-ip-plan/","summary":"Three nested-ESXi pods, byte-identical addressing — same subnets, same VLANs, same host IPs, even the same MACs — with zero reachability between them. How overlapping VPC CIDRs and deterministic subnet realization turn cookie-cutter environments into a first-class feature.","title":"Three datacenters, one IP plan: identical isolated pods with NSX VPCs"},{"content":"The pods from the last post are perfectly isolated. That\u0026rsquo;s the requirement — and immediately the problem. Every one of them needs Windows updates, a package repo, DNS, maybe a domain controller. Do I really run a WSUS per pod?\nNo. There\u0026rsquo;s a third subnet access mode for exactly this, and the design it enables is hub-and-spoke with a very specific property: spokes reach the hub; the hub cannot reach the spokes; spokes never reach each other.\nThe three access modes Everything in this series comes down to one field on a VPC subnet:\naccessMode Advertised to Use Private nobody outside the VPC workloads — isolation and overlapping CIDRs PrivateTGW every VPC attached to the org\u0026rsquo;s transit gateway shared services Public the external network internet/corp-facing endpoints PrivateTGW subnets draw their addresses from a separate transit block (here 172.31.0.0/…), not from the VPC\u0026rsquo;s own privateIPs. That\u0026rsquo;s the key: the shared range can\u0026rsquo;t collide with the pods\u0026rsquo; 172.30.0.0/16 because it comes from a different pool that all VPCs agree on.\nThe build One more VPC, shared-svc, with the same ordering rules as any other (VPC → VPCAttachment → LoadBalancer → namespace), then a single subnet:\napiVersion: crd.nsx.vmware.com/v1alpha1 kind: Subnet metadata: {name: sn-services} spec: {accessMode: PrivateTGW, ipv4SubnetSize: 32} It realized as 172.31.0.0/27, and a VM on it — svc-repo01, 172.31.0.2, serving HTTP — became the shared repo.\nThe test that matters is directional Reachability to the service is the easy claim. From esx01 in each of the three pods (ESXi ships python3, so urllib is the test client):\npod-a esx01 -\u0026gt; http://172.31.0.2/ 200 shared-svc repo01 pod-b esx01 -\u0026gt; http://172.31.0.2/ 200 shared-svc repo01 pod-c esx01 -\u0026gt; http://172.31.0.2/ 200 shared-svc repo01 default-vpc -\u0026gt; http://172.31.0.2/ 200 shared-svc repo01 Three pods with identical source addresses (172.30.0.40) all hit one service and all get answers. How does the reply find its way back to the right pod when three of them claim .40? Because pod traffic crosses the transit gateway SNAT\u0026rsquo;d to a per-VPC transit address. The service never sees 172.30.0.40; it sees three distinct transit IPs. Ambiguity never arises.\nNow the other direction — from svc-repo01 back toward a pod:\nsvc-repo01 -\u0026gt; 172.30.0.40 unreachable svc-repo01 -\u0026gt; 172.30.0.41 unreachable Not \u0026ldquo;blocked\u0026rdquo; — unroutable. Pod subnets are Private, so they were never advertised to the transit gateway, and even if they had been, 172.30.0.40 would be ambiguous across three VPCs. The hub literally cannot initiate into a spoke. For a shared service that will one day be compromised, that\u0026rsquo;s the property you want.\nWhat goes in the hub Anything that\u0026rsquo;s consumed by pods and stateless about which pod is asking: WSUS/patch mirrors, OS and package repos, container registries, NTP, DNS forwarders, license servers. Domain controllers work too, with the usual caveat that identical hostnames across pods need per-pod domains or a naming scheme.\nWhat does not go in the hub: anything that needs to reach into a pod (monitoring pollers, backup agents pulling, jump hosts). Those either live in the pod, or the pod publishes a LoadBalancer VIP for them — the deliberate door from part 0.\nTightening further The transit gateway gives you reachability; policy gives you precision. A VPCGatewayFirewallPolicy on shared-svc can restrict inbound to tcp/80,443 from the transit range and nothing else, so the repo is a repo and not a foothold. I left it open for the test; you shouldn\u0026rsquo;t.\nWhy this matters outside the lab This is the pattern that makes isolated tenants affordable. Without it, every isolated environment needs its own patch server, repository, DNS and directory — cost and drift that quietly kill the idea. With it, a customer runs one set of shared services for dozens of tenants, keeps them patched in one place, and can still show a security reviewer that the shared service has no path back into any tenant.\nThe same hub serves well beyond patching: central logging and monitoring collectors, licence servers, artifact registries, build agents — anything tenants consume but shouldn\u0026rsquo;t be able to be reached by.\nRules learned PrivateTGW is the shared-services mode: addresses from the transit block, advertised to every attached VPC, no collision with pod space. Access is one-way by construction: pods → service works (SNAT\u0026rsquo;d per VPC), service → pod has no route. Test both directions and write down both results. Identical pod addressing and shared services coexist because of the SNAT — the hub sees per-VPC transit addresses, never the overlapping private ones. Same ordering rules apply to the hub VPC (VPC → attachment → LB → namespace → subnets → wait for image sync → workloads). Add a gateway firewall policy on the hub. Reachability is not authorisation. Previously: three datacenters, one IP plan. Next: the whole pod as a single catalog item.\nLab environment; opinions my own. Output captured live, trimmed for length, never edited for outcome.\n","permalink":"https://thenestedlab.com/posts/shared-services-for-isolated-tenants/","summary":"Three pods with identical private addressing all need the same WSUS, repo and AD. One shared-services VPC with a PrivateTGW subnet serves all of them over the transit gateway — and can\u0026rsquo;t reach back into any of them. The directional test, and why SNAT is what makes it work.","title":"Shared services for isolated tenants: PrivateTGW subnets"},{"content":"Everything in this series so far was built with kubectl and API calls. That proves the platform. It doesn\u0026rsquo;t make a product. This post turns the pod into a catalog item: fill in a name, pick a VPC, click Request, and a few minutes later there\u0026rsquo;s a datacenter-in-miniature with two SSH prompts waiting.\nAll Apps in one paragraph VCF Automation 9.1 has two provisioning models side by side. VM Apps is the classic Aria Automation path — cloud templates through an IaaS engine that drives vCenter. All Apps is the supervisor-native path: the blueprint composes Kubernetes objects (a Supervisor Namespace, VM Service VMs, NSX subnets, VKS clusters) and the vSphere Supervisor\u0026rsquo;s controllers reconcile them. A blueprint is formatVersion: 2; its resources are CCI.Supervisor.Namespace and CCI.Supervisor.Resource — the latter is literally \u0026ldquo;here\u0026rsquo;s a manifest, apply it in that namespace.\u0026rdquo;\nThat makes the blueprint a composition of the manifests from the earlier posts, with two additions: inputs, and dependsOn.\nThe blueprint, section by section Inputs — the form inputs: podName: {type: string, default: nested-pod, pattern: \u0026#39;^[a-z0-9]([-a-z0-9]*[a-z0-9])?$\u0026#39;} vpcName: {type: string, description: Must exist and be Realized before deploying.} esxOva: {type: string, default: vmi-61bb062ddfc506b79} # Nested ESXi 9.1 appliance isoImage: {type: string, default: vmi-39f562e2ae9e9c501} # the ISO to attach vmClass: {type: string, default: best-effort-large, enum: [best-effort-large, best-effort-xlarge, best-effort-2xlarge]} The namespace — with libraries attached namespace: type: CCI.Supervisor.Namespace properties: generateName: ${input.podName}- # NOT name — new namespaces get a suffix className: large regionName: f06 vpcName: ${input.vpcName} # pins the namespace to the pod\u0026#39;s VPC storageClasses: [{name: vSAN Default Storage Policy, limit: 400000Mi}] zones: [{name: domain-c9, cpuLimit: 40000M, memoryLimit: 64000Mi, ...}] contentSources: - {name: ISO, type: ContentLibrary} - {name: f06-vks-lib01, type: ContentLibrary} contentSources is the line that closes the gap a lot of first attempts hit: a VCFA-created namespace has no content library, so there are no VirtualMachineImages and nothing can be deployed. Declaring the libraries here attaches them at creation.\nThe topology — ordered on purpose snTrunk: {type: CCI.Supervisor.Resource, properties: {context: ${resource.namespace.id}, manifest: \u0026lt;Subnet sn-trunk\u0026gt;}} snMgmt: {dependsOn: [snTrunk], ... manifest: \u0026lt;Subnet sn-mgmt\u0026gt;} snVmotion: {dependsOn: [snMgmt], ... manifest: \u0026lt;Subnet sn-vmotion\u0026gt;} bmMgmt: {... manifest: \u0026lt;SubnetConnectionBindingMap sn-mgmt -\u0026gt; sn-trunk, vlan 1610\u0026gt;} bmVmotion: {... manifest: \u0026lt;SubnetConnectionBindingMap sn-vmotion -\u0026gt; sn-trunk, vlan 1611\u0026gt;} The dependsOn chain is the whole reason every pod has identical CIDRs: fresh VPCs realize subnets in creation order, and the blueprint fixes that order.\nThe hosts — dual-NIC, ISO attached, bootstrapped by OVF esx01: type: CCI.Supervisor.Resource dependsOn: [bmMgmt] # no point booting before VLAN 1610 exists properties: manifest: kind: VirtualMachine # vmoperator.vmware.com/v1alpha5 spec: hardware: cdrom: [ ... the ISO, declared, connected ... ] network: interfaces: [ eth0 -\u0026gt; sn-trunk, eth1 -\u0026gt; sn-trunk ] bootstrap: vAppConfig: [ guestinfo.hostname / ipaddress / vlan / ... ] wait: fields: [{path: status.powerState, value: PoweredOn}] (Abridged — the full resource carries the image references, VM class, guest ID and the complete guestinfo set.)\nTwo vNICs, both on the trunk — the nested equivalent of a VCF host\u0026rsquo;s two pNICs. The ISO rides along as a declarative CD-ROM. And the wait block makes the deployment\u0026rsquo;s completion mean something: the request doesn\u0026rsquo;t finish until the host is powered on.\nThe doors — one VIP per host A VirtualMachineService of type LoadBalancer per host, selecting it by label and publishing 22 and 443, and a blueprint output that reads the VIP back out of the service\u0026rsquo;s status:\noutputs: esx01Ssh: {value: \u0026#34;ssh root@${resource.esx01Access.object.status.loadBalancer.ingress[0].ip}\u0026#34;} The outputs surface in the deployment view — the requester gets the SSH command, not a scavenger hunt.\nThe request flow, end to end.\nWhat the blueprint cannot express (yet) Three cluster-scoped objects have no blueprint resource type, and they must exist before the request — in this order:\nVPC — privateIPs: 172.30.0.0/16, same in every pod VPCAttachment — connectivity profile with the service gateway; the LB creation fails loudly without it LoadBalancer — silently, permanently required before the namespace Today that\u0026rsquo;s a short script or a runbook step per pod. The honest framing: the blueprint is the pod; the VPC is the tenancy, and tenancy is still created one layer up. I\u0026rsquo;d expect that layer to become blueprintable; until then, keep the three calls next to the blueprint in version control.\nPublishing: one version at a time Blueprint → BlueprintVersion → release. Validation happens at version time, not create time, and the result lives in status.validationMessages rather than the HTTP code — a 200 with ContentValid: False is a thing. And only one version can be published: unrelease 1.0.0 before releasing 1.1.0, or you get a 409. (The full list of sharp edges is its own post.)\nWhy this matters outside the lab This is where platform engineering turns into a service. The difference between \u0026ldquo;we can build you an environment\u0026rdquo; and \u0026ldquo;request one from the catalog\u0026rdquo; is the difference between days and minutes — and between a bespoke build and one that is consistent, quota-controlled and recorded every time. For an organisation that means:\nTime-to-environment measured in minutes, requested by the people who need it, without a queue. Consistency by construction — every environment comes from the same definition, so support, training material and runbooks all match. Governance built in — quotas, ownership, history and clean teardown are properties of the deployment record, not a spreadsheet. The nested-ESXi pod is one catalog item. The same approach delivers any environment shape: application stacks for developers, sandboxes for a proof of concept, demo kits for a sales team, isolated builds for a partner.\nRules learned All Apps blueprints are compositions of manifests: CCI.Supervisor.Namespace plus CCI.Supervisor.Resource per object. If it works with kubectl, it works in a blueprint. generateName, not name, for the namespace; contentSources to attach libraries at creation; zones/storageClasses flat, not wrapped. dependsOn is how you get deterministic CIDRs — order the subnets. wait.fields turns \u0026ldquo;request complete\u0026rdquo; into \u0026ldquo;host is powered on\u0026rdquo;. VPC / VPCAttachment / LoadBalancer are prerequisites outside the blueprint, in that order, before every request. One published version per blueprint; validation in status, not the HTTP response. Previously: shared services for isolated tenants. This closes the Pod Papers\u0026rsquo; core arc — the companion posts on dual-NIC, no-DHCP bootstrap and blueprint gotchas fill in the details.\nLab environment; opinions my own. Blueprint nested-esxi-pod 1.1.0 is live in the lab catalog; YAML above trimmed for length.\n","permalink":"https://thenestedlab.com/posts/nested-esxi-via-vcfa-all-apps/","summary":"The whole isolated pod — namespace, trunk subnets, binding maps, two dual-NIC nested ESXi hosts with an ISO attached, SSH/HTTPS VIPs — as one VCF Automation blueprint, published to the catalog. Anatomy of the blueprint, the ordering it enforces, and the three things it can\u0026rsquo;t express.","title":"A datacenter in a catalog tile: nested ESXi pods via VCF Automation All Apps"},{"content":"\u0026ldquo;Naturally, a VCF host has at least two NICs. Are we testing that, or have you virtualised it away?\u0026rdquo;\nFair question, and the honest answer has two halves. In a nested lab the physical redundancy is provided by the outer host — its vDS, its NSX uplinks — and a second vNIC on the nested VM adds precisely none. But VCF doesn\u0026rsquo;t know it\u0026rsquo;s nested. Bringup\u0026rsquo;s host validation and the vDS uplink teaming it configures expect two vmnics, and a host with one gets flagged. So the nested hosts get two vNICs, both on the trunk subnet, and the question becomes: does failover between them actually work inside a VPC?\nThe setup Both vNICs attach to the same sn-trunk subnet — the trunk from part 1 — and ESXi sees them as two 10G vmnics:\nvSwitch0 teams them active/active with the default originating-port-ID policy; every portgroup (Management 1610, vMotion 1611, vSAN 1612) inherits it. Nothing you wouldn\u0026rsquo;t do on metal.\nThe test: pull a NIC while watching from inside The interesting bit isn\u0026rsquo;t whether pings continue — it\u0026rsquo;s which session I\u0026rsquo;m watching from. I\u0026rsquo;m SSH\u0026rsquo;d to esx01 through its public VIP, which means my session traverses the NSX LB → the VPC → the trunk port → whichever vmnic happens to carry vmk0. If failover breaks anything, it breaks the terminal I\u0026rsquo;m typing in.\n[root@esx01-a:~] esxcli network nic list Name ... Admin Status Link Status Speed MAC Address vmnic0 ... Up Up 10000 04:50:56:00:5c:03 vmnic1 ... Up Up 10000 04:50:56:00:68:00 [root@esx01-a:~] esxcli network nic down -n vmnic0 # FAIL THE FIRST NIC vmnic0 ... Down Down 0 04:50:56:00:5c:03 vmnic1 ... Up Up 10000 04:50:56:00:68:00 [root@esx01-a:~] vmkping -I vmk1 172.30.0.71 # vMotion VLAN, now over vmnic1 3 packets transmitted, 3 packets received, 0% packet loss [root@esx01-a:~] vmkping -I vmk2 172.30.0.101 # vSAN VLAN, now over vmnic1 3 packets transmitted, 3 packets received, 0% packet loss [root@esx01-a:~] esxcli network nic up -n vmnic0 # restore # session never dropped. Every VLAN moved to vmnic1. Zero loss on vMotion and vSAN. And the management session — the one most likely to notice — never blinked.\nWhy this is a real result, not a party trick Think about what just happened at the NSX layer. vmk0\u0026rsquo;s MAC — a MAC ESXi synthesised, not the vNIC\u0026rsquo;s — was being learned on trunk port A. When vmnic0 went down, the same MAC appeared on trunk port B mid-flow, with an established TCP session riding on it.\nOn a standard VPC subnet port that is exactly the scenario SpoofGuard exists to stop: the port\u0026rsquo;s address bindings pin one MAC, and a frame from a different MAC — or the same MAC arriving on a different port — is dropped. Part 1 showed that killing the host on a standard subnet before it ever spoke. This test shows the trunk subnet tolerating the live migration of a foreign MAC between two of its ports, which is the property nested vSphere (and anything else with a vSwitch inside a VM) fundamentally needs.\nSo the second vNIC buys three things, none of them physical redundancy:\nBringup and vLCM stop complaining about a single-uplink host. The teaming policy you\u0026rsquo;ll configure in production gets exercised — uplink failover, active/standby for vSAN, whatever you\u0026rsquo;re rehearsing. A live proof that the trunk carries MAC mobility, which is the real assurance that the design isn\u0026rsquo;t relying on a quiet network. What it does not buy, and how to say so If someone asks \u0026ldquo;is this host redundant?\u0026rdquo;, the answer is \u0026ldquo;the nested host believes it is; actual redundancy lives one layer down.\u0026rdquo; In a training pod that\u0026rsquo;s the correct and useful answer — students configure and test failover exactly as they would on metal, and the outer platform does the real work. In a reproduction lab for a customer NIC-teaming issue, it\u0026rsquo;s usually enough too: most teaming bugs are in ESXi\u0026rsquo;s policy handling, not in the copper.\nWhere it\u0026rsquo;s genuinely insufficient: anything about physical link behaviour — LACP negotiation, LLDP, flapping, MTU mismatch on one uplink. The virtual fabric never fails asymmetrically, so it can\u0026rsquo;t reproduce those.\nWhy this matters outside the lab The practical value here is knowing what a nested environment can and can\u0026rsquo;t prove — which is what lets you decide when a virtual lab is enough and when it isn\u0026rsquo;t. For training, upgrade rehearsals, configuration and policy testing and the vast majority of \u0026ldquo;how does it behave when…\u0026rdquo; questions, nested is enough and dramatically cheaper. For physical link behaviour — LACP, optics, asymmetric faults — you still want metal. Being able to make that call confidently is worth more than the test itself.\nRules learned Give nested VCF hosts two vNICs on the same trunk subnet. Bringup, vLCM and vDS teaming expect ≥ 2 vmnics; humouring them costs nothing. Test failover from a session that depends on it (SSH via the VIP). Pings passing while your terminal dies is not success. The trunk subnet tolerates a vmk MAC moving between ports mid-flow — that\u0026rsquo;s the property standard subnets lack and nested vSphere needs. Be precise in the write-up: nested dual-NIC gives policy realism, not physical redundancy. Physical link faults can\u0026rsquo;t be reproduced here. Rebuilds re-run the vmk config: the appliance creates vmk0 only; vmk1/vmk2, their VLANs and override gateways are applied post-boot. Companion to nested ESXi inside an NSX VPC.\nLab environment; opinions my own. Output captured live, trimmed for length, never edited for outcome.\n","permalink":"https://thenestedlab.com/posts/dual-nic-nested-hosts/","summary":"VCF wants two pNICs per host. In a nested lab the second vNIC adds no physical redundancy — so why add it? Because bringup validation and uplink teaming expect it, and because the failover test tells you something real about the trunk. vmnic0 down, 0% loss, and the SSH session watching it never dropped.","title":"Dual-NIC nested hosts: what redundancy means when the fabric is virtual"},{"content":"The second trap from part 1 deserves its own short post, because it catches everything, not just ESXi: a VPC subnet has DHCP deactivated. Drop a stock appliance onto one and it will boot, sit at \u0026ldquo;waiting for DHCP\u0026rdquo;, and wait politely until the heat death of the universe.\nThis isn\u0026rsquo;t a gap. It\u0026rsquo;s the model: NSX allocates the address at the port and pins it there with address bindings; the guest has to be told what it was given. The VM Service does that telling through bootstrap providers, and once you know the three of them, static addressing stops being a chore and starts being a feature.\nThree providers, three guest types Guest Provider Carries Linux cloudInit user-data (users, write_files, runcmd) + network config Windows sysprep unattend XML / sysprep spec, identity, network Appliances (OVF) vAppConfig OVF properties (guestinfo.*) the appliance reads on boot All three are typed fields on the VirtualMachine object, not bolt-on customisation specs. The network side is already known to the platform — the VM Service knows which subnet each interface landed on and what NSX allocated — so for cloud-init and sysprep the addressing is injected for you. Appliances are the exception, because each one has its own idea of which properties it wants.\nAppliances: vAppConfig The nested-ESXi appliance reads guestinfo.* OVF properties. In the VM Service spec that\u0026rsquo;s:\nbootstrap: vAppConfig: properties: - {key: guestinfo.hostname, value: {value: \u0026#34;esx01.pod-a.res.lab\u0026#34;}} - {key: guestinfo.ipaddress, value: {value: \u0026#34;172.30.0.40\u0026#34;}} - {key: guestinfo.netmask, value: {value: \u0026#34;255.255.255.224\u0026#34;}} - {key: guestinfo.gateway, value: {value: \u0026#34;172.30.0.33\u0026#34;}} - {key: guestinfo.vlan, value: {value: \u0026#34;1610\u0026#34;}} - {key: guestinfo.dns, value: {value: \u0026#34;10.20.52.1\u0026#34;}} - {key: guestinfo.ssh, value: {value: \u0026#34;True\u0026#34;}} The one that trips people: the address you give must be the one NSX allocated to the port. In a standard subnet that\u0026rsquo;s enforced by SpoofGuard; on a trunk subnet the VLAN subnets have their own allocations and you\u0026rsquo;re choosing addresses within them. Either way, pick from the realized range — and remember recreating a VM reallocates its addresses, so the fixed .40/.41 in a deterministic pod is a design choice, not luck.\nLinux: cloud-init A Secret holding user-data, referenced from the VM:\nbootstrap: cloudInit: cloudConfig: users: [ ... a local user with a key ... ] write_files: - path: /var/www/html/index.html content: \u0026#34;shared-svc repo01\\n\u0026#34; runcmd: - [systemctl, enable, --now, nginx] Networking arrives via the platform\u0026rsquo;s own network-config; you don\u0026rsquo;t write it. The svc-repo01 VM from the shared-services post was exactly this — write_files + runcmd, web page verified from three pods.\nWindows: sysprep bootstrap: sysprep: sysprep: guiUnattended: {autoLogon: true, autoLogonCount: 1, timeZone: 85} identification: {joinWorkgroup: WORKGROUP} userData: {fullName: Lab, orgName: Lab, computerName: {name: win01}} Or rawSysprep with an unattend XML in a Secret if you already have one. The ISO from the blueprint post can ride along as a declarative hardware.cdrom — handy for tools and agents on first boot.\nThe ESXi footnote: one gateway, many vmks Once the appliance is up and you add vMotion and vSAN vmks on their own subnets, you hit a detail that makes the Host Client look wrong:\nESXi\u0026rsquo;s default TCP/IP stack has one default gateway — vmk0\u0026rsquo;s .33 — and the UI repeats it on every vmk row. Same-subnet vMotion never uses a gateway so nothing breaks, but each NSX subnet does have its own gateway, and cross-subnet traffic from vmk1/vmk2 would take the wrong exit. Set per-vmk override gateways:\nesxcli network ip interface ipv4 set -i vmk1 -t static -I 172.30.0.70 -N 255.255.255.224 -g 172.30.0.65 esxcli network ip interface ipv4 set -i vmk2 -t static -I 172.30.0.100 -N 255.255.255.224 -g 172.30.0.97 Now the display is truthful and the routing is correct. (The full-realism alternative is a dedicated vmotion netstack for vmk1; I kept the default stack so the service tags stay visible in the Host Client.)\nWhy this matters outside the lab For the business, \u0026ldquo;no DHCP\u0026rdquo; translates into something security and operations teams both want: predictable addressing. Every environment has a known address plan, firewall rules can be written once, and nothing turns up on the network with an address nobody expected. Bootstrap providers deliver the second benefit — images stay generic and configuration is injected at deploy time, so there are fewer golden images to maintain and far less drift between environments.\nRules learned No DHCP in VPC subnets — by design. NSX allocates at the port; the guest is told via a bootstrap provider. cloudInit (Linux), sysprep (Windows), vAppConfig (appliances) — typed fields on the VM, not customisation specs. Appliance addresses must match the realized subnet; fix the order of subnet creation if you want fixed addresses across pods. ESXi has one default gateway per stack. Set -g per vmk, or the Host Client lies to you and cross-subnet traffic exits wrong. A static IP plan is a feature in a lab: it\u0026rsquo;s what makes screenshots, runbooks and pods identical. Companion to nested ESXi inside an NSX VPC.\nLab environment; opinions my own.\n","permalink":"https://thenestedlab.com/posts/vpc-subnets-have-no-dhcp/","summary":"The nested-ESXi appliance sat at \u0026lsquo;waiting for DHCP\u0026rsquo; forever. VPC subnets don\u0026rsquo;t hand out addresses — the VM Service does, through bootstrap providers. cloud-init for Linux, sysprep for Windows, OVF guestinfo for appliances, and the per-vmk gateway detail that makes the Host Client tell the truth.","title":"VPC subnets have no DHCP — and that's fine"},{"content":"The nested-esxi-pod blueprint works. Getting there took seven distinct \u0026ldquo;ContentValid: False\u0026rdquo; (or worse: a 200 that quietly did nothing). None of them are in the docs I could find; all of them are five-minute fixes once you know. Here they are, in the order they bit.\n1. ${input.x} is illegal inside a flow mapping This looks like valid YAML and valid blueprint syntax:\n- {key: guestinfo.hostname, value: {value: \u0026#34;esx01.${input.podName}.res.lab\u0026#34;}} It fails content validation. The expression parser doesn\u0026rsquo;t reach into flow-style ({...}) mappings. Block style is fine:\n- key: guestinfo.hostname value: value: esx01.${input.podName}.res.lab Mixed style in the same list is fine too — only the entries that carry an expression need to be block-style. (This is why the blueprint\u0026rsquo;s vAppConfig list looks inconsistent; it\u0026rsquo;s deliberate.)\n2. name vs generateName for a new namespace A CCI.Supervisor.Namespace you\u0026rsquo;re creating must use generateName. metadata.name is rejected by the CCI API — the platform appends a random suffix, so pod-a- becomes pod-a-dgf5p. Everything downstream should reference ${resource.namespace.id}, never a literal name.\n3. Zones and storage classes are flat Early attempts wrapped them the way the raw CCI API does:\ninitialClassConfigOverrides: zones: [...] In a blueprint they\u0026rsquo;re top-level properties of the namespace resource:\nzones: - name: domain-c9 cpuLimit: 40000M memoryLimit: 64000Mi storageClasses: - name: vSAN Default Storage Policy limit: 400000Mi And zones are required — omit them and the API says \u0026ldquo;Zone should be specified\u0026rdquo;, which at least is a clear message.\n4. A new namespace has no content library Deploy the namespace, deploy a VM, and get: no VirtualMachineImage found. A VCFA-created namespace attaches no content libraries by default. The fix is one block:\ncontentSources: - {name: ISO, type: ContentLibrary} - {name: f06-vks-lib01, type: ContentLibrary} Without it you\u0026rsquo;re in the vSphere Client attaching libraries to a namespace by hand, which rather defeats the catalog.\n5. Images sync after attach — wait for status.disks Even with libraries attached at creation, the first VM create in a fresh namespace can be rejected:\nno disks found in image ... status.disks The image objects appear immediately; their disk metadata syncs over the next 1–3 minutes. The quota webhook checks status.disks and refuses until it\u0026rsquo;s populated. In a blueprint, put the hosts dependsOn something that takes a couple of minutes (the binding maps did the job here), or add an explicit wait. In a script, poll:\nkubectl get virtualmachineimage -n \u0026lt;ns\u0026gt; \u0026lt;vmi\u0026gt; -o jsonpath=\u0026#39;{.status.disks}\u0026#39; 6. Validation lives in status, not the HTTP code Creating a BlueprintVersion returns 200 whether or not the content is valid. Read the object back:\nstatus: contentValid: false validationMessages: - \u0026#34;... unexpected token ...\u0026#34; If your pipeline checks the response code, it will happily publish a broken blueprint. Check status.contentValid and print the messages.\n7. Only one published version — 409 on the second Release 1.1.0 while 1.0.0 is released and you get a 409. It isn\u0026rsquo;t a transient conflict; it\u0026rsquo;s the rule. Unrelease the current version, then release the new one. Practically that means a publish step is unrelease old → release new, and there\u0026rsquo;s a short window where the catalog item has no released version. Do it when nobody\u0026rsquo;s requesting.\nBonus: the things that aren\u0026rsquo;t blueprint problems Three prerequisites have no blueprint resource type and have to exist before the request — VPC, VPCAttachment, LoadBalancer, in that order. The blueprint\u0026rsquo;s vpcName input says \u0026ldquo;must exist and be Realized\u0026rdquo;, and it means it. Nothing in the blueprint fails if they\u0026rsquo;re missing; the deployment just never gets a VIP.\nWhy this matters outside the lab VCF Automation\u0026rsquo;s All Apps model is new, and new platforms have edges. None of these seven are documented; all of them stall a first project by days if you meet them cold. The value of a delivery partner who has already built on the platform isn\u0026rsquo;t the YAML — it\u0026rsquo;s that a customer\u0026rsquo;s first blueprint publishes on day one instead of week two, and that the sharp edges are encoded into templates and provisioning scripts where users never meet them.\nRules learned Expressions need block-style YAML; flow mappings don\u0026rsquo;t get parsed. generateName, and reference the namespace by ${resource.x.id}. zones and storageClasses are flat and zones are required. contentSources on the namespace, or nothing can be deployed. Wait for image status.disks before the first VM (1–3 min). Check status.contentValid — the HTTP code lies by omission. One released version per blueprint: unrelease, then release. Companion to a datacenter in a catalog tile.\nLab environment; opinions my own. Error text captured live.\n","permalink":"https://thenestedlab.com/posts/cci-blueprint-gotchas/","summary":"Everything that made the nested-esxi-pod blueprint fail validation before it worked: ${input} inside flow mappings, name vs generateName, flat zones, contentSources, one-published-version, validation-in-status, and the image-sync race. Short, specific, and each one cost me a cycle.","title":"Blueprinting the supervisor: seven CCI blueprint gotchas"},{"content":"VCF Automation 9.1 has two ways to build things, side by side, in the same organisation. If you\u0026rsquo;ve come from Aria Automation you\u0026rsquo;ll recognise one of them immediately. The other looks like Kubernetes because it is Kubernetes. Choosing between them isn\u0026rsquo;t a matter of taste — they have genuinely different shapes, and some use cases are natural in one and awkward in the other.\nI\u0026rsquo;ve now pushed the same list of requirements through both on a live VCF 9.1 lab. This is the comparison I wish I\u0026rsquo;d had at the start.\nThe two shapes VM Apps — the classic Aria Automation model:\nOrg ─ Project ─ Cloud Zone(s) │ Cloud Account (vCenter / NSX) │ flavor · image · network · storage profiles │ Cloud Template (formatVersion 1) ──▶ IaaS engine ──▶ vCenter API Cloud.vSphere.Machine, Cloud.NSX.Network, customization spec Provisioning is imperative through a broker. The IaaS engine holds the vCenter session; the tenant references abstractions (flavors, image mappings) that resolve at request time. State lives in the IaaS database.\nAll Apps — supervisor-native, via the Cloud Consumption Interface:\nOrg ─ CCI Project ─ Region │ VPC (tenant-created; overlapping privateIPs allowed) │ + VPCAttachment + LoadBalancer (before the namespace!) Supervisor Namespace (spec.vpcName pins it) │ Kubernetes objects reconciled by the Supervisor: VirtualMachine / VirtualMachineService / Subnet / BindingMap / VKS Cluster │ Blueprint (formatVersion 2, CCI.Supervisor.*) → BlueprintVersion → Catalog Provisioning is declarative. The blueprint states desired objects; the supervisor\u0026rsquo;s controllers converge reality onto them and keep it there. State lives in etcd. Networking is NSX VPC-native.\nScorecard by use case Every row below was actually built, not read about.\nUse case VM Apps All Apps Linux VM + software Cloud.vSphere.Machine + cloudConfig VM Service VM + cloud-init (typed field) Windows VM customization spec + software components bootstrap.sysprep; ISO attachable declaratively ISO / CD-ROM attach day-2 edit in vCenter hardware.cdrom in the manifest Multi-NIC NICs across network profiles multiple interfaces on VPC subnets; failover verified Self-service catalog template released to catalog blueprint → version → publish (one live version) Load-balanced access NSX LB via network profile VirtualMachineService → VPC LB VIP Kubernetes clusters separate TKG integration native Cluster object in the namespace Identical isolated environments hard: on-demand NAT nets, unique addressing usually forced natural: VPC per pod, overlapping CIDRs, zero route between Nested ESXi / VLAN networks trunk portgroups on the physical vDS — a fabric change trunk subnet + binding maps — no fabric change Shared infra (WSUS, repos) shared segment routed everywhere one PrivateTGW subnet, one-way reachability Extensibility vRO, ABX, event broker (rich) controllers, GitOps, kubectl (thinner day-2 today) Deep per-device vSphere tuning anything vCenter can do what the VM Service API models The two bold rows are the ones that decided it for me. Both are documented in this series; both are genuinely hard in VM Apps and simply the default in All Apps.\nWhere VM Apps still wins Be fair to the incumbent:\nYears of content. vRO workflows, ABX actions, template libraries and a mature day-2 action framework carry over unchanged. If you have an estate of them, that\u0026rsquo;s real value you\u0026rsquo;d be throwing away. Deep vSphere reach. Anything vCenter can do to a VM — RDMs, per-device tuning, exotic customization — the IaaS engine can do, because it drives vCenter directly. Familiar network model. Segments, portgroups, on-demand routed/NAT networks from a network profile. No new mental model. Multi-cloud lineage. The same template idiom stretched to other endpoints. The Windows Server 2025 pipeline elsewhere on this blog is a VM Apps build, and it\u0026rsquo;s a good one: Event Broker hooks for hostname allocation and placement metadata, cloudbase-init staging, a reboot-safe state machine in the guest. Nothing about it needs rewriting.\nWhere All Apps wins, and why it\u0026rsquo;s structural Declarative and self-healing. Desired state in etcd, controllers reconcile. There\u0026rsquo;s no second database to drift from vCenter. Structural multi-tenancy. A VPC per tenant is a hard NSX boundary, not an administrative one. Overlapping CIDRs are allowed, so cookie-cutter environments deploy side by side. VMs and Kubernetes are one model. The same blueprint composes a VKS cluster, VMs, secrets and networking. GitOps-able with ordinary tools. VPC networking is first-class. Trunk subnets, binding maps, PrivateTGW, per-VPC gateway firewall — none of it has a VM Apps equivalent. Modern bootstrap. cloud-init and sysprep as typed API fields. Where All Apps hurts today (all observed live) Ordering matters and the errors are opaque. The LB must exist before the namespace; new namespaces reject VMs until images sync; blueprint validation has seven sharp edges. The tenancy layer isn\u0026rsquo;t blueprintable. VPC, VPCAttachment and LoadBalancer are cluster-scoped CCI objects with no blueprint resource type — scripted, not catalogued. Two endpoints. The CCI proxy serves tenancy objects; workload manifests go to the supervisor. You\u0026rsquo;ll hold two kubeconfigs. Day-2 is thinner. No event broker; extensibility means controllers and GitOps, which is fine if that\u0026rsquo;s your team and a gap if it isn\u0026rsquo;t. Recommendation Default to All Apps for new build-outs. Every use case on the list — including the two that are genuinely hard in VM Apps — is natural in the VPC model, and the whole estate is version-controlled YAML. Keep VM Apps as the compatibility surface for existing vRA content and for the rare thing that needs direct vCenter device manipulation. They coexist per org, so migration is incremental and nobody has to rewrite a working pipeline on a deadline.\nAnd whichever you pick: codify the ordering rules into the scripts that provision tenancy, so the sharp edges stay encapsulated and the people requesting catalog items never meet them.\nWhy this matters outside the lab Most customers arriving at VCF 9 have an estate of Aria Automation content and a question: rewrite, coexist, or migrate? The answer above is the one we take into design workshops. In practice it plays out as an assessment — which existing templates and workflows still earn their keep, which use cases are genuinely better served by the VPC model, and where the boundaries sit — followed by a coexistence plan that moves workloads across opportunistically instead of on a deadline. The two architectures sharing one organisation is what makes that low-risk.\nRules learned Two shapes, not two skins: imperative-through-a-broker vs declarative-reconciled. Pick per use case, not per org. All Apps is structurally better at isolation with identical addressing and nested/VLAN networks without fabric changes. VM Apps is still the home for existing vRO/ABX content and deep vCenter device work. All Apps\u0026rsquo; pain is ordering: VPC → attachment → LB → namespace → subnets → image sync → workloads. Script it once. They coexist; migrate opportunistically. Lab environment; opinions my own. Grounded in a VCF 9.1 / vSphere Supervisor with NSX VPC networking build-out; every row was deployed.\n","permalink":"https://thenestedlab.com/posts/vm-apps-vs-all-apps/","summary":"VCF Automation 9.1 ships two provisioning architectures in one org. I ran the same use cases through both — Linux and Windows VMs, ISO attach, multi-NIC, isolated pods, nested ESXi, shared services, a catalog item. The honest scorecard, and how each use case is actually achieved.","title":"VM Apps vs All Apps: a field comparison of VCF Automation's two provisioning models"},{"content":"The Cluster manifest is the same. That\u0026rsquo;s the point of this post, and also the punchline: VKS is VKS whichever door you walk through. What differs is everything wrapped around the cluster — who can ask for it, what limits it, who can see it, and how it shows up in operations tooling.\nSo: two clusters, one supervisor, one ClusterClass, two paths.\nPath A: kubectl, the way we\u0026rsquo;ve always done it kubectl vsphere login --server \u0026lt;supervisor\u0026gt; --tanzu-kubernetes-cluster-namespace demo kubectl apply -f cluster.yaml apiVersion: cluster.x-k8s.io/v1beta1 kind: Cluster spec: clusterNetwork: { pods, services, serviceDomain } # set all three — see below topology: class: builtin-generic-v3.6.0 classNamespace: vmware-system-vks-public # the gotcha version: v1.35.5+vmware.1 controlPlane: {replicas: 1} workers: { one node pool, 2 replicas } variables: [ vmClass, storageClass ] Fifteen minutes later: a cluster. Requires a vSphere namespace that somebody (an admin) created, with a content library attached, a VM class assigned and quota set — all in the vSphere Client, by hand.\nPath B: the same manifest, as a catalog request In All Apps the cluster is a CCI.Supervisor.Resource inside a blueprint, sitting next to a CCI.Supervisor.Namespace:\nresources: namespace: type: CCI.Supervisor.Namespace properties: generateName: ${input.name}- className: large # quota comes from the class regionName: f06 vpcName: ${input.vpc} contentSources: [{name: f06-vks-lib01, type: ContentLibrary}] cluster: type: CCI.Supervisor.Resource properties: context: ${resource.namespace.id} manifest: \u0026lt;the SAME Cluster object as above\u0026gt; Publish it, and a tenant user requests it from a tile:\nFifteen minutes later: a cluster. Byte-identical Cluster object.\nWhat path B adds for free kubectl All Apps request Who can create anyone with a kubeconfig to that namespace org users with catalog entitlement (org RBAC) Namespace pre-created by an admin, by hand created by the request, from a class Quota set per namespace in the vSphere Client inherited from the namespace class (small/medium/large) Content library admin attaches manually contentSources on the blueprint Networking whatever the namespace has pinned to a tenant VPC by vpcName Sizing choices edit YAML form inputs with enums (worker count, VM class) Record kubectl get cluster a deployment with inputs, owner, history, day-2 actions Visibility supervisor only VCFA inventory and VCF Operations That last row is the one operations teams care about:\nThe VCFA-deployed cluster appears in Ops\u0026rsquo; object model — supervisor → namespace → cluster → nodes → the workloads running on it — and its demo apps show up by name in the topology tab. The kubectl-built cluster is also visible to Ops (it\u0026rsquo;s the same supervisor, after all), but it has no deployment, no owner, no request history and no quota lineage. It\u0026rsquo;s a thing that exists, not a thing that was provided.\nThe four gotchas, in order of how much time they cost Both paths share the same four traps on VKS 1.35 / VCF 9.1:\nA VCFA-created namespace has no content library. Zero VirtualMachineImages → no cluster possible. contentSources in the blueprint, or attach by hand. ClusterClass lives in vmware-system-vks-public. It 404s from the workload namespace unless the spec sets topology.classNamespace. The default Quick Start namespace is 1000M CPU / 1000Mi. Unusable for a cluster. Use a real class: small 10000M/10000Mi, medium 20000M, large 40000M. VKS 1.35 enforces PodSecurity restricted by default. Demo apps that run as root get a ReplicaSet and no pods; the events say FailedCreate. Label the app namespace pod-security.kubernetes.io/enforce=privileged (or fix the apps). Plus one that only shows up later: set clusterNetwork.serviceDomain explicitly. It\u0026rsquo;s immutable after create, and a cluster without it produces a service DNS name that some add-ons build wrongly (....svc. with no domain). And pick a pod CIDR that doesn\u0026rsquo;t shadow your VPC\u0026rsquo;s external range — the stock 192.168.0.0/16 hid the org\u0026rsquo;s 192.168.144.0/21 from inside the cluster.\nSo which one? Use kubectl when you\u0026rsquo;re the platform team proving something on a supervisor, or debugging. Use All Apps the moment a second person needs a cluster: the request form is the interface, the namespace class is the guardrail, the deployment is the audit trail, and Ops sees it as a provided service rather than a stray object.\nThe cluster\u0026rsquo;s the same either way. The service isn\u0026rsquo;t.\nWhy this matters outside the lab The business case for the second path is governance without friction. Development teams get Kubernetes clusters on request; the platform team gets quotas, ownership, RBAC and a monitoring view of every cluster for free. That\u0026rsquo;s the difference between a managed Kubernetes service and a collection of clusters nobody can account for — and it\u0026rsquo;s typically the gap that stops organisations offering Kubernetes broadly at all. Everything the developers touch stays standard Kubernetes; the control lands around it, not on it.\nRules learned The Cluster object is identical across paths — All Apps wraps it, it doesn\u0026rsquo;t change it. What All Apps adds: catalog RBAC, class-based quota, library attach, VPC pinning, deployment history, and a place in the Ops object model. Four traps on 1.35: no default library, classNamespace, tiny default quota, PodSecurity restricted. Set serviceDomain and a non-shadowing pod CIDR at create; both are immutable. Next in All Apps in Practice: the demo apps that live on this cluster.\nLab environment; opinions my own.\n","permalink":"https://thenestedlab.com/posts/vks-kubectl-vs-vcfa-all-apps/","summary":"I built the same VKS cluster twice on the same supervisor: once with kubectl apply, once as a VCF Automation All Apps request. The Cluster object is identical. Everything around it isn\u0026rsquo;t — and the payoff table is what you get for free the second way: catalog, quota class, org RBAC, VCF Operations visibility.","title":"One VKS cluster, two ways: kubectl vs VCF Automation All Apps"},{"content":"Every nested VCF lab starts the same way: a heroic PowerShell script. Ours was esxihostdeploy.ps1 — ovftool plus PowerCLI, an interactive menu asking which environment, which ESX version, which role, how many hosts. It worked. It also lived on one person\u0026rsquo;s machine, prompted for credentials, and knew nothing about everything that comes after the hosts exist.\nThis post is about what it became: a set of VCF Automation catalog items where requesting one form produces a complete nested VCF 9.1 instance — ESXi hosts, bringup (vCenter, NSX, SDDC Manager), a vSphere Supervisor, VCF Automation, Operations, identity — with the environment number as practically the only real input.\nThe shape of the factory Three stages, each a catalog item, plus a wrapper that chains them:\nStage 1 Nested ESX Hosts VM Apps template + vRO actions (the old script, reborn declaratively) Stage 2 Deploy VCF 9.1 Instance vRO workflow driving the VCF Installer API (spec generated, validated, bringup started) Day-N Supervisor · NSX Edge · VCF Automation · Ops Logs/Networks/RTM · Identity (AD) one catalog item each Wrapper \u0026#34;Deploy VCF Stack\u0026#34; one form, checkbox per component The wrapper\u0026rsquo;s form has a checkbox per component; ticking one reveals that component\u0026rsquo;s tab with every field pre-populated. One lab password feeds every credential. Tick everything, click request, and go get coffee.\nRule 1: derive everything from one number Each lab environment is f0X, and everything scales from X by formula:\nElement Pattern f03 example Names f0X-m01-* f03-m01-vc01.res.lab Subnets 10.(20+X).\u0026lt;sub\u0026gt;.0/24 10.23.1.0/24 (mgmt) VLANs 2X0n 2307 (edge TEP) The bringup spec — hundreds of lines of JSON the VCF Installer wants — is generated by a vRO action from a known-good reference spec plus X. Nobody edits a deployment spec by hand, which means nobody typo-breaks a bringup at 2am. When the old script did this, the formulas lived in string concatenation; now they live in one action with the reference spec beside it.\nThe same philosophy carried into stage 1: the script\u0026rsquo;s \u0026ldquo;next free esxNN index\u0026rdquo; scan became a vRO action bound to the request form, its VLAN/IP arithmetic became template expressions, its --prop:guestinfo.* flags became ovfProperties in the template. Porting a script isn\u0026rsquo;t rewriting it — it\u0026rsquo;s finding the declarative home for each behaviour.\nRule 2: never wait for anything you can watch instead The hard constraint that shaped the whole design: a VCF Automation request gets about two hours before the platform gives up on it. A full VCF bringup takes longer than that. So the wrapper never waits:\nBringup is fire-and-forget — the workflow authenticates to the installer, validates the spec, starts the task, and hands back a watchTaskId. Re-attach any time to check on it. Fleet deployments (VCF Automation, Ops for Logs/Networks, metrics) are server-side tasks; the items submit with waitForCompletion=false. The supervisor item submits enablement and returns; vCenter carries on. Only fast, deterministic steps (identity configuration, minutes) run to completion inside the request. A whole VCF instance as one deployment record — the request finished in under an hour while the build ran on for twelve.\nResult: a full-stack kick-off completes as a request in well under an hour (53 minutes on the run pictured), while the actual multi-hour build continues as watchable server-side tasks. The request\u0026rsquo;s job isn\u0026rsquo;t to do the work — it\u0026rsquo;s to start the work correctly and tell you where to watch it.\nThe corollary: a failed component is recorded and the remaining components still run. You fix one thing and re-run one item, not the world.\nRule 3: plan mode for infrastructure Every item in the chain supports validateOnly — and the wrapper cascades it. Tick everything, set validateOnly, and the entire stack is planned against the live environment with zero changes: specs generated, prerequisites checked, name/IP collisions caught. A smoke runner exercises exactly this on every change to the automation itself.\nIf you build nothing else into your lab automation, build this. The number of 2am bringups saved by a five-minute dry run is not small.\nThe traps that shaped the rules Some of the design above exists because of scars:\nHardware validation hates virtual NVMe. Bringup\u0026rsquo;s HCL check will block nested hosts; the spec generator has to account for it, or you discover it two hours in — twice, if you\u0026rsquo;re us. Small disks, surprising layouts. Nested hosts with 64 GB disks ship ESX-OSDATA at essentially the whole disk; a post-provision step relocates scratch or stage-2 fills the disk with logs. DNS is a prerequisite, not a step. The installer\u0026rsquo;s pre-flight wants every record resolvable before it starts; a one-shot script creates the per-environment records ahead of the request. vRO\u0026rsquo;s content-source lag. A new or changed workflow takes 15–20 minutes of data-collection before the catalog sees it. Publish, wait, then test — or you\u0026rsquo;ll debug a ghost. Wrapper inputs are duplicated by necessity. vRO requires every sub-workflow input to be passed explicitly, so adding an input to a component means updating the wrapper\u0026rsquo;s call too. Null-guards in each component turn a forgotten field into a loud failure instead of a silent default. Why bother? Because the payoff compounds. Once a full VCF instance is a catalog request, everything downstream changes character: upgrade rehearsals happen on freshly-built instances instead of precious pets; a broken environment is redeployed, not repaired; and the lab stops being a snowflake collection and becomes a product — versioned, validated, reproducible.\nWhere this is heading: the factory\u0026rsquo;s output feeding per-student pods with identical addressing.\nThe factory\u0026rsquo;s next customers, funnily enough, are the isolated VPC pods from the other series on this blog — same philosophy, one layer further down.\nWhy this matters outside the lab A complete VCF instance from one form changes what an environment costs to have. Environments that used to be precious — because building one took a week — become disposable, and a lot follows from that:\nProofs of concept run on an environment built for the customer\u0026rsquo;s scenario, not on whatever happens to be free. Upgrade and migration rehearsals happen on a fresh instance of the right version, then it\u0026rsquo;s deleted. Training and enablement get a real VCF per person or per team. Reference builds exist for every supported release, on demand. This is how Comms-care provides a dedicated instance to every consultant. The same factory, pointed at a customer\u0026rsquo;s requirements, is a repeatable way to deliver environments rather than a one-off project each time.\nRules learned Derive names, subnets and VLANs from a single environment number; generate specs, never hand-edit them. Respect the request-duration ceiling: start long work, return a task handle, re-attach to watch. Never block a wrapper on an hours-long task. validateOnly on every item, cascaded by the wrapper — dry-run the whole stack before touching anything. Componentise failure: one broken step re-runs alone. Pre-create DNS; expect HCL friction on virtual hardware; budget for vRO\u0026rsquo;s content-source lag. Lab environment; opinions my own. The automation described builds nested VCF 9.1 instances for lab and rehearsal use — patterns transfer, specifics are ours.\n","permalink":"https://thenestedlab.com/posts/one-catalog-item-one-vcf-instance/","summary":"How an interactive PowerShell script grew into a catalog-driven factory that stands up complete nested VCF 9.1 instances — hosts, bringup, supervisor, fleet components — from a single request form. The design rules that made it survivable, and the traps that shaped them.","title":"One catalog item, one VCF instance: building a lab factory"},{"content":"Terraform has plan. Kubernetes has --dry-run=server. Ansible has --check. Every mature infrastructure tool grew a way to say \u0026ldquo;tell me what you\u0026rsquo;d do, then don\u0026rsquo;t\u0026rdquo; — because the alternative is finding out at 2am, two hours into a bringup, that a hostname doesn\u0026rsquo;t resolve.\nvRO workflows don\u0026rsquo;t come with one. This is the case for adding it to every single one you write, and cascading it through every wrapper.\nThe shape Every catalog item in the lab factory has a boolean input, validateOnly, default false. When true the workflow does everything it can without changing anything:\nauthenticate to every endpoint it would touch resolve every name it would use, and fail on the ones that don\u0026rsquo;t generate every spec it would submit, and run the target\u0026rsquo;s own validation API on it where one exists (the VCF Installer has one; use it) check for collisions — names, IPs, existing objects report what it would have created, then return CREATE_SUCCESSFUL The wrapper — the one form that chains hosts, bringup, supervisor, fleet components, identity — has the same checkbox, and cascades it to every component. Tick everything, tick validateOnly, request. Thirty seconds to a few minutes later you have a full-stack plan against the live environment, and nothing has moved.\nWhat it caught Not hypothetically. On a built environment, the cascaded dry run of the whole stack reported:\nedge cluster: already exists — correctly recorded, wrapper carried on Ops for Logs: IP_IN_USE on the planned address — right, it\u0026rsquo;s deployed Ops for Networks: same supervisor: the existing one would be reused; the per-service plan listed which services were already active identity: bind succeeded, group resolved, no changes needed That\u0026rsquo;s a plan output. On a fresh environment the same run has caught, at various times: a DNS record missing for one of ~40 required names (the installer\u0026rsquo;s own pre-flight found it, in seconds, instead of bringup finding it in hour two); a stale content-library image ID; a form field arriving null because a custom form hadn\u0026rsquo;t finished re-importing — which is a publishing bug the dry run surfaced before anyone requested anything real.\nThe argument against, answered \u0026ldquo;It doubles the code.\u0026rdquo; It doesn\u0026rsquo;t — it moves the if (!validateOnly) guard around the mutating call, and the validation logic is code you should have had anyway. What it does force is separating \u0026ldquo;compute what to do\u0026rdquo; from \u0026ldquo;do it\u0026rdquo;, which is how the workflows should have been structured in the first place.\n\u0026ldquo;Some things can\u0026rsquo;t be validated without doing them.\u0026rdquo; True. Say so in the result summary — \u0026ldquo;would deploy X; no pre-validation available\u0026rdquo; — rather than skipping the item. Partial plans are still plans.\n\u0026ldquo;We have a test environment.\u0026rdquo; You have a test environment. A dry run against the target is what catches the collision with the thing that\u0026rsquo;s already there.\nMake it the smoke test The best consequence: a validateOnly request against a known environment is a regression test for the automation itself, runnable on every change. The factory\u0026rsquo;s smoke runner does exactly this — request every item with validateOnly: true, assert CREATE_SUCCESSFUL, diff the plan summary against the last run. It takes minutes and it has caught more bugs in the workflows than any amount of code review.\nThe same form, real or dry-run. One checkbox decides.\nWhy this matters outside the lab For anyone who has sat through a failed change window, the value is obvious: a full dry run against the real estate before anything moves. Fewer failed changes, shorter windows, and a plan output that answers the change board\u0026rsquo;s questions before they\u0026rsquo;re asked. It also gives auditors something they rarely get from infrastructure automation — evidence of what was going to happen, produced by the same tooling that then did it.\nRules learned Add validateOnly to every workflow. Default false. Wrappers cascade it. Dry-run does everything but mutate: auth, resolve, generate, call the target\u0026rsquo;s validator, check collisions, report. Where a step truly can\u0026rsquo;t be pre-validated, say so in the summary. Never skip it silently. A dry run against the real target is a plan. A dry run on every change is a smoke test. Same checkbox. The refactor it forces — compute, then act — is the one you wanted. Part of The Lab Factory.\nLab environment; opinions my own.\n","permalink":"https://thenestedlab.com/posts/validateonly-everywhere/","summary":"Terraform has plan. Kubernetes has \u0026ndash;dry-run. Your vRO workflows have nothing — unless you give them a validateOnly input and make the wrapper cascade it. A short argument for the single most valuable checkbox in the lab factory, with the failures it caught.","title":"validateOnly everywhere: plan mode for infrastructure"},{"content":"Every lab has one: the script that builds the nested hosts. Ours was esxihostdeploy.ps1 — ovftool plus PowerCLI, an interactive menu for environment, ESX version, role, size, host count, then a loop of ovftool --prop:guestinfo.*, Set-VM, New-NetworkAdapter, Set-HardDisk. It worked for years. It also prompted for credentials, lived on one machine, and knew nothing about the bringup that came after.\nPorting it to a VCF Automation catalog item (VM Apps — a cloud template plus vRO) is not a rewrite. It\u0026rsquo;s a sorting exercise: every behaviour in the script has a natural home in the declarative model, and the skill is finding it. Here\u0026rsquo;s the whole table, then the four rows that took thought.\nThe mapping Script behaviour Where it lives now Environment menu (f01–f10) environment input (enum) Version menu → OVA path on a share esxVersion input → image mapping → content library item Role menu (management / workload / both) role input; \u0026ldquo;both\u0026rdquo; = two requests Size menu / auto-detect from an existing host size input (auto-detect dropped — see below) vCenter + ESXi credential prompts vCenter creds gone (cloud account); esxiRootPassword an encrypted input Next-free esxNN index scan (gap-filling) vRO action getNextEsxHostIndexes, bound to the request form VLAN / IP / gateway arithmetic template expressions (same formulas) ovftool --prop:guestinfo.* ovfProperties on Cloud.vSphere.Machine Folder lookup / New-Folder allocation-phase subscription creates the folder if missing Set-VM cpu / mem cpuCount / totalMemoryMB in the template 2× New-NetworkAdapter (Vmxnet3) networks array, deviceIndex 0/1/2 3× Set-HardDisk grow post-provision subscription NestedHVEnabled = $true post-provision subscription \u0026ldquo;Power on after?\u0026rdquo; prompt powerOn input, honoured post-provision Summary table printed at the end the deployment view in the UI Five homes, in decreasing order of preference: input, template expression, platform abstraction (image mapping, network profile, cloud account), form action (read-only lookup at request time), subscription (imperative work at a lifecycle stage). Push each behaviour as far up that list as it will go.\nThe four decisions that weren\u0026rsquo;t obvious 1. Drop auto-detect; the platform already remembers The script inspected an existing host to infer size. That was a workaround for having no record. The catalog is the record — deployment history shows what size every existing host was requested at — so the input simply asks. Fewer moving parts, and the requester sees the choice.\n2. The index scan is a form action, not a workflow step \u0026ldquo;Next free esx07\u0026rdquo; has to be known at request time so the requester sees the names they\u0026rsquo;ll get. That\u0026rsquo;s a vRO action bound to the custom form (getNextEsxHostIndexes(environment, role, count) → array of strings), not a step inside provisioning. Forms can call actions; use it for anything that\u0026rsquo;s a lookup.\n3. Rename and folder go in Compute Allocation, hardware goes in Post Provision Two blocking subscriptions, filtered by a custom property on the template:\nSubscription 1 Subscription 2 Topic Compute allocation Compute post provision Runnable \u0026ldquo;Set VM Name \u0026amp; Folder\u0026rdquo; \u0026ldquo;Finalize Hardware\u0026rdquo; Does sets resourceNames, creates folder grows 3 disks, NestedHVEnabled, power on Timeout 10 min 30 min The rename must be at allocation — it\u0026rsquo;s the only stage where a workflow output named resourceNames is applied to the machine. Disk growth and nested-HV need a VM that exists, so they wait for post-provision. Both are blocking: the deployment doesn\u0026rsquo;t proceed until they return.\n4. nestedEsx: 'yes' — not true The subscription condition is event.data.customProperties.nestedEsx == \u0026quot;yes\u0026quot;. Why not \u0026quot;true\u0026quot;? Because boolean-looking strings can arrive in the event payload as typed booleans, and true == \u0026quot;true\u0026quot; is false in the condition evaluator and in the vRO code. It fails silently — the subscription just never fires. yes can\u0026rsquo;t be coerced. Small thing; two hours.\nWhat stayed exactly the same The formulas. 10.(20+X).\u0026lt;sub\u0026gt;.0/24, VLAN 2X0n, gateway .254 — they were string concatenation in PowerShell and they\u0026rsquo;re template expressions now, character for character. The guestinfo.* property names — identical, because the OVA didn\u0026rsquo;t change. Porting a script well means most of it survives; only the plumbing moves.\nThe bits that still bite Network profile without IP ranges. Addressing is injected via guestinfo, not the platform\u0026rsquo;s IPAM. Tag the trunk portgroup, add no ranges, or IPAM and guestinfo will disagree. Two template revisions in the repo. v1 is what the guide documents; v2 grew later. Both kept deliberately, both labelled. Check which one the org has imported before editing either. Content-source lag. A new or changed vRO action needs ~15–20 minutes of data collection before the form sees it. Publish, wait, then test. Small disks and OSDATA. Nested hosts with 64 GB disks ship ESX-OSDATA at essentially the whole disk. Templates now provision 128 GB; a relocate-scratch script mitigates existing hosts. Every menu prompt from the script is now a field; Host Indexes is the form action\u0026rsquo;s answer to \u0026ldquo;next free esxNN\u0026rdquo;.\nWhy this matters outside the lab Almost every organisation has these scripts: valuable, trusted, and stuck on one person\u0026rsquo;s machine. The message of this post for them is that modernising doesn\u0026rsquo;t mean rewriting. The logic survives; what changes is where it lives — behind a request form with access control, an audit trail, consistent inputs and a deployment record. That\u0026rsquo;s how a team turns tribal knowledge into a service without losing the years of edge cases the script already handles.\nRules learned Porting is sorting: input → expression → platform abstraction → form action → subscription. Push each behaviour as far up as it goes. Lookups the requester needs to see are form actions. Rename at allocation (resourceNames output); hardware at post-provision. Both blocking. Filter subscriptions on a custom property, and make its value a word that can\u0026rsquo;t be coerced to a boolean. Drop workarounds for missing state; the catalog is the state. Keep the formulas. Move the plumbing. Part of The Lab Factory. Next: driving the VCF Installer API from vRO.\nLab environment; opinions my own.\n","permalink":"https://thenestedlab.com/posts/porting-a-powershell-deploy-script/","summary":"esxihostdeploy.ps1 was 400 lines of ovftool and PowerCLI behind a menu. It became a cloud template, two vRO actions and two subscriptions — and the interesting part is deciding where each behaviour belongs. The full mapping, four non-obvious decisions, and the \u0026lsquo;yes\u0026rsquo; that isn\u0026rsquo;t \u0026rsquo;true\u0026rsquo;.","title":"Porting a PowerShell deploy script to a catalog item: the mapping table is the post"},{"content":"The nested hosts exist. Now they need to become a VCF instance: vCenter, NSX, SDDC Manager, the fleet components. The VCF 9.1 Installer appliance does that from a deployment spec — a few hundred lines of JSON — through an API. This post is the vRO workflow that drives it, and the three design constraints that shaped it: nobody edits the spec by hand, the request can\u0026rsquo;t outlive two hours, and nested hosts fail hardware validation.\nUnlike stage 1 this isn\u0026rsquo;t VM provisioning, so it\u0026rsquo;s not a cloud template. It\u0026rsquo;s a vRO workflow published directly as a catalog item through an Orchestrator content source.\nThe spec is generated, never edited A vRO action, buildVcfDeploymentSpec(environment, hostFqdns, labPassword, …), returns the whole spec as a string. Its structure was reconciled against a validated export from a real bringup — the installer UI lets you export the spec it accepted — and everything variable derives from the environment number X:\nElement Pattern f03 Names f0X-m01-* f03-m01-vc01.res.lab Subnets 10.(20+X).\u0026lt;sub\u0026gt;.0/24 10.23.1.0/24 (mgmt) VLANs 2X0\u0026lt;sub\u0026gt; 2301 mgmt … 2306 TEP Gateways .254 10.23.1.254 vMotion / vSAN ranges .1–.16 10.23.3.1-16 NSX TEP pool .6.1–.6.32 10.23.6.1-32 SDDC Manager f0X-vcf01.res.lab f03-vcf01.res.lab var n = parseInt(environment.substring(1), 10); // \u0026#34;f03\u0026#34; -\u0026gt; 3 var pfx = environment + \u0026#34;-m01\u0026#34;; var net = \u0026#34;10.\u0026#34; + (20 + n); function vlan(o) { return 2000 + (n * 100) + o; } function gw(sub) { return net + \u0026#34;.\u0026#34; + sub + \u0026#34;.254\u0026#34;; } Static across environments: DNS, NTP, subdomain, component sizes, vSAN ESA FTT=1, and the component build versions pinned to the installer binaries. One lab password feeds every credential field (the UI export scrubs them; the action puts them back per the API schema).\nTwo spec-level decisions worth stealing:\nskipEsxThumbprintValidation: true instead of carrying per-host sslThumbprint. Supported, and the right trade-off for a lab. Ops and Automation are checkboxes that add their blocks to the spec — and the installer only accepts a licenseServerSpec when Ops is present, so the action adds them together or not at all. The workflow: authenticate → validate → start → return 1. POST /v1/tokens installer login 2. POST /v1/sddcs/validations the installer\u0026#39;s OWN pre-flight on the spec (poll until COMPLETED; fail on any FAILED check) 3. if validateOnly -\u0026gt; return the validation report; touch nothing 4. POST /v1/sddcs start bringup -\u0026gt; sddcTaskId 5. return { sddcTaskId, installerUrl } Step 2 is the validateOnly story made concrete: the installer will tell you, in seconds, that f03-m01-nsx01.res.lab doesn\u0026rsquo;t resolve, that an IP is in use, that a host isn\u0026rsquo;t reachable. Two hours into a bringup is a bad time to learn that. Every one of the ~40 DNS records the pre-flight wants is created ahead of time by a one-shot PowerShell script (New-LabEnvDnsRecords.ps1) — DNS is a prerequisite, not a step.\nThe two-hour leash, and how to slip it A request from the catalog carries a token with a roughly two-hour lifetime, and a bringup takes around eight. So by default the workflow is fire-and-forget: waitForCompletion=false, return the task id, watch progress in the installer UI. A watchTaskId input lets you re-attach later and poll an already-running bringup from a new request.\nThe wrapper that chains everything — hosts, bringup, then the day-N components that need bringup to be finished — has a neater trick. The catalog-bound parent deploys the hosts, submits bringup, and then re-executes itself as a plain vRO run (Orchestrator → Run, no catalog token, no two-hour kill) carrying the hidden bringupWatchTaskId. That continuation polls the installer task to completion — eight hours, fine — and then runs certificates, fleet items, edge, supervisor and identity. Watch it under Orchestrator → Activity → Runs. The catalog request itself completes in under an hour — 47 minutes on the run pictured below — having started the work correctly and handed off.\nTwo rows, one build. The parent returns inside the catalog\u0026rsquo;s window; the continuation waits out the bringup and does the day-N work.\nNested-host frictions Three things a physical bringup never meets:\nHCL validation vs virtual NVMe. The installer\u0026rsquo;s hardware check blocks the virtual NVMe controller. Fix at the vLCM layer: enforce_hcl_validation=false on the image policy. The vSAN health test nvmeonhcl also complains; silenced via the vSAN API, best-effort with manual fallback. DVS compatibility appears late. After bringup, NSX takes 1–2 hours to settle before the supervisor\u0026rsquo;s zones endpoint stops returning 500. If the supervisor stage fails \u0026ldquo;No compatible DVS\u0026rdquo; on a fresh instance, wait and re-run just that item. TSM-SSH. Bringup wants SSH on the hosts; the wrapper enables it host-direct via SOAP before submitting. Stale schema: the failure that looks like a bug and isn\u0026rsquo;t Add an input to the vRO workflow after the catalog item exists and the form will show the new field, the request will record its value, and the workflow will receive null — Service Broker keeps the old request schema until the content source re-imports. The workflow null-guards every boolean and aborts with \u0026ldquo;inputs not mapped\u0026rdquo; rather than running with silently-wrong options. Fix: re-import the content source, confirm the schema, submit a new request (resubmitting an old one reuses the old payload).\nThere are actually three async layers between \u0026ldquo;publish\u0026rdquo; and \u0026ldquo;mappable request\u0026rdquo; — vRO processing the import, the catalog schema after re-import, and the form service still enforcing the previous custom form for a minute or two. Same symptom for all three. Check timing before assuming a bug.\nWhy this matters outside the lab Repeatable, generated VCF deployments matter well beyond a lab: a second site, a disaster-recovery instance, a new business unit, an environment per supported release. Generating the specification from a validated reference removes the class of errors that comes from editing hundreds of lines of JSON by hand, and running the installer\u0026rsquo;s own validation first turns \u0026ldquo;find out in hour two\u0026rdquo; into \u0026ldquo;find out in minute one\u0026rdquo;. It\u0026rsquo;s the difference between a VCF deployment being a project and being a procedure.\nRules learned Generate the spec from a validated export plus one number. Nobody hand-edits JSON at 2am. Run the installer\u0026rsquo;s own validation first, and make it a mode you can request on its own. Pre-create DNS. Enable SSH. Disable HCL enforcement on virtual NVMe. Respect the request lifetime: start, return a task id, re-attach. For a long chain, have the workflow re-run itself outside the catalog. Null-guard every input and fail loud; stale schemas are a fact of life after adding inputs. On a fresh instance, give NSX an hour before you expect DVS compatibility. Part of The Lab Factory. Previously: porting the host script.\nLab environment; opinions my own. Bringup verified end-to-end on a rebuilt environment: 305/305 tasks, COMPLETED_WITH_SUCCESS.\n","permalink":"https://thenestedlab.com/posts/driving-the-vcf-installer-api-from-vro/","summary":"Stage 2 of the lab factory: a vRO workflow that turns an environment number into a complete VCF 9.1 deployment spec, runs the installer\u0026rsquo;s own validation, starts bringup and hands back a task id — because the request dies long before the eight-hour build does. Plus how the wrapper slips the two-hour leash.","title":"Driving the VCF Installer API from vRO: generate, validate, start, walk away"},{"content":"I\u0026rsquo;m Adam Stretton — a Lead Consultant who has spent the last decade designing, deploying and automating VMware Cloud Foundation platforms for enterprise customers, usually several layers of virtualisation deep.\nThe day job is end-to-end cloud transformation: VCF architecture and deployment across private and hybrid estates; day-zero-to-day-two automation with VCF/Aria Automation, PowerShell and infrastructure-as-code; and governance, lifecycle and observability with VCF/Aria Operations. The thread through all of it is the same — take the manual overhead out of infrastructure delivery, standardise it, and give teams self-service they can trust. I\u0026rsquo;ve found that works when technical precision, process and the people involved move together, which is why a lot of what I do is bridging operations, architecture and development teams rather than just building things.\nThis blog is the lab notebook behind that work. Certifications, for the curious: credly.com/users/adam-stretton.\nWhat this blog is VCF 9.x platform engineering, proven live. Nested ESXi, NSX VPC networking, VCF Automation \u0026ldquo;All Apps\u0026rdquo;, VKS, the observability stack — and the failure modes the documentation doesn\u0026rsquo;t mention. Every post follows one rule: the commands, output and error messages you see were captured from a running environment. Trimmed for length, never edited for outcome. Where a capture is synthetic (a test-mode run, a simulated scrape) the caption says so.\nThe posts group into series, and the series read in order:\nThe VPC Pod Papers — what an NSX VPC is, why nested ESXi blackholes inside one, and the trunk-subnet design that turns cookie-cutter isolated environments into a catalog item. The Lab Factory — a complete nested VCF 9.1 instance from one request form: hosts, bringup, day-N, validateOnly everywhere. All Apps in Practice — VKS clusters and demo stacks through VCF Automation\u0026rsquo;s supervisor-native path. Observability on VCF and LLM Ops on VCF — fluent-bit, Telegraf and a vLLM metrics pipeline into VCF Operations. Dark Site Notes — air-gapped VKS and GPU Operator upgrades, and everything that has to cross the airlock intact. The Windows Build Pipeline — a reboot-safe Windows Server 2025 provisioning state machine on Aria Automation. The lab None of this would exist without the lab, and the lab exists because of Comms-care, where I work. A team there builds, runs and keeps improving a shared physical VCF platform on which every consultant gets their own dedicated nested VCF instance — a complete environment, not a slice of a shared one. The automation that stamps those instances out is a team effort too; the Lab Factory series describes it, but plenty of hands made it work. Mine is f06; the identifiers you\u0026rsquo;ll see throughout the posts (res.lab, 172.30.0.0/16, 192.168.144.x) are that instance\u0026rsquo;s own.\nEach instance is a full VCF 9.x stack — vCenter, NSX, SDDC Manager, the fleet components, a vSphere Supervisor with NSX VPC networking — running on nested ESXi hosts, built and rebuilt by the automation described in The Lab Factory. A fresh instance is a catalog request away, which changes how you treat it: it\u0026rsquo;s somewhere to break things on purpose.\nThat\u0026rsquo;s what makes it useful well beyond one blog:\nCustomer demonstrations on the real product. When a customer wants to see VCF Automation\u0026rsquo;s catalog provision an isolated environment, or a VKS cluster appear in VCF Operations, they see it running — on the same release they\u0026rsquo;re deploying, not a slide. Proving use cases before they hit production. The isolated-pod design, the shared-services VPC, the observability pipelines in these posts were all worked out here first, with the failure modes found and documented in the lab rather than on a customer\u0026rsquo;s change window. Upgrade rehearsals. A per-consultant instance means an upgrade path can be rehearsed end to end — and torn down and rehearsed again — before the runbook is trusted with a live estate. Learning by breaking. Because every consultant has their own, nobody is sharing blast radius. The blackholed vmk0 in part 1 of the Pod Papers cost an afternoon of my lab, and no one else\u0026rsquo;s. The posts are mine; the platform that made them possible belongs to the Comms-care team, and I\u0026rsquo;m grateful for it. Nothing here is a customer environment.\nDisclaimer This is a personal blog. Opinions are my own and not my employer\u0026rsquo;s or Broadcom\u0026rsquo;s. Everything described here was performed in a lab environment; test before you trust anything in production. Product names are the property of their respective owners.\nIf something here saved you an afternoon — or cost you one because I got it wrong — tell me.\n","permalink":"https://thenestedlab.com/about/","summary":"Who writes this, what the lab is, and the one rule every post follows.","title":"About"},{"content":"The best way to reach me is LinkedIn — linkedin.com/in/adam-stretton-32471943. Connection requests with a note about which post brought you here get a faster reply than blank ones.\nOr email: adam.stretton [at] comms-care.com .\nWhat I\u0026rsquo;m glad to hear about Corrections. If a post says something that isn\u0026rsquo;t true on your build, tell me the version and what you saw. I\u0026rsquo;d rather fix it than be quoted wrong. \u0026ldquo;It behaved differently for me.\u0026rdquo; Especially on the ordering rules — those are observed behaviour on specific releases, and releases move. Follow-up questions that are really \u0026ldquo;have you tried X?\u0026rdquo; — often the answer is no and it becomes a post. What I can\u0026rsquo;t do Support your production environment. Everything here is lab work; Broadcom Support is the right door for anything you\u0026rsquo;re paying for. Share customer material. Techniques are generalised deliberately. ","permalink":"https://thenestedlab.com/contact/","summary":"Corrections, questions, and the occasional argument about subnet ordering.","title":"Contact"}]