VCF Automation on The Nested Lab

Every resource in the VCF Automation 9.1 blueprint designer: the complete guide

Thu, 01 Oct 2026 00:00:00 +0100

The blueprint designer in a VCF Automation 9.1 All Apps organization offers sixteen items in three groups. Drag one onto the canvas and you get a few lines of YAML. What you may write underneath them is spread across the VM Service, VKS, NSX VPC and Automation guides. For some items, it’s written down nowhere at all.

This guide puts it in one place. For each item, you get:

what it creates, and the YAML behind it;
every field the platform accepts;
a minimal snippet, and recipes for the common jobs;
the status fields worth reading;
the traps we hit.

Three things make it more than a list from memory, which, given my memory, is just as well:

The field lists come from the platform. VCF Automation publishes the schema of every blueprint resource type through its API. The designer carries the schema of every palette item. The Supervisor publishes the definition of every Kubernetes kind it serves, and VCF Automation’s VPC API publishes its own. Where they disagree (and they do), the tables follow what the platform enforces.
Every snippet was checked. Each one went through VCF Automation’s validation API, and every Kubernetes manifest through a server-side dry run on the Supervisor. Five test blueprints (all of them in the downloads) deployed every item for real. The remaining recipes are ones our lab catalog deploys every day.
The traps are real. Several of the most useful lines in this guide come from things that went wrong while testing: a NAT rule that ignored its port, a firewall rule that grew an “Any”, a request that waits for an address that can never come.

We checked it on our lab platform, f06:

VCF Automation 9.1.0;
a Supervisor on Kubernetes 1.32.9, with the VM Operator API at v1alpha5;
VKS with ClusterClasses up to builtin-generic-v3.6.0, and Kubernetes releases up to 1.35.5;
NSX VPCs.

Names in the examples (f06, nested-pod, vpc-student05, vsan-default-storage-policy) are ours; use yours.

In the field tables, Values gives a field’s choices and default. Where the schema has neither, it gives an example, marked e.g.: the value from our tested blueprints wherever one of them set the field, otherwise a typical one. For an object or a list, the example is a short YAML flow value built from its fields (... marks the ones left out). And <...> stands for a name of yours.

The shape of a blueprint

An All Apps blueprint is YAML with formatVersion: 2, its inputs, its resources and its outputs:

formatVersion: 2
inputs:
  vmName:
    type: string
    title: VM name
    default: web-01
    pattern: '^[a-z0-9]([-a-z0-9]*[a-z0-9])?$'
  size:
    type: string
    title: Size
    default: small
    oneOf:
      - {title: Small, const: small}
      - {title: Medium, const: medium}
resources:
  namespace:
    type: CCI.Supervisor.Namespace
    properties:
      name: team-a-dev
      existing: true
  vm1:
    type: CCI.Supervisor.Resource
    properties:
      context: ${resource.namespace.id}
      manifest:
        apiVersion: vmoperator.vmware.com/v1alpha5
        kind: VirtualMachine
        metadata:
          name: ${input.vmName}
        spec:
          className: ${'best-effort-' + input.size}
          imageName: ubuntu-24.04-server-cloudimg-amd64
          storageClass: vsan-default-storage-policy
outputs:
  vmName:
    value: ${resource.vm1.object.metadata.name}

What the expressions can read:

Expression	Value
`${input.}`	A request input. An input group from a property group is `${input..}`.
`${resource..}`	Another resource’s property; this also orders the two. `object` holds the live Kubernetes object of a Supervisor Resource.
`${propgroup..}`	A constant property group’s value.
`${secret.}`	A VCF Automation secret, from the organization or the project.
`${env.deploymentName}`, `${count.index}`	The deployment’s name; the instance number in a counted resource.
`${to_k8s_name(env.deploymentName, 63)}`	A string made safe for a Kubernetes name, as Broadcom’s own samples use for `generateName`.

Besides type and properties, each resource has dependsOn (an explicit order) and allocatePerInstance (see count below). formatVersion: 2 also allows metadata, variables, and an output named __deploymentOverview, whose Markdown value becomes the deployment’s overview page.

How the palette maps to YAML

Behind the sixteen items there are only five resource types, and one of them isn’t in the palette. Most items are a type with part of its YAML already filled in. For the workload items, that’s a Kubernetes apiVersion and kind; for the VPC items, it’s a VPC configuration kind.

Palette item	YAML `type`	Pre-filled
Supervisor Namespace	`CCI.Supervisor.Namespace`
VPC group
VPC	`CCI.VPC`
VPC Configuration	`CCI.VPC.Configuration`	nothing: you set `kind`
Attachment	`CCI.VPC.Configuration`	`kind: VPCAttachment`
IP Address Allocation	`CCI.VPC.Configuration`	`kind: VPCIPAddressAllocation`
NAT Rule	`CCI.VPC.Configuration`	`kind: VPCNATRule`
Group	`CCI.VPC.Configuration`	`kind: VPCNetworkSecurityGroup`
Gateway Firewall Policy	`CCI.VPC.Configuration`	`kind: VPCGatewayFirewallPolicy`
Workload group
Supervisor Resource	`CCI.Supervisor.Resource`	nothing: any manifest
Virtual Machine	`CCI.Supervisor.Resource`	`vmoperator.vmware.com/v1alpha5` `VirtualMachine`
Virtual Machine Group	`CCI.Supervisor.Resource`	`vmoperator.vmware.com/v1alpha5` `VirtualMachineGroup`
Virtual Machine Service	`CCI.Supervisor.Resource`	`vmoperator.vmware.com/v1alpha3` `VirtualMachineService`
Subnet	`CCI.Supervisor.Resource`	`crd.nsx.vmware.com/v1alpha1` `Subnet`
Persistent Volume Claim	`CCI.Supervisor.Resource`	`v1` `PersistentVolumeClaim`
Secret	`CCI.Supervisor.Resource`	`v1` `Secret`
Kubernetes Cluster	`CCI.Supervisor.Resource`	`cluster.x-k8s.io/v1beta1` `Cluster`
(not in the palette)	`Util.PasswordEntry`

Four consequences, worth knowing before any of the detail:

Anything the Supervisor understands can go in a blueprint. The workload items are shortcuts. The generic Supervisor Resource takes any manifest the namespace accepts. Our lab blueprints rely on a kind the palette doesn’t offer, SubnetConnectionBindingMap, to carry VLANs.
The VPC items are a closed list. CCI.VPC.Configuration takes exactly the five kinds above. A VPC’s load balancer is a sixth kind in VCF Automation’s VPC API, and a blueprint can’t make one (see VPC).
VCF Automation validates the outside, the platform the inside. The validation API checks the resource type’s own properties: required fields, patterns, the namespace’s two shapes. It doesn’t look inside a manifest or a configs[].spec: in our test, powerState: Sideways passed validation. Those are checked when the request runs.
The designer’s schemas are not the platform’s. The palette’s forms come from schemas bundled with VCF Automation, while the Supervisor and the VPC API check against their own. They disagree in a handful of places, listed next.

Where the designer and the platform disagree

We compared each palette item’s schema with the platform’s definition of the same apiVersion and kind. We went field by field (every bit as gripping as it sounds) and tested every difference:

Item	The designer offers	What the platform does
Virtual Machine	`affinity.zoneAffinity`, `affinity.zoneAntiAffinity`, and VM affinity terms named `...IgnoredDuringExecution`	Rejects them as unknown fields. VM affinity and anti-affinity take `requiredDuringSchedulingPreferredDuringExecution` and `preferredDuringSchedulingPreferredDuringExecution`.
Virtual Machine	no `linuxPrep.password`, `linuxPrep.scriptText`, `crypto.vTPMMode`, `currentSnapshotName`, or disk options at volume level	Accepts all of them.
Virtual Machine	`bootOptions.firmware` as `BIOS` or `EFI`, and `bootOptions.enterBootSetup`	`spec.bootOptions.firmware: Unsupported value: "BIOS": supported values: "bios", "efi"`, and `strict decoding error: unknown field "spec.bootOptions.enterBootSetup"`.
Virtual Machine `v1alpha4`, `v1alpha3`	`network.nameServers`	The field is `nameservers`, lower case.
Subnet	`regionName`, `ipBlockNames`, `description`	Rejects them. Accepts `vlanConnectionName`, which the designer doesn’t show.
Persistent Volume Claim	`accessMode`	`strict decoding error: unknown field "spec.accessMode"`. The field is `accessModes`.
Kubernetes Cluster	`cluster.x-k8s.io/v1beta1`	Serves it, with `cluster.x-k8s.io/v1beta1 Cluster is deprecated; use cluster.x-k8s.io/v1beta2 Cluster`.
Group	`vmSelectors[].selector`, `podSelectors[].selector`	`spec.vmSelectors[0]: Required value: must specify at least one selector`. The field is `labelSelector`; the API also offers `propertySelector`.
Gateway Firewall Policy	`ruleCount`	Computed by the API; not accepted. A rule’s `from` is required, though the schema says empty means any.

None of this stops the designer from saving a blueprint. It surfaces when the request runs.

What every resource shares

Properties on every type

Property	On	What it does
`count`	all	How many instances to create. Default 1.
`allocatePerInstance`	all (beside `type`)	Required for `${count.index}`: without it the validator refuses the blueprint with `should have allocatePerInstance property set to True`.
`dependsOn`	all (beside `type`)	Explicit order. A reference to another resource orders the two already.
`context`	Supervisor Resource items	Required. The namespace the manifest goes into: `${resource..id}`.
`existing`	Namespace, Supervisor Resource items	`true` adopts what already exists instead of creating it.
`wait`	Supervisor Resource items, VPC Configuration	When the resource counts as done.

Recipe, two Secrets from one resource:

  sec:
    type: CCI.Supervisor.Resource
    allocatePerInstance: true
    properties:
      count: 2
      context: ${resource.ns.id}
      manifest:
        apiVersion: v1
        kind: Secret
        metadata:
          name: ${'ref-s-' + count.index}
        type: Opaque
        stringData:
          note: created by instance ${count.index}

The deployment then holds sec[0] and sec[1], and the namespace ref-s-0 and ref-s-1.

`wait`: when a resource is finished

Without a wait, a Supervisor Resource is finished as soon as the Supervisor accepts the manifest. That’s fine for a Secret, and wrong for a VM whose address an output needs. wait takes conditions, fields, or both:

      wait:
        conditions:                      # status.conditions[] of the object
          - type: Ready
            status: "True"
          - type: Ready
            status: "False"
            reason: Failed
            indicatesFailure: true       # this one fails the resource instead of finishing it
        fields:                          # any field of the object, by path
          - path: status.powerState
            value: PoweredOn
        skipWaitOnDelete: false          # true: deletion does not wait for the object to be gone

A Supervisor Resource can also watch log output with executionLogs: progressMessagePattern while a matching message appears, and failureMessagePattern to fail the resource.

The designer pre-fills a wait for some items:

Item	Designer’s default `wait`
Virtual Machine	condition `VirtualMachineCreated` = `True`
Virtual Machine Group	condition `Ready` = `True`
NAT Rule, Group	condition `Realized` = `True`
everything else	none

The VM’s default came from a 9.0 problem: VM status could come back empty, and Broadcom’s KB 435137 gave this wait as the workaround. That condition is true before the VM is even on, let alone has an address.

VCF Automation also waits by itself in one place: a Virtual Machine Service of type LoadBalancer is not finished until it has an external address, with or without a wait. In a VPC without a load balancer, that address never comes, and the request stays in progress until it times out. Ours was still PARTIAL after ten minutes, with the service on the Supervisor.

Reading a resource’s live state

Every Supervisor Resource exposes the object as the Supervisor sees it under object, and a VPC Configuration exposes its objects under configs:

outputs:
  vmIp:
    value: ${resource.vm1.object.status.network.primaryIP4}
  vmHost:
    value: ${resource.vm1.object.status.nodeName}
  lbIp:
    value: ${resource.webLb.object.status.loadBalancer.ingress[0].ip}
  publicIp:
    value: ${resource.publicIp.configs[0].spec.allocationIPs}

Outputs are computed once, when the request finishes, and not refreshed afterwards. A VM that waited only for PoweredOn finished before its guest reported an address, and its IP output stayed empty for good. Waiting on the condition that marks the guest’s network as configured fixes it. This one gave the output 172.30.0.2:

      wait:
        conditions:
          - type: VirtualMachineGuestNetworkConfigSynced
            status: "True"

Checking a manifest before you deploy it

The fastest check we found is a server-side dry run with strict field validation, against the namespace the blueprint will use:

kubectl apply --dry-run=server --validate=strict -n  -f vm.yaml

It runs the Supervisor’s schema checks and admission webhooks, flags unknown fields, and creates nothing. It works for every workload kind. It does not work for the VPC kinds: VCF Automation’s VPC API ignores dryRun and creates the object, as one of our probes found.

Supervisor Namespace

type: CCI.Supervisor.Namespace. Creates a vSphere Namespace through VCF Automation, inside the project’s allocation, or adopts one that exists. Everything in the Workload group needs one: their context points at it.

The schema has two shapes, and the validator enforces them: an existing namespace takes name and existing: true and nothing else; a new one needs generateName, className, regionName and vpcName.

Property	Notes
`generateName`	New namespace. Prefix; VCF Automation adds `-` and five random characters (`ns-ref-bstk7`). Must match `^[a-z0-9]([-a-z0-9]*[a-z0-9])?$`, so it cannot end in a hyphen.
`className`	New namespace. The namespace class: limits, VM classes, storage classes and content libraries.
`regionName`	New namespace. The region.
`vpcName`	New namespace. The VPC its workloads use: an existing VPC’s name, or `${resource..name}`.
`name` + `existing: true`	Existing namespace. `name` alone matches neither shape.
`zones[]`	Per vSphere Zone: `name`, `cpuLimit`, `cpuReservation`, `memoryLimit`, `memoryReservation`, all five required.
`storageClasses[]`	`name` (the storage policy as the namespace names it) and `limit`.
`vmClasses[]`, `contentSources[]`	VM classes and image sources beyond the class’s own.
`sharedSubnetNames[]`, `infraPolicyNames[]`, `segName`, `description`	Shared subnets, extra infrastructure policies, an Avi Service Engine Group, a description.

Every field the platform accepts (25)

Windows Server 2025 via Aria Automation, part 3: validation as a product, and the details that hurt

Thu, 17 Sep 2026 06:30:00 +0100

Most provisioning pipelines end with “Deployment completed”. This one ends with a document a human can read. It changed how the platform team and the server team talk to each other.

Instead of “it’s built, go check it”, the requester gets a page that says what was built, where it landed, what passed, and how long each step took.

Part 1 was the architecture and part 2 the state machine. This is the payoff. After it come the details that were nowhere in any documentation.

The report

06-validate-build.ps1 is a pure transformation: data-validation.json in, one self-contained HTML file out. There’s no server and there are no external assets. CSS, SVG icons and the timeline chart are all inline, so it opens as-is from the file share or as an email attachment.

The top of the report. Rendered by the real renderer from an anonymised payload: the hostnames, domain and vendor names are fictional, the code that drew it is not. Open the full report (it’s one self-contained HTML file) or the JSON it was built from.

The header shows the VM, execution time, project, deployment and requester. The banner says BUILD SUCCESSFUL / BUILD FAILED, by the strict rule from part 2. A sticky status bar keeps the hostname and verdict in view while you scroll. Then:

Section	Contents
System	hostname, OS, latest patch, CPU, RAM, uptime, time zone, pending reboot
Security & AD	domain, trust status, OU, NTP source; WinRM/RDP/UAC status cards; local admins
Software	one card per agent: pass/fail badge + the individual checks (flag, service, path)
Network	per adapter: alias, MAC, IP, mask (converted from prefix), gateway — joined to the vSphere portgroup via the guestinfo NIC/MAC map
Storage	per volume: capacity bars (warn < 20 % free, critical < 10 %) + backing datastore chips
Tags	every deployment tag as a colour chip (palette chosen deterministically by key hash, so the same tag is always the same colour)
Placement	vCenter → datacenter → cluster → host chain, VM folder; “metadata pending” if guestinfo was never readable
Timeline	SVG Gantt of every step at true wall-clock position; phase colours; automatic REBOOT detection (gaps > 30 s shaded and labelled)
Installed apps	collapsible full inventory

Sections 3 to 5: one card per agent with its three checks; adapters joined to their portgroups; volumes with their backing datastore.

The network table is the one that gets the “oh” reaction. The guest knows its adapters and vCenter knows the portgroups. The guestinfo bridge from part 1 lets one table show both, joined on MAC, with no credentials crossing the boundary.

Tags and placement: all of it from the request and from guestinfo, none of it from a credential in the guest.

The Gantt chart is the one operations teams use. When a build takes 90 minutes instead of 40, the chart shows why. It might be the updates step, or a slow installer. Or it might be a 20-minute gap where the machine sat at a boot prompt, waiting patiently for someone to notice.

Section 8. Phase colours, true wall-clock positions, and the reboot detected from a gap over 30 seconds — nothing logged “rebooting now”.

The details that hurt

Every one of these cost hours, and none of them is in a manual.

vmtoolsd mangles arguments under SYSTEM in Session 0

Reading guestinfo.vra.infrastructure with & vmtoolsd.exe --cmd "info-get guestinfo.vra.infrastructure" works interactively. It fails silently as SYSTEM in a startup task, because the argument quoting gets mangled on the way through.

The fix is to build the process explicitly with System.Diagnostics.Process. Set Arguments as one string, redirect stdout, and read it yourself. Also retry (15 × 10 s), because the vRO subscription that writes the value can land after the guest starts looking.

Ejecting an ISO with no Explorer

Session 0 has no shell, so Shell.Application ejection does nothing. A P/Invoke to winmm.dll:

[mciSend]::mciSendString("set cdaudio door open", $null, 0, [IntPtr]::Zero)

opens the tray. It feels like 1998. It works.

Exit 1003 is the only reboot you should ever request from cloudbase-init

Call Restart-Computer from a cloudbase-init script, and you race the plugin’s own state tracking. Exit 1003 instead. cloudbase-init reboots, runs the part again on next boot, and your flag file short-circuits it.

The installers’ 3010 is a different animal. That one means “success, reboot wanted”, and the build master decides when.

cloudbase-init has to remove itself

A delivered server with cloudbase-init still installed has an unattended execution surface. Anyone who can present a config drive owns the box.

The cleanup phase stops the service, kills the processes, runs the uninstaller silently and deletes the directory. It does all that before validation, so the report can confirm it’s gone.

UAC was off. Turn it back on.

The base image relaxes EnableLUA and FilterAdministratorToken so the build runs without prompts. If the cleanup forgets to restore them, you ship a server with UAC disabled and a report that says SUCCESS with a perfectly straight face. The “UAC enabled” card in the security section exists so this can never be silent.

Secrets: encrypt to the machine, then scrub

The two share passwords are the only secrets that ever touch the guest’s disk. They’re AES-encrypted with a key derived from the BIOS UUID (Win32_ComputerSystemProduct.UUID), so they’re useless off the box. They’re decrypted only in memory, and overwritten with *** SCRUBBED *** before the final reboot.

That’s the right containment for a transient build secret. It’s not a vault, and shouldn’t be described as one.

The requester is told too early

The “your deployment completed” email fires at Compute Post Provision. That’s when vCenter has finished, not when the guest has. So users open a server that’s mid-Windows-Updates, which is nobody’s idea of a warm welcome. Move the email to a deployment-completion topic, or at least include the report’s future share path.

Hostname allocation has a race

Read-then-allocate against AD isn’t synchronised. Two deployments running at the same time can see the same highest suffix and pick the same name. Within one multi-machine deployment, the count-based batch is safe. Across deployments, wrap the search-and-generate in a vRO LockingSystem lock.

What I’d carry to any build pipeline

Strip out the Windows specifics (a fair few of them reboots), and five ideas survive:

The report is the deliverable. Build it from a single JSON document so it’s testable without a build.
Join guest facts to platform facts via a one-way metadata channel.
Detect reboots from timing gaps, not from logging “rebooting now”.
Health is three checks, never the installer’s exit code.
Clean up before you validate, so “clean” is a checkable claim.

Why this matters outside the lab

The report is the part customers remember. It replaces “your server is ready” with evidence:

what was installed, and whether it’s healthy;
where the server landed in vCenter;
how the network was configured;
how long each step took, and where the reboots were.

Service desks use it to close the request. Security teams use it to confirm the controls, and platform teams use the timeline to spot regressions. The same idea (validate, then publish proof) carries over to any provisioning pipeline, Windows or not.

Rules learned

Ship a report, not a status. Self-contained HTML, one JSON source.
Under SYSTEM in Session 0: build processes explicitly, eject media via mciSendString, expect no shell.
Exit 1003 for cloudbase-init reboots; 3010 is the installer’s word for “later”.
Remove cloudbase-init and restore UAC, and make both of them checks in the report.
Machine-keyed encryption + scrub is right for transient secrets. Say what it is.
Fix the two timing bugs: the early requester email, and the hostname race under parallel deployments.

Broadcom documentation

Event topics provided with Automation Assembler: Compute post provision against Deployment completed, for the requester’s email
Create an extensibility subscription: subscribing an Orchestrator workflow to an event topic, with a filter and blocking
Query Information using GuestInfo Variable: vmtoolsd --cmd with info-get and info-set for guestinfo variables
Windows Automation Assembler image for vSphere: how Cloudbase-Init is installed on the template, before the build removes it

Previously: the state machine. This VM is also where the Telegraf and fluent-bit Windows agents land.

Lab write-up of a production pattern; customer specifics removed. Opinions my own.

Windows Server 2025 via Aria Automation, part 2: a state machine that survives four reboots

Thu, 17 Sep 2026 06:20:00 +0100

Part 1 ended with a startup scheduled task registered and 05-build-master.ps1 staged locally. From here, the machine reboots at least twice more. Windows Updates insists, as it tends to, and the build ends with a clean reboot.

Every one of those boots runs the same script. So the script has to converge on a finished build, however many times it runs.

That’s a state machine, and it’s built from three very boring mechanisms.

The three mechanisms

1. Master kill switch. If 100_build_complete.flag exists, exit straight away. A stray task run after completion does nothing.

2. Per-step flags. Every install step writes NN__installed.flag when it succeeds, and is skipped on later runs.

3. Persisted state. data-state.json holds the provisioning start time and the timing of each step, and it’s reloaded on every boot. So the final report shows the true duration of every step across the whole build, not just the last boot. A step seen again as SKIPPED after a reboot never overwrites its real recorded duration.

every boot:
  if 100_build_complete.flag → exit
  load data-state.json
  for step in $softwarePayload:
      if NN_step_installed.flag → SKIP (keep recorded timing)
      run installer from local Staging (timeout ceiling)
      exit 0 or 3010 → write flag, record timing
      if step.RebootAfter → Restart-Computer; exit      ← next boot resumes here
  cleanup → validate → publish → self-destruct → final reboot

The loop above, boot by boot.

Phase 1: software, declaratively

The stack is a single array, and that array is the designed extension point:

$softwarePayload = @(
  @{ Name='MonitoringAgent'; Folder=$p.monPath;  File=$p.monInstallFile;  Timeout=20; Reboot=$false;
     Service='HealthService';       Path='...\Monitoring Agent\HealthService.exe' },
  @{ Name='InventoryAgent';  Folder=$p.invPath;  File=$p.invInstallFile;  Timeout=15; Reboot=$false;
     Service='InventoryAgent';      Path='...\Inventory\agent.exe' },
  @{ Name='EndpointSecurity';Folder=$p.epsPath;  File=$p.epsInstallFile;  Timeout=30; Reboot=$false;
     Service='masvc';               Path='...\Agent\masvc.exe' },
  @{ Name='LogAgent';        Folder=$p.logPath;  File=$p.logInstallFile;  Timeout=15; Reboot=$false;
     Service='LogAgentService';     Path='...\Log Agent\liwinsvc.exe' },
  @{ Name='WindowsUpdates';  Folder=$p.updPath;  File=$p.updInstallFile;  Timeout=45; Reboot=$true }
)

Adding a product means adding an element. Each installer runs from the local staging copy, with a timeout in case it hangs. Exit codes 0 and 3010 (success, reboot required) both count as success.

When a Reboot=$true step succeeds, the script restarts the machine and exits. On the next boot the task runs it again. The completed steps skip via their flags, and the run carries on at the next step.

Post-install health isn’t “the installer said OK”. Installers say OK with great confidence and variable accuracy. Health is flag present AND service running AND install path exists, with a 120-second wait for delayed-start services. Windows Updates is flag-only, as there’s no service to check.

Phase 2: cleanup — leave nothing behind

Eject the config-drive ISO. Session 0 has no Explorer, so the usual shell ejection doesn’t work. A P/Invoke to winmm’s mciSendString("set cdaudio door open") does.
Restore UAC. EnableLUA and FilterAdministratorToken were relaxed in the base image so the build could run unattended. Turn both back on.
Uninstall cloudbase-init. Stop and delete the service, kill its processes, run the uninstaller silently and remove the directory. Guarded by 99_cleanup_complete.flag.

The startup task is deleted before validation. That way the health check can truthfully report “no automation task remains”.

Phase 3: collect the evidence

The script assembles data-validation.json, the single input to the report in part 3:

Collector	Captures
Network	per active adapter: alias, IPv4, prefix, gateway, MAC, DNS
Infrastructure	`guestinfo.vra.infrastructure` via `vmtoolsd`, 15 attempts × 10 s (the vRO subscription can land late)
OS / hardware	domain, CPUs, RAM, caption, uptime, time zone, pending reboot, latest hotfix
AD / security	local Administrators, secure-channel test, machine OU
Disks	per volume: letter, label, size, free
Post-build health	WinRM, RDP, UAC enabled, startup task gone, domain DNS resolves, NTP source
Software	per app: flag + service + path
Installed apps	both uninstall hives (64-bit and WOW6432)

The verdict is strict: GuestStatus = Success only if every software item is healthy and the machine is domain-joined. Anything else turns the report banner red. The report doesn’t do “mostly fine”.

Phases 4 and 5: publish, then self-destruct

Remap the share with the write account, with 5 × 10 s retries. It’s a different account from the read-only one used for pulls, so a compromised build guest can’t tamper with the engine.

Render the HTML report locally. Write 100_build_complete.flag: kill switch armed. Copy the report plus Data\, Flags\ and Logs\ to \\share\Builds\\\. Stop the transcript before copying the logs, so the final log is complete and unlocked.

Then:

delete the startup task;
overwrite both share passwords in the on-disk payload with *** SCRUBBED ***;
delete Data\ and Staging\ (and Flags\ and Logs\, if the share copy succeeded);
delete the sibling scripts and itself;
remove Scripts\;
reboot one final time.

The delivered server boots clean and domain-joined, with its agents running. It carries no credentials, payloads or tooling.

The reboot sequence of a nominal build

After static networking (00, exit 1003)
After the engine pull (03, exit 1003), which ends the cloudbase-init phase
After Windows Updates (05, Reboot=$true)
Final, after publish and self-destruct

Four reboots, and only one of them is Windows Updates’ doing. We asked for the other three.

The report’s timeline chart finds the update reboot by itself. Any gap over 30 seconds between steps is shaded and labelled REBOOT.

When it’s done, the Flags\ folder is the whole history of the build, in file names:

Flags  00_config-network.flag
  01_config-disks.flag
  02_join-domain.flag
  03_init-puller.flag
  04_stage-payloads.flag
  01_MonitoringAgent_installed.flag
  02_InventoryAgent_installed.flag
  03_EndpointSecurity_installed.flag
  04_LogAgent_installed.flag
  05_WindowsUpdates_installed.flag
  99_cleanup_complete.flag
  100_build_complete.flag        <- kill switch

And the persisted timings turn into this, in the report from part 3:

Every bar sits at its true wall-clock position across the reboots because the state file carried the timings. The amber band is the reboot after Windows Updates, found automatically from the gap.

Why this matters outside the lab

What customers get from a reboot-safe build is predictability. Every server takes the same steps in the same order, survives the reboots Windows insists on, and finishes clean. The delivered machine has no tooling, no credentials and no leftover tasks.

That last part matters to security reviewers as much as the first part matters to operations. And because every step’s timing is recorded, “why did this build take twice as long?” has an answer instead of a guess.

Rules learned

A reboot-safe build is kill switch + per-step flags + persisted timings. Nothing cleverer is needed.
Treat exit 3010 as success. Let the step declare whether to reboot; the loop handles it.
Health = flag and service and path. Installer exit codes lie.
Read platform-injected metadata with retries. The workflow that injects it may land after the guest starts looking.
Two share accounts: read-only for pulls, write-only for publishing.
Delete the task before validating, so “no task remains” is checkable.
Stop the transcript before you copy the logs.
Scrub, delete yourself, reboot. The customer gets a server, not a build environment.

Broadcom documentation

Windows Automation Assembler image for vSphere: how Cloudbase-Init gets onto the template, the install the cleanup removes
Event topics provided with Automation Assembler: Compute post provision, issued once per machine after it is provisioned
How do I track workflow runs: Extensibility > Activity > Workflow Runs, to see when the metadata workflow ran
Query Information using GuestInfo Variable: info-get on a guestinfo variable from inside the guest

Part 3: validation as a product, and the details that hurt.

Lab write-up of a production pattern; customer specifics removed. Opinions my own.

Windows Server 2025 via Aria Automation, part 1: the pipeline

Thu, 17 Sep 2026 06:10:00 +0100

A user fills in a form: environment, size, disks, networks, tags. Some time later there’s a Windows Server 2025 VM with:

a sequential hostname, allocated from Active Directory (AD);
static IPs on up to four NICs;
its data disks laid out and lettered;
domain membership;
the standard agent stack, installed and healthy;
a pixel-perfect HTML build report on a file share.

And there’s none of the automation tooling left on the box. The build behaves like the ideal house guest: it makes itself useful, tidies up and leaves.

This three-part series is how that works. It’s a VM Apps build: a classic Aria Automation cloud template plus vRO (Aria Automation Orchestrator). It’s also a good example of the pattern. The platform does the allocation and the metadata; the guest does the guest.

Part 1 is the architecture. Part 2 is the state machine that survives four reboots. Part 3 is the validation report and the details that hurt.

Customer-specific names, products and policies are generalised throughout. The patterns are the point.

Three layers, three reasons

Each layer owns what only it can do, and no vCenter or AD-admin credential ever enters the guest.

Each seam exists for a reason you can state in one sentence:

Aria/vRO owns what needs platform credentials: querying AD for the next hostname, and reading vCenter for where the VM landed. The guest never holds a vCenter or AD-admin credential.
cloudbase-init owns what must happen before anything else. The network has to work before the share can be reached. The disk layout has to exist before installers land on it. And the domain join changes the security context that everything after it runs in. That last one is the whole reason the next layer exists (more on that below).
The engine is pulled, not embedded. Installers change and agents get new versions, and re-releasing a cloud template for every payload update is how automation dies. Update a script on the share, and every build after that gets it.

vCenter guest customization is disabled (customizeGuestOs: false). One code path owns hostname, networking, disks and join. Two would fight.

The control plane: two vRO hooks that matter

Compute Allocation: a sequential hostname. The template builds a prefix from the location, classification, environment and application inputs. A vRO workflow queries AD computer objects with that prefix and allocates highest existing suffix + 1.

Gaps are never reused. With -001, -002, -003 and -005 present, the next is -006. A gap usually means a deleted machine, and deleted machines can be less gone than we’d like. The old name may still live in DNS, a backup catalogue or the configuration management database (CMDB). Reusing it is how you restore the wrong server.

Compute Post Provision: placement metadata into the guest. A second workflow asks vCenter where the VM actually landed: vCenter, datacenter, cluster, host, folder, datastores, and which portgroup each NIC is on. It writes all that as JSON into the VM’s extraConfig, as guestinfo.vra.infrastructure.

Later, inside the guest, vmtoolsd.exe --cmd "info-get guestinfo.vra.infrastructure" reads it back. That’s the bridge that lets the build report say “this VM is on host X, datastore Y” with zero vCenter credentials in the guest.

Three more Post Provision workflows email the backup team, the security operations centre (SOC) and the requester. (A note for part 3: the requester’s “your deployment has completed” email fires here, before the in-guest build has even started.)

First boot: cloudbase-init, four scripts, two reboots

The template’s cloudConfig is a MIME multipart. It holds one cloud-config part (set the hostname, write the tags to disk) and four #ps1_sysnative scripts, run in order. Three conventions make them safe to re-run (idempotent):

Flag files under Flags\. A script exits straight away if its flag exists, so re-runs after a reboot do nothing.
Transcripts under Logs\, one per script.
Exit 1003 means “reboot me and run this part again on next boot” (the flag then short-circuits it). Exit 0 means done. Exit 1 means failure.

Script	Does	Exit
`00-config-network`	pairs adapters (by ifIndex) with `to_json(self.networks)` (by deviceIndex); static IP/gateway/DNS per NIC	1003 — reboot 1
`01-config-disks`	extends C:, onlines/initialises/partitions/formats each extra disk from the request array	0
`02-join-domain`	runs the join script via a one-shot scheduled task as local admin; verifies `CsDomain ≠ WORKGROUP`	0
`03-init-puller`	writes `data-payload.json`, encrypts the share secrets, pulls 04/05/06 from the share, launches staging	1003 — reboot 2

That’s two reboots before a single agent has been installed. By Windows standards, it’s barely clearing its throat.

The network script carries an assumption worth writing on the wall. It pairs OS adapters with the request’s NICs by position. That holds for freshly cloned VMs where the NICs were added in PCI order. If anyone ever changes the NIC order after cloning, revisit it.

Why not just cloudbase-init (or guest customization) all the way down?

The obvious design is the one we started with: let the platform do the guest. vCenter guest customization sets the hostname and IP and joins the domain. Then one cloudbase-init userdata installs the agents and patches, and reports back. No scheduled tasks, no pulled engine, no state machine.

It works on a workgroup machine. It stops working the moment the machine joins a real domain, and it stops in a way that is easy to misread.

The domain join changes the rules mid-build. Everything up to the join runs as a fresh, local, unmanaged Windows install. That means the local Administrator, the default execution policy, and no central policy.

At the join, the machine lands in its target organisational unit (OU). On the next policy refresh, which the reboot guarantees, Group Policy applies. In this environment, the policy for member servers includes the usual security baseline:

controls on script execution;
limits on what may run, from where, and under which accounts;
hardening of the local administrator context.

None of that is negotiable, and none of it should be. It’s the same policy every production server gets.

What that does to a first-boot pipeline. cloudbase-init runs its plugins as a service, as LocalSystem, running scripts from its own directory. Before the join, that context can do anything. After the join, it’s exactly the kind of context the baseline is designed to restrict.

So everything that comes after the join is in trouble: the installer pulls, the agent installs, the reboots, the validation. They either fail outright or, worse, quietly do nothing. cloudbase-init logs the plugin as executed, the script never ran anything, and the build “completes” with an unpatched server carrying no agents.

The first few builds looked exactly like that. The log was technically correct, which is the least useful kind of correct.

Two ways out, one of them wrong. You can relax policy for the build: a staging OU with a weaker baseline, a Group Policy Object (GPO) exemption for the cloudbase-init path, or a delayed join. But then the server is built under one set of rules and delivered under another. And the join becomes a late step that nothing after it exercises.

Or you can accept the policy as the environment it is, and run the post-join work in a context the policy permits.

The permitted context is a scheduled task under an explicit identity. Each task runs as a named account: a domain-joined local admin for the pull and the join check, and SYSTEM for the build master. It runs at the highest run level, with -ExecutionPolicy Bypass on each call, from a staging path the policy allows. That’s an ordinary, auditable pattern, and the baseline was written to allow for it.

So cloudbase-init doesn’t run the OS commands itself; it hands them to tasks. Its job shrinks to “get the network up, lay out disks, join, hand off”. The design also picks up three things it now depends on:

a per-task execution ceiling (1 h for the pull, 2 h for the build) that contains a hung installer instead of leaving a half-built VM;
an at-startup trigger, which is what makes a multi-reboot state machine possible after cloudbase-init has been uninstalled;
a clean security story: the identities that do the work are the ones the domain already governs, and they stop existing on the box when the build is done.

Why not Aria’s own in-guest mechanisms? Guest customization only covers hostname, IP and join. Letting it and cloudbase-init own the same settings means two code paths fighting (hence customizeGuestOs: false).

Driving the guest from outside means vRO calling into the VM for two hours. That needs guest credentials held centrally, and it keeps a management path open for the whole build. ABX (Aria’s action-based extensibility) can’t reach inside the guest at all.

The platform’s job is what needs platform credentials: the hostname from AD and the placement facts from vCenter. The guest does the guest, under the domain’s rules, from the first reboot after the join.

Even if the GPO were relaxed tomorrow, don’t simplify the wrappers away. The ceiling and the startup trigger are worth having on their own.

The hand-off: `03-init-puller`

This is where the code baked into the template stops, and the centrally managed engine starts:

Create Flags\ Logs\ Data\ Scripts\ under the log folder.
Write data-payload.json, the one document everything after this reads. It holds the share path and accounts, the image, the project, deployment and requester, the installer folder and file pairs, the tags and the placement facts.
Encrypt the two share passwords with AES, using a key derived from the machine’s BIOS UUID. Copied off the box, the payload is useless.
Generate a runner, and run it through a one-shot task as local admin. It maps the share read-only, copies 04-stage-payloads, 05-build-master and 06-validate-build locally, unmaps, and launches staging.
Write the flag and exit 1003. Reboot 2. cloudbase-init’s job is done, though nobody has mentioned the uninstall to it yet.

04-stage-payloads copies every installer to local disk. Installers never run across SMB, so they’re immune to network blips and file locks mid-install. It also registers the Build-Master startup task. From here on, every boot runs 05-build-master.ps1 until the build is complete.

What the requester actually sees is a short form. Leaving out the site-specific enum values, the inputs are:

inputs:
  location:        # site code -> hostname prefix
  classification:  # security zone -> hostname prefix, OU
  environment:     # prod / pre-prod / test -> hostname prefix, tags
  application:     # application code -> hostname prefix, folder
  image:           # Windows2025 (the template is image-versioned)
  flavor:          # Small / Medium / Large -> vCPU + RAM
  count:           # number of identical machines
  bootDiskSizeGB:  # C: (extended in-guest by 01-config-disks)
  primaryNetwork:  # required
  network2..4:     # optional; each becomes a static NIC
  additionalDisks: # [{number, name, letter, sizeGB}] -> D:, L:, ...
  tags:            # free-form key/value -> written to disk, shown in the report

Each of those shapes the hostname, lands in guestinfo, or is written to disk for the build engine to read. Nothing is entered twice.

Why this matters outside the lab

For an organisation, this pipeline turns a Windows server from something a person builds into something the platform delivers. A request goes into a catalog, and a domain-joined server with its agents comes out. The security team’s controls (naming, join, hardening, monitoring agents) apply every time, because they’re in the pipeline, not in a checklist.

The separation of concerns is what keeps it maintainable. The platform holds the credentials and the guest does the work. A change to the software stack is a script update on a share, not a template re-release.

Rules learned

Split by who holds the credential. The platform queries AD and vCenter; the guest never does. guestinfo is the one-way bridge.
Disable vCenter customization when cloudbase-init owns the guest. One owner.
Pull the engine from a share. Embed only what must run before the network exists.
Flag files, transcripts and exit 1003 make first-boot scripts idempotent and reboot-safe.
Anything after the domain join runs in a scheduled task under an explicit identity: for policy, for the execution ceiling, and for the startup trigger.
Never reuse hostname gaps.

Broadcom documentation

Cloudbase-Init commands for Windows in Automation Assembler: a cloudConfig section of Cloudbase-Init commands in the cloud template
Windows Automation Assembler image for vSphere: the Windows template with Cloudbase-Init installed to run as LocalSystem
vSphere static IP addresses in Automation Assembler: why customizeGuestOs must be false when the cloudConfig sets the network
Event topics provided with Automation Assembler: Compute allocation, where resource names can still change, and Compute post provision
How do I modify virtual machine properties using a Automation Orchestrator Client workflow subscription: an Orchestrator workflow subscribed to Compute allocation to set the VM name
Query Information using GuestInfo Variable: reading guestinfo variables from inside the guest with VMware Tools

Part 2: the state machine that survives four reboots.

Lab write-up of a production pattern; customer specifics removed. Opinions my own.

Seven demo apps, one request: deploying a showcase stack via VCF Automation

Wed, 16 Sep 2026 08:40:00 +0100

Every platform needs a demo stack: something that looks alive on a projector and quietly exercises every layer underneath. Ours is seven apps on a vSphere Kubernetes Service (VKS) cluster that VCF Automation provisioned, in a tenant VPC, each behind its own load-balancer VIP.

kubedoom       192.168.144.20:5900   (VNC — yes, it kills pods)
kubeinvaders   192.168.144.21
kube-ops-view  192.168.144.22
pacman         192.168.144.23        (+ MongoDB on a PVC — persistent high scores)
podinfo        192.168.144.24
goldpinger     192.168.144.25        (DaemonSet incl. control plane)
prometheus     (in-cluster: KSM + node-exporter, 11/11 targets up)

Captured from the VIPs the platform handed out. KubeDoom is VNC-only and podinfo is an API, so they sit this one out; the Ops topology tile shows all seven by name.

The path that matters: tenanted, not shortcut

I deployed the first version of this stack against the supervisor: an admin kubeconfig, a vSphere namespace, kubectl apply. It worked, and it was wrong, for the reason the previous post spells out. Nothing about it was provided to anyone.

So I tore it down: seven apps, cluster, VPC and VIPs, gone in about seven minutes. Demolition, as ever, was the quick part. Then I rebuilt it the proper way:

dev-01 org → default-project → SupervisorNamespace (class large, region f06, VPC default-f06)
          → VKS cluster vks-demo01 → seven apps

Everything went in through VCF Automation: the Cloud Consumption Interface (CCI) API for the namespace and cluster, then the apps through the cluster’s own kubeconfig. Two auth facts are worth writing down; each cost me a cycle:

A provider service account reaches the cloud API only. CCI (/cci/kubernetes/apis/...) rejects provider tokens with a 401. It needs an org-scoped service account.
Device-flow login uses the service account’s own UUID as the client_id (not its software ID), against the tenant endpoint /oauth/tenant//device_authorization.

What the platform does for free

Each app is an ordinary Deployment plus a Service of type LoadBalancer. The supervisor turns each service into an NSX VPC load-balancer virtual server, and hands back a VIP from the org’s external block. No ingress controller, no MetalLB, no port-forwarding. Seven services, seven VIPs, done.

Pac-Man’s MongoDB asks for a persistent volume and gets one on vSAN, through the CSI driver the supervisor already installed. The high scores are on enterprise storage. Priorities.

Goldpinger runs as a DaemonSet on every node, the control plane included, and draws the node-to-node mesh live.

That’s three platform services (load balancing, storage and networking) exercised by apps that know nothing about VCF.

The traps

The supervisor refuses cluster-scoped RBAC, even to admin. Demo apps that need ClusterRoles (kube-ops-view, Goldpinger, KubeDoom) must live on a guest cluster. You can’t run them as vSphere Pods on the supervisor.

PodSecurity restricted is the VKS 1.35 default. Half the demo set runs as root. The symptom: the Deployment shows 0 UP-TO-DATE, the ReplicaSet exists, zero pods, and the events say FailedCreate. The fix: label the namespace pod-security.kubernetes.io/enforce=privileged. Then say so in the demo, because it’s a teaching moment.

node-exporter without hostNetwork. The VPC fabric blocks scrapes from a pod to a node IP, so the stock DaemonSet’s hostNetwork: true doesn’t help. Run it as a normal pod and let Prometheus scrape it in-cluster.

Docker Hub is flaky from behind a proxy. TLS handshake timeouts put pods into kubelet’s image-pull backoff, where they sit and sulk. Deleting the stuck pods gets round the backoff. A Harbor proxy-cache project fixes it properly.

Pod CIDR shadowing. The stock 192.168.0.0/16 pod range hid the org’s 192.168.144.0/21 external block from inside the cluster, so apps couldn’t reach their neighbours’ VIPs. The pod CIDR is now 172.16.0.0/16.

Automation notes

The whole stack is one checkbox on the lab’s catalog item that deploys a supervisor. installDemoApps creates the cluster, with the newest compatible Kubernetes release and the newest built-in ClusterClass, both auto-detected. Then it applies the seven apps and reports their URLs in the deployment summary.

On a fresh environment, an integrated run takes 16.7 minutes to CREATE_SUCCESSFUL. An immediate re-run takes 3.3 minutes, every step idempotent. That’s the number I actually care about: it makes a broken demo a re-run, not a rebuild. Demos have an uncanny sense of when they’re being watched.

Why this matters outside the lab

A demo stack sounds like a toy. To be fair, one of the apps is Pac-Man. But it’s actually the fastest way to make a platform legible to people who don’t read YAML. A customer watches a request become a cluster, watches seven services get their own addresses, then opens one and plays it.

Everything underneath gets exercised (self-service Kubernetes, load balancing, persistent storage, isolation), and a non-technical stakeholder can see it working. The same stack is what we put in front of a new team on day one. And the same idempotent deploy is what makes it safe to demonstrate live: if it breaks on stage, it re-runs in three minutes.

Rules learned

Demo apps that need cluster-scoped RBAC must run on a guest cluster. The supervisor won’t grant it, even to admin.
Build the stack through the tenanted path: org service account, then CCI, then namespace, then cluster, then apps. Same apps, but now they’re provided, quota’d and visible in Ops.
VKS 1.35 has restricted PodSecurity by default. Label the namespace and say why.
One LoadBalancer per app means one NSX VIP per app. No ingress needed for a demo.
Pick a pod CIDR that doesn’t overlap the VPC external block.
Make the deploy idempotent. A demo that re-runs in 3 minutes is one you can afford to break on stage.

Broadcom documentation

Creating and Managing Namespaces: a SupervisorNamespace with a class, region and VPC, through the All Apps API
Create a Service Account in Your VCF Automation Organization: organization service accounts and their device-bound API tokens
Pod Deployment with Load Balancer Service: a Service of type LoadBalancer on a VKS cluster and its external IP
Storage for VKS Clusters: persistent volumes from the storage classes assigned to the namespace
Configure PSA for VKr 1.25 and Later: restricted enforced by default from VKr 1.26, and the namespace label that relaxes it

Previously: one VKS cluster, two ways. The Pac-Man instance here is the one from What’s a VPC?.

Lab environment; opinions my own.

A datacenter in a catalog tile: nested ESXi pods via VCF Automation All Apps

Wed, 16 Sep 2026 07:40:00 +0100

Everything in this series so far was built with kubectl and API calls. That proves the platform. It doesn’t make a product.

This post turns the pod into a catalog item. Fill in a name, pick a VPC, click Request, and a few minutes later there’s a datacenter in miniature with two SSH prompts waiting. Barely time to put the kettle on.

All Apps in one paragraph

VCF Automation 9.1 has two provisioning models side by side. VM Apps is the classic Aria Automation path: cloud templates run through an IaaS (infrastructure as a service) engine that drives vCenter. All Apps is the supervisor-native path. Its blueprint composes Kubernetes objects (a Supervisor Namespace, VM Service VMs, NSX subnets, vSphere Kubernetes Service clusters), and the vSphere Supervisor’s controllers reconcile them.

A blueprint is formatVersion: 2, and its resources are CCI.Supervisor.Namespace and CCI.Supervisor.Resource. The second is literally “here’s a manifest, apply it in that namespace.” So the blueprint is a composition of the manifests from the earlier posts, with two additions: inputs, and dependsOn.

The blueprint, section by section

Inputs — the form

inputs:
  podName:  {type: string, default: nested-pod, pattern: '^[a-z0-9]([-a-z0-9]*[a-z0-9])?$'}
  vpcName:  {type: string, description: Must exist and be Realized before deploying.}
  esxOva:   {type: string, default: vmi-61bb062ddfc506b79}   # Nested ESXi 9.1 appliance
  isoImage: {type: string, default: vmi-39f562e2ae9e9c501}   # the ISO to attach
  vmClass:  {type: string, default: best-effort-large, enum: [best-effort-large, best-effort-xlarge, best-effort-2xlarge]}

The namespace — with libraries attached

namespace:
  type: CCI.Supervisor.Namespace
  properties:
    generateName: ${input.podName}-        # NOT name — new namespaces get a suffix
    className: large
    regionName: f06
    vpcName: ${input.vpcName}              # pins the namespace to the pod's VPC
    storageClasses: [{name: vSAN Default Storage Policy, limit: 400000Mi}]
    zones: [{name: domain-c9, cpuLimit: 40000M, memoryLimit: 64000Mi, ...}]
    contentSources:
      - {name: ISO, type: ContentLibrary}
      - {name: f06-vks-lib01, type: ContentLibrary}

contentSources is the line that closes the gap a lot of first attempts hit. Our libraries were plain vCenter libraries, and a namespace created by VCF Automation got none of them. No libraries meant no VirtualMachineImages, so nothing could be deployed. An empty namespace is very tidy, and of no use to anyone.

Declaring the libraries here attaches them at creation. (The 9.1 docs say a namespace class is assigned a content library automatically, and provider libraries are shared with every namespace.)

The topology — ordered on purpose

snTrunk:   {type: CCI.Supervisor.Resource, properties: {context: ${resource.namespace.id}, manifest: }}
snMgmt:    {dependsOn: [snTrunk], ...  manifest: }
snVmotion: {dependsOn: [snMgmt],  ...  manifest: }
bmMgmt:    {... manifest:  sn-trunk, vlan 1610>}
bmVmotion: {... manifest:  sn-trunk, vlan 1611>}

The dependsOn chain is the whole reason every pod has identical CIDRs. A fresh VPC realizes its subnets in the order they were created, and the blueprint fixes that order.

The hosts — dual-NIC, ISO attached, bootstrapped by OVF

esx01:
  type: CCI.Supervisor.Resource
  dependsOn: [bmMgmt]                    # no point booting before VLAN 1610 exists
  properties:
    manifest:
      kind: VirtualMachine                # vmoperator.vmware.com/v1alpha5
      spec:
        hardware:
          cdrom: [ ... the ISO, declared, connected ... ]
        network:
          interfaces: [ eth0 -> sn-trunk, eth1 -> sn-trunk ]
        bootstrap:
          vAppConfig: [ guestinfo.hostname / ipaddress / vlan / ... ]
  wait:
    fields: [{path: status.powerState, value: PoweredOn}]

(Abridged: the full resource carries the image references, VM class, guest ID and the complete guestinfo set.)

Two vNICs, both on the trunk: the nested equivalent of a VCF host’s two pNICs. The ISO rides along as a declarative CD-ROM.

The wait block makes the deployment’s completion mean something: the request doesn’t finish until the host is powered on. Finishing any earlier would be optimism, not automation.

The doors — one VIP per host

Each host gets a VirtualMachineService of type LoadBalancer, which selects it by label and publishes 22 and 443. A blueprint output then reads the VIP back out of the service’s status:

outputs:
  esx01Ssh: {value: "ssh root@${resource.esx01Access.object.status.loadBalancer.ingress[0].ip}"}

The outputs show up in the deployment view, so the requester gets the SSH command rather than a scavenger hunt.

The request flow, end to end: request → deployment in progress → complete.

What the blueprint cannot express (yet)

Three cluster-scoped objects must exist before the request, in this order:

VPC: privateIPs: 172.30.0.0/16, the same in every pod.
VPCAttachment: the connectivity profile with the service gateway. Without it, creating the load balancer fails loudly.
LoadBalancer: silently, permanently required before the namespace.

VCF Automation 9.1 does have blueprint types for the first two: CCI.VPC, and CCI.VPC.Configuration with kind: VPCAttachment. One of its sample blueprints uses both. There is none for the third.

A later test confirmed it. CCI.VPC.Configuration rejects a LoadBalancer kind, and a VPC created that way comes up with load balancing off. Its namespace’s VIPs would sit pending, waiting for a load balancer that isn’t coming.

Today that’s a short script or a runbook step per pod. The honest framing: the blueprint is the pod, the VPC is the tenancy, and tenancy is still created one layer up. I’d expect the load balancer to become blueprintable too. Until then, keep the three calls next to the blueprint in version control.

Publishing: one version at a time

The order is blueprint, then BlueprintVersion, then release. Validation happens at version time, not at create time. The result lives in status.validationMessages rather than the HTTP code, so a 200 with ContentValid: False is a thing: the HTTP equivalent of “yes, but no”.

And only one version can be published. Unrelease 1.0.0 before releasing 1.1.0, or you get a 409. (The full list of sharp edges is its own post.)

Why this matters outside the lab

This is where platform engineering turns into a service. The gap between “we can build you an environment” and “request one from the catalog” is the gap between days and minutes. It’s also the gap between a bespoke build and one that is consistent, quota-controlled and recorded every time. For an organisation that means:

Time-to-environment measured in minutes, requested by the people who need it, without a queue.
Consistency by construction. Every environment comes from the same definition, so support, training material and runbooks all match.
Governance built in. Quotas, ownership, history and clean teardown are properties of the deployment record, not a spreadsheet.

The nested-ESXi pod is one catalog item. The same approach delivers any shape of environment: application stacks for developers, sandboxes for a proof of concept, demo kits for a sales team, isolated builds for a partner.

Rules learned

All Apps blueprints are compositions of manifests: CCI.Supervisor.Namespace plus CCI.Supervisor.Resource per object. If it works with kubectl, it works in a blueprint.
Use generateName, not name, for the namespace, and contentSources to attach libraries at creation. Keep zones/storageClasses flat, not wrapped.
dependsOn is how you get deterministic CIDRs: order the subnets.
wait.fields turns “request complete” into “host is powered on”.
VPC, VPCAttachment and LoadBalancer are prerequisites outside the blueprint, in that order, before every request.
One published version per blueprint. Validation results live in status, not in the HTTP response.

Broadcom documentation

Sample Blueprints in VCF Automation: CCI.Supervisor.Namespace and CCI.Supervisor.Resource examples, with generateName and context.
Creating bindings and dependencies between resources in blueprints in VCF Automation: dependsOn and property bindings, which set the build order.
Specifying formatVersion in Blueprints in VCF Automation: what formatVersion: 2 adds, outputs included.
Versioning Blueprints in VCF Automation: blueprint versions, and releasing one to the catalog.
Creating and Managing Content Libraries for Stand-Alone VMs in vSphere Supervisor: VM content libraries and the namespaces they are associated with.
Deploy VMs with Configurable OVF Properties in vSphere Supervisor: OVF properties set through the VM Service’s vAppConfig transport.

Previously: shared services for isolated tenants. This closes the Pod Papers’ core arc. The companion posts on dual-NIC, no-DHCP bootstrap and blueprint gotchas fill in the details.

Lab environment; opinions my own. Blueprint nested-esxi-pod 1.1.0 is live in the lab catalog; YAML above trimmed for length.

Blueprinting the supervisor: seven CCI blueprint gotchas

Wed, 16 Sep 2026 07:10:00 +0100

The nested-esxi-pod blueprint works. Getting there took seven distinct “ContentValid: False” failures, or worse, a 200 that quietly did nothing. None of them are in the docs I could find. All of them are five-minute fixes once you know. Here they are, in the order they bit.

1. `${input.x}` is illegal inside a flow mapping

This looks like valid YAML and valid blueprint syntax:

- {key: guestinfo.hostname, value: {value: "esx01.${input.podName}.res.lab"}}

It fails content validation. The expression parser doesn’t reach into flow-style ({...}) mappings. Block style is fine:

- key: guestinfo.hostname
  value:
    value: esx01.${input.podName}.res.lab

Mixed style in the same list is fine too: only the entries that carry an expression need to be block style. (That’s why the blueprint’s vAppConfig list looks inconsistent. It’s deliberate.)

2. `name` vs `generateName` for a new namespace

A CCI.Supervisor.Namespace you’re creating must use generateName. The Cloud Consumption Interface (CCI) API rejects metadata.name. The platform appends a random suffix, so pod-a- becomes pod-a-dgf5p, which really rolls off the tongue. Everything downstream should reference ${resource.namespace.id}, never a literal name.

3. Zones and storage classes are flat

My early attempts wrapped them the way the raw CCI API does:

initialClassConfigOverrides:
  zones: [...]

In a blueprint, they’re top-level properties of the namespace resource:

zones:
  - name: domain-c9
    cpuLimit: 40000M
    memoryLimit: 64000Mi
storageClasses:
  - name: vSAN Default Storage Policy
    limit: 400000Mi

And zones are required. Omit them and the API says “Zone should be specified”, which is at least a clear message.

4. A new namespace has no content library

Deploy the namespace, deploy a VM, and you get: no VirtualMachineImage found. Our libraries were plain vCenter libraries, and a namespace created by VCF Automation attached none of them. (The 9.1 docs say a namespace class is assigned a content library automatically, and that provider libraries are shared with every namespace.) The fix is one block:

contentSources:
  - {name: ISO, type: ContentLibrary}
  - {name: f06-vks-lib01, type: ContentLibrary}

Without it, you’re in the vSphere Client attaching libraries to a namespace by hand, which rather defeats the catalog.

5. Images sync after attach — wait for `status.disks`

Even with libraries attached at creation, the first VM create in a fresh namespace can be rejected:

no disks found in image ... status.disks

The image objects appear at once, but their disk metadata syncs over the next 1–3 minutes. The quota webhook checks status.disks and refuses until it’s populated.

In a blueprint, make the hosts dependsOn something that takes a couple of minutes (the binding maps did the job here), or add an explicit wait. In a script, poll:

kubectl get virtualmachineimage -n   -o jsonpath='{.status.disks}'

6. Validation lives in `status`, not the HTTP code

Creating a BlueprintVersion returns 200 whether or not the content is valid. A 200 here means “I heard you”, not “I agree”. Read the object back:

status:
  contentValid: false
  validationMessages:
    - "... unexpected token ..."

If your pipeline checks the response code, it will happily publish a broken blueprint. Check status.contentValid and print the messages.

7. Only one published version — 409 on the second

Release 1.1.0 while 1.0.0 is released and you get a 409. It isn’t a transient conflict; it’s the rule. Unrelease the current version, then release the new one.

In practice, a publish step is unrelease old → release new, and there’s a short window where the catalog item has no released version. Do it when nobody’s requesting.

Bonus: the things that aren’t blueprint problems

Three prerequisites have to exist before the request: VPC, VPCAttachment and LoadBalancer, in that order. VCF Automation 9.1 has blueprint types for the first two: CCI.VPC, and CCI.VPC.Configuration with kind: VPCAttachment (see its sample blueprints). But it has none for the load balancer, and a VPC created that way comes up with load balancing off.

So we create all three before the request. The blueprint’s vpcName input says “must exist and be Realized”, and it means it. Nothing in the blueprint fails if they’re missing; the deployment just never gets a VIP.

Why this matters outside the lab

VCF Automation’s All Apps model is new, and new platforms have edges. Most of these seven are not documented, and all of them stall a first project by days if you meet them cold.

The value of a delivery partner who has already built on the platform isn’t the YAML. It’s that the first blueprint a customer publishes goes live on day one instead of week two. And it’s that the sharp edges are encoded into templates and provisioning scripts, where users never meet them.

Rules learned

Expressions need block-style YAML; flow mappings don’t get parsed.
generateName, and reference the namespace by ${resource.x.id}.
zones and storageClasses are flat, and zones are required.
contentSources on the namespace for vCenter libraries, or nothing can be deployed.
Wait for image status.disks before the first VM (1–3 min).
Check status.contentValid. The HTTP code lies by omission.
One released version per blueprint: unrelease, then release.

Broadcom documentation

Sample Blueprints in VCF Automation: a new namespace with generateName, and resources placed in it with context.
Create a Namespace Class in VCF Automation: VM and storage classes, per-zone limits, and the content libraries a namespace gets.
Creating and Managing Content Libraries for Stand-Alone VMs in vSphere Supervisor: associating VM content libraries with a namespace.
Creating bindings and dependencies between resources in blueprints in VCF Automation: dependsOn, the explicit build order.
Versioning Blueprints in VCF Automation: versions, and releasing one to the catalog.

Companion to a datacenter in a catalog tile.

Lab environment; opinions my own. Error text captured live.

VM Apps vs All Apps: a field comparison of VCF Automation's two provisioning models

Wed, 16 Sep 2026 07:00:00 +0100

VCF Automation 9.1 has two ways to build things, side by side, as two types of organisation. If you’ve come from Aria Automation, you’ll recognise one of them immediately. The other looks like Kubernetes because it is Kubernetes.

Choosing between them isn’t a matter of taste. They have genuinely different shapes, and some use cases are natural in one and awkward in the other.

I’ve now pushed the same list of requirements through both on a live VCF 9.1 lab. This is the comparison I wish I’d had at the start. Hindsight, as usual, turned up late.

The two shapes

The same request, brokered on the left and declared on the right. The text versions below carry the detail.

VM Apps, the classic Aria Automation model:

Org ─ Project ─ Cloud Zone(s)
                   │
        Cloud Account (vCenter / NSX)
                   │
   flavor · image · network · storage profiles
                   │
   Cloud Template (formatVersion 1) ──▶ IaaS engine ──▶ vCenter API
        Cloud.vSphere.Machine, Cloud.NSX.Network, customization spec

Provisioning is imperative through a broker. The IaaS engine holds the vCenter session. The tenant references abstractions (flavors, image mappings) that resolve at request time. State lives in the IaaS database.

All Apps, supervisor-native, through the Cloud Consumption Interface:

Org ─ CCI Project ─ Region
          │
   VPC (tenant-created; overlapping privateIPs allowed)
          │   + VPCAttachment   + LoadBalancer (before the namespace!)
   Supervisor Namespace (spec.vpcName pins it)
          │
   Kubernetes objects reconciled by the Supervisor:
     VirtualMachine / VirtualMachineService / Subnet / BindingMap / VKS Cluster
          │
   Blueprint (formatVersion 2, CCI.Supervisor.*) → BlueprintVersion → Catalog

Provisioning is declarative. The blueprint states the desired objects, and the supervisor’s controllers make reality match them and keep it that way. State lives in etcd. Networking is NSX VPC-native.

Scorecard by use case

Every row below was actually built, not read about. It’s a slow way to fill in a table, but an honest one.

Use case	VM Apps	All Apps
Linux VM + software	`Cloud.vSphere.Machine` + cloudConfig	VM Service VM + cloud-init (typed field)
Windows VM	customization spec + software components	`bootstrap.sysprep`; ISO attachable declaratively
ISO / CD-ROM attach	day-2 edit in vCenter	`hardware.cdrom` in the manifest
Multi-NIC	NICs across network profiles	multiple `interfaces` on VPC subnets; failover verified
Self-service catalog	template released to catalog	blueprint → version → publish (one live version)
Load-balanced access	NSX LB via network profile	`VirtualMachineService` → VPC LB VIP
Kubernetes clusters	separate TKG integration	native `Cluster` object in the namespace
Identical isolated environments	hard: on-demand NAT nets, unique addressing usually forced	natural: VPC per pod, overlapping CIDRs, zero route between
Nested ESXi / VLAN networks	trunk portgroups on the physical vDS — a fabric change	trunk subnet + binding maps — no fabric change
Shared infra (WSUS, repos)	shared segment routed everywhere	one `PrivateTGW` subnet, one-way reachability
Extensibility	vRO, ABX, event broker (rich)	controllers, GitOps, kubectl (thinner day-2 today)
Deep per-device vSphere tuning	anything vCenter can do	what the VM Service API models

The two bold rows are the ones that decided it for me. Both are documented in this series. Both are genuinely hard in VM Apps, and simply the default in All Apps.

Where VM Apps still wins

Be fair to the incumbent:

Years of content. vRO workflows, ABX actions, template libraries and a mature day-2 action framework carry over unchanged. If you have an estate of them, that’s real value you’d be throwing away.
Deep vSphere reach. Anything vCenter can do to a VM (RDMs, per-device tuning, exotic customization), the IaaS engine can do, because it drives vCenter directly.
Familiar network model. Segments, portgroups, on-demand routed/NAT networks from a network profile. No new mental model.
Multi-cloud lineage. The same template idiom stretched to other endpoints.

The Windows Server 2025 pipeline elsewhere on this blog is a VM Apps build, and it’s a good one. It has Event Broker hooks for hostname allocation and placement metadata, cloudbase-init staging, and a reboot-safe state machine in the guest. Nothing about it needs rewriting, which, for a pipeline, is high praise.

Where All Apps wins, and why it’s structural

Declarative and self-healing. Desired state lives in etcd, and controllers reconcile to it. There’s no second database to drift from vCenter.
Structural multi-tenancy. A VPC per tenant is a hard NSX boundary, not an administrative one. Overlapping CIDRs are allowed, so cookie-cutter environments deploy side by side.
VMs and Kubernetes are one model. The same blueprint composes a VKS cluster, VMs, secrets and networking. It’s GitOps-able with ordinary tools.
VPC networking is first-class. Trunk subnets, binding maps, PrivateTGW, a per-VPC gateway firewall: none of it has a VM Apps equivalent.
Modern bootstrap. cloud-init and sysprep as typed API fields.

Where All Apps hurts today (all observed live)

Ordering matters, and the errors are opaque. The LB must exist before the namespace. New namespaces reject VMs until images sync. Blueprint validation has seven sharp edges, and I found every one of them by walking into it.
The tenancy layer is only partly blueprintable. VPC and VPCAttachment have blueprint types (CCI.VPC, CCI.VPC.Configuration). But there’s no LoadBalancer kind, and a blueprint-made VPC comes up with load balancing off. So all three go through the VPC API: scripted, not catalogued.
Two endpoints. The CCI proxy serves tenancy objects, and workload manifests go to the supervisor. You’ll hold two kubeconfigs.
Day-2 is thinner. There’s no event broker. Extensibility means controllers and GitOps, which is fine if that’s your team and a gap if it isn’t.

Recommendation

Default to All Apps for new build-outs. Every use case on the list is natural in the VPC model, including the two that are genuinely hard in VM Apps. And the whole estate is version-controlled YAML.

Keep VM Apps as the compatibility surface for existing vRA content, and for the rare thing that needs direct vCenter device manipulation. They coexist as organisations of different types on one platform. So migration is incremental, and nobody has to rewrite a working pipeline on a deadline.

And whichever you pick, codify the ordering rules into the scripts that provision tenancy. The sharp edges stay wrapped up in there, and the people requesting catalog items never meet them.

Why this matters outside the lab

Most customers arriving at VCF 9 have an estate of Aria Automation content, and a question: rewrite, coexist, or migrate? The answer above is the one we take into design workshops.

In practice it starts with an assessment:

which existing templates and workflows still earn their keep;
which use cases are genuinely better served by the VPC model;
where the boundaries sit.

Then comes a coexistence plan that moves workloads across opportunistically, not on a deadline. What makes that low-risk is the two architectures sharing one platform, as organisations of different types.

Rules learned

Two shapes, not two skins: imperative-through-a-broker vs declarative-reconciled. Pick per use case, not once for the estate.
All Apps is structurally better at isolation with identical addressing and nested/VLAN networks without fabric changes.
VM Apps is still the home for existing vRO/ABX content and deep vCenter device work.
All Apps’ pain is ordering, in this order: VPC, attachment, LB, namespace, subnets, image sync, then workloads. Script it once.
They coexist, so migrate opportunistically.

Broadcom documentation

Organization Management: the two organization types in VCF Automation 9.1, All Apps and VM Apps
Bimodal Consumption Design for VM Apps and All Apps Workloads: VM Apps and All Apps organizations on shared infrastructure, and a phased move between them
Sample Blueprints in VCF Automation: All Apps blueprints built from CCI.Supervisor.Namespace and CCI.Supervisor.Resource
Deploying and Managing Virtual Machines in vSphere Supervisor: the VM Service, its declarative API, VM classes and images
Managing Networking in VCF Automation Organizations: VPCs in an organization, whose private CIDRs need not be unique
vSphere resource examples in VCF Automation for VM Apps: VM Apps cloud templates built on Cloud.vSphere.Machine

Lab environment; opinions my own. Grounded in a VCF 9.1 / vSphere Supervisor with NSX VPC networking build-out; every row was deployed.

One VKS cluster, two ways: kubectl vs VCF Automation All Apps

Wed, 16 Sep 2026 06:50:00 +0100

The Cluster manifest is the same. That’s the point of this post, and also the punchline, so I’ve rather given away the ending. VKS is VKS whichever door you walk through.

What differs is everything wrapped around the cluster: who can ask for it, what limits it, who can see it, and how it shows up in operations tooling.

So: two clusters, one supervisor, one ClusterClass, two paths.

Path A: kubectl, the way we’ve always done it

kubectl vsphere login --server  --tanzu-kubernetes-cluster-namespace demo
kubectl apply -f cluster.yaml

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
spec:
  clusterNetwork: { pods, services, serviceDomain }   # set all three — see below
  topology:
    class: builtin-generic-v3.6.0
    classNamespace: vmware-system-vks-public          # the gotcha
    version: v1.35.5+vmware.1
    controlPlane: {replicas: 1}
    workers: { one node pool, 2 replicas }
    variables: [ vmClass, storageClass ]

Fifteen minutes later: a cluster.

It does need a vSphere namespace first, and somebody (an admin) makes that by hand in the vSphere Client. They attach a content library, assign a VM class and set the quota. In this lab, that somebody is me.

Path B: the same manifest, as a catalog request

In All Apps, the cluster is a CCI.Supervisor.Resource inside a blueprint, sitting next to a CCI.Supervisor.Namespace:

resources:
  namespace:
    type: CCI.Supervisor.Namespace
    properties:
      generateName: ${input.name}-
      className: large                      # quota comes from the class
      regionName: f06
      vpcName: ${input.vpc}
      contentSources: [{name: f06-vks-lib01, type: ContentLibrary}]
  cluster:
    type: CCI.Supervisor.Resource
    properties:
      context: ${resource.namespace.id}
      manifest:

Publish it, and a tenant user requests it from a tile:

Fifteen minutes later: a cluster. Byte-identical Cluster object.

What path B adds for free

	kubectl	All Apps request
Who can create	anyone with a kubeconfig to that namespace	org users with catalog entitlement (org RBAC)
Namespace	pre-created by an admin, by hand	created by the request, from a class
Quota	set per namespace in the vSphere Client	inherited from the namespace class (`small`/`medium`/`large`)
Content library	admin attaches manually	`contentSources` on the blueprint
Networking	whatever the namespace has	pinned to a tenant VPC by `vpcName`
Sizing choices	edit YAML	form inputs with enums (worker count, VM class)
Record	`kubectl get cluster`	a deployment with inputs, owner, history, day-2 actions
Visibility	supervisor only	VCFA inventory and VCF Operations

That last row is the one operations teams care about:

The VCFA-deployed cluster appears in the Ops object model: supervisor, then namespace, then cluster, then nodes, then the workloads running on it. Its demo apps show up by name in the topology tab.

The kubectl-built cluster is also visible to Ops (it’s the same supervisor, after all). But it has no deployment, no owner, no request history and no quota lineage. It’s a thing that exists, not a thing that was provided. As far as the paperwork goes, it simply turned up one day.

The four gotchas, in order of how much time they cost

Both paths share the same four traps on VKS 1.35 / VCF 9.1:

Our VCFA-created namespace had no content library. Zero VirtualMachineImages means no cluster. Set contentSources in the blueprint, or attach one by hand. The 9.1 docs say a namespace class gets a content library automatically; our large class had none assigned.
ClusterClass lives in vmware-system-vks-public. It 404s from the workload namespace unless the spec sets topology.classNamespace.
The default Quick Start namespace is 1000M CPU / 1000Mi. Unusable for a cluster. Use a real class: small 10000M/10000Mi, medium 20000M, large 40000M.
VKS 1.35 enforces PodSecurity restricted by default. Demo apps that run as root get a ReplicaSet and no pods, and the events say FailedCreate. Label the app namespace pod-security.kubernetes.io/enforce=privileged (or fix the apps).

Plus one that waits until later to bite, as the best ones do. Set clusterNetwork.serviceDomain explicitly. It’s immutable after create, and without it some add-ons build the service DNS name wrongly (....svc. with no domain).

And pick a pod CIDR that doesn’t shadow your VPC’s external range. The stock 192.168.0.0/16 hid the org’s 192.168.144.0/21 from inside the cluster.

So which one?

Use kubectl when you’re the platform team proving something on a supervisor, or debugging.

Use All Apps the moment a second person needs a cluster. The request form is the interface, and the namespace class is the guardrail. The deployment is the audit trail, and Ops sees a provided service rather than a stray object.

The cluster’s the same either way. The service isn’t.

Why this matters outside the lab

The business case for the second path is governance without friction. Development teams get Kubernetes clusters on request. The platform team gets quotas, ownership, RBAC and a monitoring view of every cluster, for free.

That’s the difference between a managed Kubernetes service and a collection of clusters nobody can account for. It’s typically the gap that stops organisations offering Kubernetes broadly at all. Everything the developers touch stays standard Kubernetes; the control lands around it, not on it.

Rules learned

The Cluster object is identical across paths. All Apps wraps it; it doesn’t change it.
What All Apps adds: catalog RBAC, class-based quota, library attach, VPC pinning, deployment history, and a place in the Ops object model.
Four traps on 1.35: no library on our namespace class, classNamespace, tiny default quota, PodSecurity restricted.
Set serviceDomain and a non-shadowing pod CIDR at create time. Both are immutable.

Broadcom documentation

Workflow for Provisioning VKS Clusters Using kubectl: path A, from Supervisor login to an applied cluster YAML
Using the Versioned ClusterClass: the built-in ClusterClass in vmware-system-vks-public and spec.topology.classNamespace
Sample Blueprints in VCF Automation: a Cluster as a CCI.Supervisor.Resource beside a CCI.Supervisor.Namespace
Create a Namespace Class in VCF Automation: CPU and memory limits, VM classes, storage and content libraries per class, and the default small, medium and large
Configure PSA for VKr 1.25 and Later: restricted enforced by default from VKr 1.26, and the namespace label that relaxes it
Monitoring VKS Clusters Using VCF Operations: VKS clusters in VCF Operations, monitored by default on VCF 9.1

Next in All Apps in Practice: the demo apps that live on this cluster.

Lab environment; opinions my own.

One catalog item, one VCF instance: building a lab factory

Wed, 16 Sep 2026 06:40:00 +0100

Every nested VCF lab starts the same way: a heroic PowerShell script. Ours was esxihostdeploy.ps1: ovftool plus PowerCLI, with an interactive menu asking which environment, which ESX version, which role and how many hosts.

It worked. It also lived on one person’s machine, prompted for credentials, and knew nothing about anything that comes after the hosts exist. Heroic, then, but not much of a team player.

This post is about what it became: a set of VCF Automation catalog items where requesting one form produces a complete nested VCF 9.1 instance. That means ESXi hosts, bringup (vCenter, NSX, SDDC Manager), a vSphere Supervisor, VCF Automation, Operations and identity. The environment number is practically the only real input.

The shape of the factory

Three stages, each a catalog item, plus a wrapper that chains them:

Stage 1  Nested ESX Hosts        VM Apps template + vRO actions
         (the old script, reborn declaratively)
Stage 2  Deploy VCF 9.1 Instance vRO workflow driving the VCF Installer API
         (spec generated, validated, bringup started)
Day-N    Supervisor · NSX Edge · VCF Automation · Ops Logs/Networks/RTM ·
         Identity (AD)           one catalog item each
Wrapper  "Deploy VCF Stack"      one form, checkbox per component

The wrapper’s form has a checkbox per component. Ticking one reveals that component’s tab, with every field already filled in. One lab password feeds every credential. Tick everything, click request, and go and make a coffee. Possibly several.

Rule 1: derive everything from one number

Each lab environment is f0X, and everything scales from X by formula:

Element	Pattern	f03 example
Names	`f0X-m01-*`	`f03-m01-vc01.res.lab`
Subnets	`10.(20+X)._.0/24`	`10.23.1.0/24` (mgmt)
VLANs	`2X0n`	2307 (edge TEP)

The bringup spec is hundreds of lines of JSON, which the VCF Installer wants and nobody wants to type. A vRO action generates it from a known-good reference spec plus X. Nobody edits a deployment spec by hand, which means nobody typo-breaks a bringup at 2am.

When the old script did this, the formulas lived in string concatenation. Now they live in one action, with the reference spec beside it.

The same philosophy carried into stage 1:

the script’s “next free esxNN index” scan became a vRO action bound to the request form;
its VLAN/IP arithmetic became template expressions;
its --prop:guestinfo.* flags became ovfProperties in the template.

Porting a script isn’t rewriting it. It’s finding the declarative home for each behaviour.

Rule 2: never wait for anything you can watch instead

One constraint shaped the whole design. A VCF Automation request gets about two hours before the platform gives up on it. Our runs died at exactly two hours with Delegating token is not service token, which is a roundabout way of saying “time’s up”.

The project’s request timeout also defaults to two hours, and its Provisioning tab can raise it. We never tried. A full VCF bringup takes longer than that, and it shouldn’t hold a request open for hours anyway. So the wrapper never waits:

Bringup is fire-and-forget. The workflow authenticates to the installer, validates the spec, starts the task, and hands back a watchTaskId. Re-attach any time to check on it.
Fleet deployments (VCF Automation, Ops for Logs/Networks, metrics) are server-side tasks. The items submit with waitForCompletion=false.
The supervisor item submits enablement and returns; vCenter carries on.
Only fast, deterministic steps (identity configuration, minutes) run to completion inside the request.

A whole VCF instance as one deployment record. The request finished in under an hour, while the build ran on for twelve.

The result: a full-stack kick-off completes as a request in well under an hour (53 minutes on the run pictured). The actual multi-hour build carries on as server-side tasks you can watch. The request’s job isn’t to do the work. It’s to start the work correctly and tell you where to watch it.

The corollary: a failed component is recorded, and the remaining components still run. You fix one thing and re-run one item, not the world.

Rule 3: plan mode for infrastructure

Every item in the chain supports validateOnly, and the wrapper cascades it. Tick everything, set validateOnly, and the entire stack is planned against the live environment with zero changes. Specs are generated, prerequisites checked, and name and IP collisions caught. A smoke runner does exactly this on every change to the automation itself.

If you build nothing else into your lab automation, build this. The number of 2am bringups saved by a five-minute dry run is not small.

The traps that shaped the rules

Some of the design above exists because of scars:

Hardware validation hates virtual NVMe. Bringup’s hardware compatibility list (HCL) check will block nested hosts. The spec generator has to account for it, or you discover it two hours in. Twice, if you’re us.
Small disks, surprising layouts. Nested hosts with 64 GB disks ship ESX-OSDATA at essentially the whole disk. A post-provision step relocates scratch, or stage 2 fills the disk with logs.
DNS is a prerequisite, not a step. The installer’s pre-flight wants every record resolvable before it starts. A one-shot script creates the per-environment records ahead of the request.
vRO’s content-source lag. A new or changed workflow takes 15–20 minutes of data collection before the catalog sees it. Publish, wait, then test, or you’ll debug a ghost.
Wrapper inputs are duplicated by necessity. vRO requires every sub-workflow input to be passed explicitly, so adding an input to a component means updating the wrapper’s call too. Null-guards in each component turn a forgotten field into a loud failure instead of a silent default.

Why bother?

Because the payoff compounds. Once a full VCF instance is a catalog request, everything downstream changes character:

upgrade rehearsals happen on freshly built instances instead of precious pets;
a broken environment is redeployed, not repaired;
the lab stops being a collection of snowflakes and becomes a product: versioned, validated, reproducible.

Where this is heading: the factory’s output feeding per-student pods with identical addressing.

The factory’s next customers, funnily enough, are the isolated VPC pods from the other series on this blog. Same philosophy, one layer further down.

Why this matters outside the lab

A complete VCF instance from one form changes what an environment costs to have. Environments used to be precious, because building one took a week. Now they’re disposable, and a lot follows from that:

Proofs of concept run on an environment built for the customer’s scenario, not on whatever happens to be free.
Upgrade and migration rehearsals happen on a fresh instance of the right version, then it’s deleted.
Training and enablement get a real VCF per person or per team.
Reference builds exist for every supported release, on demand.

This is how Comms-care provides a dedicated instance to every consultant. The same factory, pointed at a customer’s requirements, is a repeatable way to deliver environments, rather than a one-off project each time.

Rules learned

Derive names, subnets and VLANs from a single environment number; generate specs, never hand-edit them.
Respect the request-duration ceiling: start long work, return a task handle, re-attach to watch. Never block a wrapper on an hours-long task.
validateOnly on every item, cascaded by the wrapper: dry-run the whole stack before touching anything.
Componentise failure: one broken step re-runs alone.
Pre-create DNS; expect HCL friction on virtual hardware; budget for vRO’s content-source lag.

Broadcom documentation

Use a JSON Specification File to Deploy VMware Cloud Foundation or vSphere Foundation: deploying VCF 9.1 from a JSON spec, which the installer validates first
First VCF Instance FQDNs and IP addresses: the FQDNs, static IPs and forward and reverse DNS every component needs
Add VCF Operations Orchestrator Client workflows to the VCF Automation catalog: vRO workflows as catalog items, through an Orchestrator content source
vSAN ESA Deployment: Override HCL Validation for Non-Certified Hardware: the installer’s vSAN ESA disk check against the HCL, and the documented override
Bill of Materials 9.1.0: the components and builds of VCF 9.1.0
Add a project for your VCF Automation for VM Apps development team: a project’s request Timeout on its Provisioning tab, two hours by default

Lab environment; opinions my own. The automation described builds nested VCF 9.1 instances for lab and rehearsal use — patterns transfer, specifics are ours.

validateOnly everywhere: plan mode for infrastructure

Wed, 16 Sep 2026 06:30:00 +0100

Terraform has plan. Kubernetes has --dry-run=server. Ansible has --check. Every mature infrastructure tool grew a way to say “tell me what you’d do, then don’t”.

The alternative is finding out at 2am, two hours into a bringup, that a hostname doesn’t resolve. Two hours is a long time to wait for bad news from DNS.

vRO workflows don’t come with a dry run. This is the case for adding one to every workflow you write, and cascading it through every wrapper.

The shape

Every catalog item in the lab factory has a boolean input, validateOnly, which defaults to false. When it’s true, the workflow does everything it can without changing anything:

authenticate to every endpoint it would touch
resolve every name it would use, and fail on the ones that don’t
generate every spec it would submit, and run the target’s own validation API on it where one exists (the VCF Installer has one; use it)
check for collisions: names, IPs, existing objects
report what it would have created, then return CREATE_SUCCESSFUL

The wrapper is the one form that chains hosts, bringup, supervisor, fleet components and identity. It has the same checkbox, and it cascades it to every component. Tick everything, tick validateOnly, request.

Thirty seconds to a few minutes later, you have a full-stack plan against the live environment, and nothing has moved. It’s the most productive way I know of doing nothing.

What it caught

This isn’t hypothetical. On an already-built environment, the cascaded dry run of the whole stack reported:

edge cluster: already exists; correctly recorded, and the wrapper carried on
Ops for Logs: IP_IN_USE on the planned address. Quite right: it’s deployed
Ops for Networks: same
supervisor: the existing one would be reused; the per-service plan listed which services were already active
identity: bind succeeded, group resolved, no changes needed

That’s a plan output, and a reassuringly dull one.

On a fresh environment, the same run has caught, at various times:

a DNS record missing for one of about 40 required names. The installer’s own pre-flight found it in seconds, instead of bringup finding it in hour two.
a stale content-library image ID.
a form field arriving null because a custom form hadn’t finished re-importing. That’s a publishing bug, and the dry run surfaced it before anyone requested anything real.

The argument against, answered

“It doubles the code.” It doesn’t. It puts the if (!validateOnly) guard around the mutating call, and the validation logic is code you should have had anyway.

What it does force is separating “compute what to do” from “do it”. That’s how the workflows should have been structured in the first place.

“Some things can’t be validated without doing them.” True. Say so in the result summary (“would deploy X; no pre-validation available”) rather than skipping the item. Partial plans are still plans.

“We have a test environment.” You have a test environment. A dry run against the target is what catches the collision with the thing that’s already there, which the test environment has never had the pleasure of meeting.

Make it the smoke test

The best consequence: a validateOnly request against a known environment is a regression test for the automation itself, and you can run it on every change.

The factory’s smoke runner does exactly that. It requests every item with validateOnly: true, asserts CREATE_SUCCESSFUL, and diffs the plan summary against the last run. It takes minutes. It has caught more bugs in the workflows than any amount of code review, which says something about the bugs, or possibly about my code reviews.

The same form, real or dry-run. One checkbox decides.

Why this matters outside the lab

For anyone who has sat through a failed change window, the value is obvious: a full dry run against the real estate before anything moves. Fewer failed changes, shorter windows, and a plan output that answers the change board’s questions before they’re asked.

It also gives auditors something they rarely get from infrastructure automation: evidence of what was going to happen, produced by the same tooling that then did it.

Rules learned

Add validateOnly to every workflow. Default false. Wrappers cascade it.
Dry-run does everything but mutate: auth, resolve, generate, call the target’s validator, check collisions, report.
Where a step truly can’t be pre-validated, say so in the summary. Never skip it silently.
A dry run against the real target is a plan. A dry run on every change is a smoke test. Same checkbox.
The refactor it forces (compute, then act) is the one you wanted.

Broadcom documentation

Use a JSON Specification File to Deploy VMware Cloud Foundation or vSphere Foundation: the installer’s own validation of a spec, with errors and warnings, before deployment
First VCF Instance FQDNs and IP addresses: the names and forward and reverse DNS records every VCF component needs
Add VCF Operations Orchestrator Client workflows to the VCF Automation catalog: vRO workflows as catalog items, through an Orchestrator content source

Part of The Lab Factory.

Lab environment; opinions my own.

Porting a PowerShell deploy script to a catalog item: the mapping table is the post

Wed, 16 Sep 2026 06:20:00 +0100

Every lab has one: the script that builds the nested hosts. Ours was esxihostdeploy.ps1, ovftool plus PowerCLI. An interactive menu asked for environment, ESX version, role, size and host count. Then came a loop of ovftool --prop:guestinfo.*, Set-VM, New-NetworkAdapter and Set-HardDisk.

It worked for years. It also prompted for credentials, lived on one machine, and knew nothing about the bringup that came after. Loyal, if a little needy.

Porting it to a VCF Automation catalog item (VM Apps: a cloud template plus vRO) is not a rewrite. It’s a sorting exercise. Every behaviour in the script has a natural home in the declarative model, and the skill is finding it. Here’s the whole table, then the four rows that took some thought.

The mapping

Script behaviour	Where it lives now
Environment menu (f01–f10)	`environment` input (enum)
Version menu → OVA path on a share	`esxVersion` input → image mapping → content library item
Role menu (management / workload / both)	`role` input; “both” = two requests
Size menu / auto-detect from an existing host	`size` input (auto-detect dropped — see below)
vCenter + ESXi credential prompts	vCenter creds gone (cloud account); `esxiRootPassword` an encrypted input
Next-free `esxNN` index scan (gap-filling)	vRO action `getNextEsxHostIndexes`, bound to the request form
VLAN / IP / gateway arithmetic	template expressions (same formulas)
`ovftool --prop:guestinfo.*`	`ovfProperties` on `Cloud.vSphere.Machine`
Folder lookup / `New-Folder`	allocation-phase subscription creates the folder if missing
`Set-VM` cpu / mem	`cpuCount` / `totalMemoryMB` in the template
2× `New-NetworkAdapter` (Vmxnet3)	`networks` array, `deviceIndex` 0/1/2
3× `Set-HardDisk` grow	post-provision subscription
`NestedHVEnabled = $true`	post-provision subscription
“Power on after?” prompt	`powerOn` input, honoured post-provision
Summary table printed at the end	the deployment view in the UI

That’s five homes, in decreasing order of preference:

input
template expression
platform abstraction (image mapping, network profile, cloud account)
form action (read-only lookup at request time)
subscription (imperative work at a lifecycle stage)

Push each behaviour as far up that list as it will go.

The four decisions that weren’t obvious

1. Drop auto-detect; the platform already remembers

The script inspected an existing host to work out its size. That was a workaround for having no record. The catalog is the record: deployment history shows what size every existing host was requested at. So the input simply asks. Fewer moving parts, and the requester sees the choice.

2. The index scan is a form action, not a workflow step

“Next free esx07” has to be known at request time, so the requester sees the names they’ll get. That makes it a vRO action bound to the custom form, not a step inside provisioning: getNextEsxHostIndexes(environment, role, count) returns an array of strings. Forms can call actions, so use one for anything that’s a lookup.

3. Rename and folder go in Compute Allocation, hardware goes in Post Provision

Two blocking subscriptions, filtered by a custom property on the template:

	Subscription 1	Subscription 2
Topic	Compute allocation	Compute post provision
Runnable	“Set VM Name & Folder”	“Finalize Hardware”
Does	sets `resourceNames`, creates folder	grows 3 disks, `NestedHVEnabled`, power on
Timeout	10 min	30 min

The rename must happen at allocation. It’s the only stage where a workflow output named resourceNames is applied to the machine. Growing the disks and enabling nested hardware virtualisation both need a VM that exists, so they wait for post-provision. Both subscriptions are blocking: the deployment doesn’t proceed until they return.

4. `nestedEsx: 'yes'` — not `true`

The subscription condition is event.data.customProperties.nestedEsx == "yes". Why not "true"? Because boolean-looking strings can arrive in the event payload as typed booleans, and true == "true" is false in the condition evaluator and in the vRO code.

It fails silently: the subscription just never fires. yes can’t be coerced. Small thing; two hours. The ratio still stings.

What stayed exactly the same

The formulas. 10.(20+X)._.0/24, VLAN 2X0n, gateway .254: they were string concatenation in PowerShell, and they’re template expressions now, character for character. The guestinfo.* property names are identical too, because the OVA didn’t change.

Porting a script well means most of it survives. Only the plumbing moves.

The bits that still bite

Network profile without IP ranges. Addressing comes in through guestinfo, not the platform’s IP address management (IPAM). Tag the trunk port group and add no ranges, or IPAM and guestinfo will disagree, each utterly sure of itself.
Two template revisions in the repo. v1 is what the guide documents; v2 grew later. Both are kept deliberately, and both are labelled. Check which one the org has imported before editing either.
Content-source lag. A new or changed vRO action needs about 15–20 minutes of data collection before the form sees it. Publish, wait, then test.
Small disks and OSDATA. Nested hosts with 64 GB disks ship ESX-OSDATA at essentially the whole disk. Templates now provision 128 GB, and a relocate-scratch script limits the damage on existing hosts.

Every menu prompt from the script is now a field; Host Indexes is the form action’s answer to “next free esxNN”.

Why this matters outside the lab

Almost every organisation has these scripts: valuable, trusted, and stuck on one person’s machine. The message of this post for them is that modernising doesn’t mean rewriting.

The logic survives. What changes is where it lives: behind a request form, with access control, an audit trail, consistent inputs and a deployment record. That’s how a team turns tribal knowledge into a service, without losing the years of edge cases the script already handles.

Rules learned

Porting is sorting: input, then expression, then platform abstraction, then form action, then subscription. Push each behaviour as far up as it goes.
Lookups the requester needs to see are form actions.
Rename at allocation (resourceNames output); hardware at post-provision. Both blocking.
Filter subscriptions on a custom property, and make its value a word that can’t be coerced to a boolean.
Drop workarounds for missing state; the catalog is the state.
Keep the formulas. Move the plumbing.

Broadcom documentation

vSphere resource examples in VCF Automation for VM Apps: Cloud.vSphere.Machine with CPU and memory, several NICs, a vCenter folder and ovfProperties
How to add image mapping in VCF Automation for VM Apps to access common operating systems: image mappings, named images per cloud account and region
Using VCF Operations orchestrator actions in the custom form designer in VCF Automation for VM Apps: form fields filled by a vRO action that takes other fields as inputs
Event topics provided with VCF Automation for VM Apps in: the Compute allocation topic, where resource names can still change, and Compute post provision
Create an extensibility subscription: conditions on event.data, blocking, and the workflow a subscription runs

Part of The Lab Factory. Next: driving the VCF Installer API from vRO.

Lab environment; opinions my own.

Driving the VCF Installer API from vRO: generate, validate, start, walk away

Wed, 16 Sep 2026 06:10:00 +0100

The nested hosts exist. Now they need to become a VCF instance: vCenter, NSX, SDDC Manager and the fleet components. The VCF 9.1 Installer appliance does that through an API, from a deployment spec of a few hundred lines of JSON.

This post is the vRO workflow that drives it. Three design constraints shaped it:

nobody edits the spec by hand;
the request gets two hours by default;
nested hosts fail hardware validation.

Unlike stage 1, this isn’t VM provisioning, so it’s not a cloud template. It’s a vRO workflow published directly as a catalog item through an Orchestrator content source.

The spec is generated, never edited

A vRO action, buildVcfDeploymentSpec(environment, hostFqdns, labPassword, …), returns the whole spec as a string. Its structure was reconciled against a validated export from a real bringup (the installer UI lets you export the spec it accepted). Everything variable derives from the environment number X:

Element	Pattern	f03
Names	`f0X-m01-*`	`f03-m01-vc01.res.lab`
Subnets	`10.(20+X)._.0/24`	`10.23.1.0/24` (mgmt)
VLANs	`2X0`	2301 mgmt … 2306 TEP
Gateways	`.254`	`10.23.1.254`
vMotion / vSAN ranges	`.1–.16`	`10.23.3.1-16`
NSX TEP pool	`.6.1–.6.32`	`10.23.6.1-32`
SDDC Manager	`f0X-vcf01.res.lab`	`f03-vcf01.res.lab`

var n   = parseInt(environment.substring(1), 10);   // "f03" -> 3
var pfx = environment + "-m01";
var net = "10." + (20 + n);
function vlan(o) { return 2000 + (n * 100) + o; }
function gw(sub) { return net + "." + sub + ".254"; }

Some things are static across environments: DNS, NTP, subdomain, component sizes, vSAN ESA FTT=1, and the component build versions pinned to the installer binaries.

One lab password feeds every credential field. The UI export scrubs them, and the action puts them back as the API schema expects.

Two spec-level decisions worth stealing:

skipEsxThumbprintValidation: true instead of carrying per-host sslThumbprint. Supported, and the right trade-off for a lab.
Ops and Automation are checkboxes that add their blocks to the spec. The installer only accepts a licenseServerSpec when Ops is present, so the action adds them together or not at all.

The workflow: authenticate → validate → start → return

1. POST /v1/tokens                 installer login
2. POST /v1/sddcs/validations      the installer's OWN pre-flight on the spec
   (poll until COMPLETED; fail on any FAILED check)
3. if validateOnly -> return the validation report; touch nothing
4. POST /v1/sddcs                  start bringup -> sddcTaskId
5. return { sddcTaskId, installerUrl }

Step 2 is the validateOnly story made concrete. In seconds, the installer will tell you that f03-m01-nsx01.res.lab doesn’t resolve, that an IP is in use, or that a host isn’t reachable. Two hours into a bringup is a bad time to learn that. Seconds in, it’s merely embarrassing.

Every one of the roughly 40 DNS records the pre-flight wants is created ahead of time by a one-shot PowerShell script (New-LabEnvDnsRecords.ps1). DNS is a prerequisite, not a step.

The two-hour leash, and how to slip it

A request from the catalog carries a token with a roughly two-hour lifetime, and a bringup takes around eight. The arithmetic is not encouraging. When the token runs out, the workflow dies with Delegating token is not service token.

The project’s request timeout also defaults to two hours, and its Provisioning tab can raise it. We never tried, because eight hours is too long to hold a request open.

So by default the workflow is fire-and-forget: waitForCompletion=false, return the task id, and watch progress in the installer UI. A watchTaskId input lets you re-attach later, and poll an already-running bringup from a new request.

The wrapper that chains everything (the hosts, then bringup, then the day-N components that need bringup to be finished) has a neater trick.

The catalog-bound parent deploys the hosts, submits bringup, and then re-executes itself as a plain vRO run (Orchestrator > Run: no catalog token, no two-hour kill). It carries the hidden bringupWatchTaskId with it. That continuation polls the installer task to completion (eight hours, fine), and then runs certificates, fleet items, edge, supervisor and identity.

Watch it under Orchestrator > Activity > Runs. The catalog request itself completes in under an hour (47 minutes on the run pictured below), having started the work correctly and handed off.

Two rows, one build. The parent returns inside the catalog’s window; the continuation waits out the bringup and does the day-N work.

Nested-host frictions

Three things a physical bringup never meets:

HCL validation vs virtual NVMe. The installer’s hardware compatibility list (HCL) check blocks the virtual NVMe controller. Fix it at the vSphere Lifecycle Manager (vLCM) layer: enforce_hcl_validation=false on the image policy. The vSAN health test nvmeonhcl also complains. We silence it through the vSAN API, best-effort, with a manual fallback.
DVS compatibility appears late. After bringup, NSX takes 1–2 hours to settle before the supervisor’s zones endpoint stops returning 500. If the supervisor stage fails with “No compatible DVS” on a fresh instance, wait and re-run just that item. Patience, it turns out, is a deployment step.
TSM-SSH. Bringup wants SSH on the hosts, so the wrapper enables it host-direct through SOAP before submitting.

Stale schema: the failure that looks like a bug and isn’t

Add an input to the vRO workflow after the catalog item exists, and three things happen. The form shows the new field. The request records its value. And the workflow receives null, because Service Broker keeps the old request schema until the content source re-imports. Two out of three, which here counts as a fail.

The workflow null-guards every boolean, and aborts with “inputs not mapped” rather than running with silently wrong options. The fix: re-import the content source, confirm the schema, and submit a new request. Resubmitting an old one reuses the old payload.

There are actually three asynchronous layers between “publish” and “mappable request”:

vRO processing the import;
the catalog schema after re-import;
the form service still enforcing the previous custom form for a minute or two.

Same symptom for all three. Check timing before assuming a bug.

Why this matters outside the lab

Repeatable, generated VCF deployments matter well beyond a lab: a second site, a disaster-recovery instance, a new business unit, an environment per supported release.

Generating the specification from a validated reference removes a whole class of errors: the ones that come from editing hundreds of lines of JSON by hand. Running the installer’s own validation first turns “find out in hour two” into “find out in minute one”. It’s the difference between a VCF deployment being a project and being a procedure.

Rules learned

Generate the spec from a validated export plus one number. Nobody hand-edits JSON at 2am.
Run the installer’s own validation first, and make it a mode you can request on its own.
Pre-create DNS. Enable SSH. Disable HCL enforcement on virtual NVMe.
Respect the request lifetime: start, return a task id, re-attach. For a long chain, have the workflow re-run itself outside the catalog.
Null-guard every input and fail loudly. Stale schemas are a fact of life after adding inputs.
On a fresh instance, give NSX an hour before you expect DVS compatibility.

Broadcom documentation

Use a JSON Specification File to Deploy VMware Cloud Foundation or vSphere Foundation: the deployment spec, its validation and retrying failed tasks; points to the VCF Installer API reference
First VCF Instance FQDNs and IP addresses: the FQDNs and forward and reverse DNS records to plan for every component
vSAN ESA Deployment: Override HCL Validation for Non-Certified Hardware: the installer’s vSAN ESA HCL check, and the documented override for non-certified disks
Add VCF Operations Orchestrator Client workflows to the VCF Automation catalog: a vRO workflow as a catalog item, through an Orchestrator content source
Bill of Materials 9.1.0: the component versions and builds of VCF 9.1.0
Add a project for your VCF Automation for VM Apps development team: a project’s request Timeout on its Provisioning tab, two hours by default

Part of The Lab Factory. Previously: porting the host script.

Lab environment; opinions my own. Bringup verified end-to-end on a rebuilt environment: 305/305 tasks, COMPLETED_WITH_SUCCESS.

VCF Automation on The Nested Lab

Every resource in the VCF Automation 9.1 blueprint designer: the complete guide

The shape of a blueprint

How the palette maps to YAML

Where the designer and the platform disagree

What every resource shares

Properties on every type

wait: when a resource is finished

Reading a resource’s live state

Checking a manifest before you deploy it

Supervisor Namespace

Windows Server 2025 via Aria Automation, part 3: validation as a product, and the details that hurt

The report

The details that hurt

vmtoolsd mangles arguments under SYSTEM in Session 0

Ejecting an ISO with no Explorer

Exit 1003 is the only reboot you should ever request from cloudbase-init

cloudbase-init has to remove itself

UAC was off. Turn it back on.

Secrets: encrypt to the machine, then scrub

The requester is told too early

Hostname allocation has a race

What I’d carry to any build pipeline

Why this matters outside the lab

Rules learned

Broadcom documentation

Windows Server 2025 via Aria Automation, part 2: a state machine that survives four reboots

The three mechanisms

Phase 1: software, declaratively

Phase 2: cleanup — leave nothing behind

Phase 3: collect the evidence

Phases 4 and 5: publish, then self-destruct

The reboot sequence of a nominal build

Why this matters outside the lab

Rules learned

Broadcom documentation

Windows Server 2025 via Aria Automation, part 1: the pipeline

Three layers, three reasons

The control plane: two vRO hooks that matter

First boot: cloudbase-init, four scripts, two reboots

Why not just cloudbase-init (or guest customization) all the way down?

The hand-off: 03-init-puller

Why this matters outside the lab

Rules learned

Broadcom documentation

Seven demo apps, one request: deploying a showcase stack via VCF Automation

The path that matters: tenanted, not shortcut

What the platform does for free

The traps

Automation notes

Why this matters outside the lab

Rules learned

Broadcom documentation

A datacenter in a catalog tile: nested ESXi pods via VCF Automation All Apps

All Apps in one paragraph

The blueprint, section by section

Inputs — the form

The namespace — with libraries attached

The topology — ordered on purpose

The hosts — dual-NIC, ISO attached, bootstrapped by OVF

The doors — one VIP per host

What the blueprint cannot express (yet)

Publishing: one version at a time

Why this matters outside the lab

Rules learned

Broadcom documentation

Blueprinting the supervisor: seven CCI blueprint gotchas

1. ${input.x} is illegal inside a flow mapping

2. name vs generateName for a new namespace

3. Zones and storage classes are flat

4. A new namespace has no content library

5. Images sync after attach — wait for status.disks

6. Validation lives in status, not the HTTP code

7. Only one published version — 409 on the second

Bonus: the things that aren’t blueprint problems

Why this matters outside the lab

Rules learned

Broadcom documentation

VM Apps vs All Apps: a field comparison of VCF Automation's two provisioning models

The two shapes

`wait`: when a resource is finished

The hand-off: `03-init-puller`

1. `${input.x}` is illegal inside a flow mapping

2. `name` vs `generateName` for a new namespace

5. Images sync after attach — wait for `status.disks`

6. Validation lives in `status`, not the HTTP code

4. `nestedEsx: 'yes'` — not `true`