This post is about what it became: a set of VCF Automation catalog items where requesting one form produces a complete nested VCF 9.1 instance — ESXi hosts, bringup (vCenter, NSX, SDDC Manager), a vSphere Supervisor, VCF Automation, Operations, identity — with the environment number as practically the only real input.

The shape of the factory

Three stages, each a catalog item, plus a wrapper that chains them:

Stage 1  Nested ESX Hosts        VM Apps template + vRO actions
         (the old script, reborn declaratively)
Stage 2  Deploy VCF 9.1 Instance vRO workflow driving the VCF Installer API
         (spec generated, validated, bringup started)
Day-N    Supervisor · NSX Edge · VCF Automation · Ops Logs/Networks/RTM ·
         Identity (AD)           one catalog item each
Wrapper  "Deploy VCF Stack"      one form, checkbox per component

The wrapper’s form has a checkbox per component; ticking one reveals that component’s tab with every field pre-populated. One lab password feeds every credential. Tick everything, click request, and go get coffee.

Rule 1: derive everything from one number

Each lab environment is f0X, and everything scales from X by formula:

The bringup spec — hundreds of lines of JSON the VCF Installer wants — is generated by a vRO action from a known-good reference spec plus X. Nobody edits a deployment spec by hand, which means nobody typo-breaks a bringup at 2am. When the old script did this, the formulas lived in string concatenation; now they live in one action with the reference spec beside it.

The same philosophy carried into stage 1: the script’s “next free esxNN index” scan became a vRO action bound to the request form, its VLAN/IP arithmetic became template expressions, its --prop:guestinfo.* flags became ovfProperties in the template. Porting a script isn’t rewriting it — it’s finding the declarative home for each behaviour.

Rule 2: never wait for anything you can watch instead

The hard constraint that shaped the whole design: a VCF Automation request gets about two hours before the platform gives up on it. A full VCF bringup takes longer than that. So the wrapper never waits:

• Bringup is fire-and-forget — the workflow authenticates to the installer, validates the spec, starts the task, and hands back a watchTaskId. Re-attach any time to check on it.
• Fleet deployments (VCF Automation, Ops for Logs/Networks, metrics) are server-side tasks; the items submit with waitForCompletion=false.
• The supervisor item submits enablement and returns; vCenter carries on.
• Only fast, deterministic steps (identity configuration, minutes) run to completion inside the request.

A whole VCF instance as one deployment record — the request finished in under an hour while the build ran on for twelve.

Result: a full-stack kick-off completes as a request in well under an hour (53 minutes on the run pictured), while the actual multi-hour build continues as watchable server-side tasks. The request’s job isn’t to do the work — it’s to start the work correctly and tell you where to watch it.

The corollary: a failed component is recorded and the remaining components still run. You fix one thing and re-run one item, not the world.

Rule 3: plan mode for infrastructure

Every item in the chain supports validateOnly — and the wrapper cascades it. Tick everything, set validateOnly, and the entire stack is planned against the live environment with zero changes: specs generated, prerequisites checked, name/IP collisions caught. A smoke runner exercises exactly this on every change to the automation itself.

If you build nothing else into your lab automation, build this. The number of 2am bringups saved by a five-minute dry run is not small.

The traps that shaped the rules

Some of the design above exists because of scars:

• Hardware validation hates virtual NVMe. Bringup’s HCL check will block nested hosts; the spec generator has to account for it, or you discover it two hours in — twice, if you’re us.
• Small disks, surprising layouts. Nested hosts with 64 GB disks ship ESX-OSDATA at essentially the whole disk; a post-provision step relocates scratch or stage-2 fills the disk with logs.
• DNS is a prerequisite, not a step. The installer’s pre-flight wants every record resolvable before it starts; a one-shot script creates the per-environment records ahead of the request.
• vRO’s content-source lag. A new or changed workflow takes 15–20 minutes of data-collection before the catalog sees it. Publish, wait, then test — or you’ll debug a ghost.
• Wrapper inputs are duplicated by necessity. vRO requires every sub-workflow input to be passed explicitly, so adding an input to a component means updating the wrapper’s call too. Null-guards in each component turn a forgotten field into a loud failure instead of a silent default.

Why bother?

Because the payoff compounds. Once a full VCF instance is a catalog request, everything downstream changes character: upgrade rehearsals happen on freshly-built instances instead of precious pets; a broken environment is redeployed, not repaired; and the lab stops being a snowflake collection and becomes a product — versioned, validated, reproducible.

Where this is heading: the factory’s output feeding per-student pods with identical addressing.

The factory’s next customers, funnily enough, are the isolated VPC pods from the other series on this blog — same philosophy, one layer further down.

Why this matters outside the lab

A complete VCF instance from one form changes what an environment costs to have. Environments that used to be precious — because building one took a week — become disposable, and a lot follows from that:

• Proofs of concept run on an environment built for the customer’s scenario, not on whatever happens to be free.
• Upgrade and migration rehearsals happen on a fresh instance of the right version, then it’s deleted.
• Training and enablement get a real VCF per person or per team.
• Reference builds exist for every supported release, on demand.

This is how Comms-care provides a dedicated instance to every consultant. The same factory, pointed at a customer’s requirements, is a repeatable way to deliver environments rather than a one-off project each time.

Rules learned

• Derive names, subnets and VLANs from a single environment number; generate specs, never hand-edit them.
• Respect the request-duration ceiling: start long work, return a task handle, re-attach to watch. Never block a wrapper on an hours-long task.
• validateOnly on every item, cascaded by the wrapper — dry-run the whole stack before touching anything.
• Componentise failure: one broken step re-runs alone.
• Pre-create DNS; expect HCL friction on virtual hardware; budget for vRO’s content-source lag.

Lab environment; opinions my own. The automation described builds nested VCF 9.1 instances for lab and rehearsal use — patterns transfer, specifics are ours.

vRO workflows don’t come with one. This is the case for adding it to every single one you write, and cascading it through every wrapper.

The shape

Every catalog item in the lab factory has a boolean input, validateOnly, default false. When true the workflow does everything it can without changing anything:

• authenticate to every endpoint it would touch
• resolve every name it would use, and fail on the ones that don’t
• generate every spec it would submit, and run the target’s own validation API on it where one exists (the VCF Installer has one; use it)
• check for collisions — names, IPs, existing objects
• report what it would have created, then return CREATE_SUCCESSFUL

The wrapper — the one form that chains hosts, bringup, supervisor, fleet components, identity — has the same checkbox, and cascades it to every component. Tick everything, tick validateOnly, request. Thirty seconds to a few minutes later you have a full-stack plan against the live environment, and nothing has moved.

What it caught

Not hypothetically. On a built environment, the cascaded dry run of the whole stack reported:

• edge cluster: already exists — correctly recorded, wrapper carried on
• Ops for Logs: IP_IN_USE on the planned address — right, it’s deployed
• Ops for Networks: same
• supervisor: the existing one would be reused; the per-service plan listed which services were already active
• identity: bind succeeded, group resolved, no changes needed

That’s a plan output. On a fresh environment the same run has caught, at various times: a DNS record missing for one of ~40 required names (the installer’s own pre-flight found it, in seconds, instead of bringup finding it in hour two); a stale content-library image ID; a form field arriving null because a custom form hadn’t finished re-importing — which is a publishing bug the dry run surfaced before anyone requested anything real.

The argument against, answered

“It doubles the code.” It doesn’t — it moves the if (!validateOnly) guard around the mutating call, and the validation logic is code you should have had anyway. What it does force is separating “compute what to do” from “do it”, which is how the workflows should have been structured in the first place.

“Some things can’t be validated without doing them.” True. Say so in the result summary — “would deploy X; no pre-validation available” — rather than skipping the item. Partial plans are still plans.

“We have a test environment.” You have a test environment. A dry run against the target is what catches the collision with the thing that’s already there.

Make it the smoke test

The best consequence: a validateOnly request against a known environment is a regression test for the automation itself, runnable on every change. The factory’s smoke runner does exactly this — request every item with validateOnly: true, assert CREATE_SUCCESSFUL, diff the plan summary against the last run. It takes minutes and it has caught more bugs in the workflows than any amount of code review.

The same form, real or dry-run. One checkbox decides.

Why this matters outside the lab

For anyone who has sat through a failed change window, the value is obvious: a full dry run against the real estate before anything moves. Fewer failed changes, shorter windows, and a plan output that answers the change board’s questions before they’re asked. It also gives auditors something they rarely get from infrastructure automation — evidence of what was going to happen, produced by the same tooling that then did it.

Rules learned

• Add validateOnly to every workflow. Default false. Wrappers cascade it.
• Dry-run does everything but mutate: auth, resolve, generate, call the target’s validator, check collisions, report.
• Where a step truly can’t be pre-validated, say so in the summary. Never skip it silently.
• A dry run against the real target is a plan. A dry run on every change is a smoke test. Same checkbox.
• The refactor it forces — compute, then act — is the one you wanted.

Part of The Lab Factory.

Lab environment; opinions my own.

Porting it to a VCF Automation catalog item (VM Apps — a cloud template plus vRO) is not a rewrite. It’s a sorting exercise: every behaviour in the script has a natural home in the declarative model, and the skill is finding it. Here’s the whole table, then the four rows that took thought.

The mapping

Five homes, in decreasing order of preference: input, template expression, platform abstraction (image mapping, network profile, cloud account), form action (read-only lookup at request time), subscription (imperative work at a lifecycle stage). Push each behaviour as far up that list as it will go.

The four decisions that weren’t obvious

1. Drop auto-detect; the platform already remembers

The script inspected an existing host to infer size. That was a workaround for having no record. The catalog is the record — deployment history shows what size every existing host was requested at — so the input simply asks. Fewer moving parts, and the requester sees the choice.

2. The index scan is a form action, not a workflow step

“Next free esx07” has to be known at request time so the requester sees the names they’ll get. That’s a vRO action bound to the custom form (getNextEsxHostIndexes(environment, role, count) → array of strings), not a step inside provisioning. Forms can call actions; use it for anything that’s a lookup.

3. Rename and folder go in Compute Allocation, hardware goes in Post Provision

Two blocking subscriptions, filtered by a custom property on the template:

The rename must be at allocation — it’s the only stage where a workflow output named resourceNames is applied to the machine. Disk growth and nested-HV need a VM that exists, so they wait for post-provision. Both are blocking: the deployment doesn’t proceed until they return.

4. nestedEsx: 'yes' — not true

The subscription condition is event.data.customProperties.nestedEsx == "yes". Why not "true"? Because boolean-looking strings can arrive in the event payload as typed booleans, and true == "true" is false in the condition evaluator and in the vRO code. It fails silently — the subscription just never fires. yes can’t be coerced. Small thing; two hours.

What stayed exactly the same

The formulas. 10.(20+X).<sub>.0/24, VLAN 2X0n, gateway .254 — they were string concatenation in PowerShell and they’re template expressions now, character for character. The guestinfo.* property names — identical, because the OVA didn’t change. Porting a script well means most of it survives; only the plumbing moves.

The bits that still bite

• Network profile without IP ranges. Addressing is injected via guestinfo, not the platform’s IPAM. Tag the trunk portgroup, add no ranges, or IPAM and guestinfo will disagree.
• Two template revisions in the repo. v1 is what the guide documents; v2 grew later. Both kept deliberately, both labelled. Check which one the org has imported before editing either.
• Content-source lag. A new or changed vRO action needs ~15–20 minutes of data collection before the form sees it. Publish, wait, then test.
• Small disks and OSDATA. Nested hosts with 64 GB disks ship ESX-OSDATA at essentially the whole disk. Templates now provision 128 GB; a relocate-scratch script mitigates existing hosts.

Every menu prompt from the script is now a field; Host Indexes is the form action’s answer to “next free esxNN”.

Why this matters outside the lab

Almost every organisation has these scripts: valuable, trusted, and stuck on one person’s machine. The message of this post for them is that modernising doesn’t mean rewriting. The logic survives; what changes is where it lives — behind a request form with access control, an audit trail, consistent inputs and a deployment record. That’s how a team turns tribal knowledge into a service without losing the years of edge cases the script already handles.

Rules learned

• Porting is sorting: input → expression → platform abstraction → form action → subscription. Push each behaviour as far up as it goes.
• Lookups the requester needs to see are form actions.
• Rename at allocation (resourceNames output); hardware at post-provision. Both blocking.
• Filter subscriptions on a custom property, and make its value a word that can’t be coerced to a boolean.
• Drop workarounds for missing state; the catalog is the state.
• Keep the formulas. Move the plumbing.

Part of The Lab Factory. Next: driving the VCF Installer API from vRO.

Lab environment; opinions my own.

Unlike stage 1 this isn’t VM provisioning, so it’s not a cloud template. It’s a vRO workflow published directly as a catalog item through an Orchestrator content source.

The spec is generated, never edited

A vRO action, buildVcfDeploymentSpec(environment, hostFqdns, labPassword, …), returns the whole spec as a string. Its structure was reconciled against a validated export from a real bringup — the installer UI lets you export the spec it accepted — and everything variable derives from the environment number X:

var n   = parseInt(environment.substring(1), 10);   // "f03" -> 3
var pfx = environment + "-m01";
var net = "10." + (20 + n);
function vlan(o) { return 2000 + (n * 100) + o; }
function gw(sub) { return net + "." + sub + ".254"; }

Static across environments: DNS, NTP, subdomain, component sizes, vSAN ESA FTT=1, and the component build versions pinned to the installer binaries. One lab password feeds every credential field (the UI export scrubs them; the action puts them back per the API schema).

Two spec-level decisions worth stealing:

• skipEsxThumbprintValidation: true instead of carrying per-host sslThumbprint. Supported, and the right trade-off for a lab.
• Ops and Automation are checkboxes that add their blocks to the spec — and the installer only accepts a licenseServerSpec when Ops is present, so the action adds them together or not at all.

The workflow: authenticate → validate → start → return

1. POST /v1/tokens                 installer login
2. POST /v1/sddcs/validations      the installer's OWN pre-flight on the spec
   (poll until COMPLETED; fail on any FAILED check)
3. if validateOnly -> return the validation report; touch nothing
4. POST /v1/sddcs                  start bringup -> sddcTaskId
5. return { sddcTaskId, installerUrl }

Step 2 is the validateOnly story made concrete: the installer will tell you, in seconds, that f03-m01-nsx01.res.lab doesn’t resolve, that an IP is in use, that a host isn’t reachable. Two hours into a bringup is a bad time to learn that. Every one of the ~40 DNS records the pre-flight wants is created ahead of time by a one-shot PowerShell script (New-LabEnvDnsRecords.ps1) — DNS is a prerequisite, not a step.

The two-hour leash, and how to slip it

A request from the catalog carries a token with a roughly two-hour lifetime, and a bringup takes around eight. So by default the workflow is fire-and-forget: waitForCompletion=false, return the task id, watch progress in the installer UI. A watchTaskId input lets you re-attach later and poll an already-running bringup from a new request.

The wrapper that chains everything — hosts, bringup, then the day-N components that need bringup to be finished — has a neater trick. The catalog-bound parent deploys the hosts, submits bringup, and then re-executes itself as a plain vRO run (Orchestrator → Run, no catalog token, no two-hour kill) carrying the hidden bringupWatchTaskId. That continuation polls the installer task to completion — eight hours, fine — and then runs certificates, fleet items, edge, supervisor and identity. Watch it under Orchestrator → Activity → Runs. The catalog request itself completes in under an hour — 47 minutes on the run pictured below — having started the work correctly and handed off.

Two rows, one build. The parent returns inside the catalog’s window; the continuation waits out the bringup and does the day-N work.

Nested-host frictions

Three things a physical bringup never meets:

• HCL validation vs virtual NVMe. The installer’s hardware check blocks the virtual NVMe controller. Fix at the vLCM layer: enforce_hcl_validation=false on the image policy. The vSAN health test nvmeonhcl also complains; silenced via the vSAN API, best-effort with manual fallback.
• DVS compatibility appears late. After bringup, NSX takes 1–2 hours to settle before the supervisor’s zones endpoint stops returning 500. If the supervisor stage fails “No compatible DVS” on a fresh instance, wait and re-run just that item.
• TSM-SSH. Bringup wants SSH on the hosts; the wrapper enables it host-direct via SOAP before submitting.

Stale schema: the failure that looks like a bug and isn’t

Add an input to the vRO workflow after the catalog item exists and the form will show the new field, the request will record its value, and the workflow will receive null — Service Broker keeps the old request schema until the content source re-imports. The workflow null-guards every boolean and aborts with “inputs not mapped” rather than running with silently-wrong options. Fix: re-import the content source, confirm the schema, submit a new request (resubmitting an old one reuses the old payload).

There are actually three async layers between “publish” and “mappable request” — vRO processing the import, the catalog schema after re-import, and the form service still enforcing the previous custom form for a minute or two. Same symptom for all three. Check timing before assuming a bug.

Why this matters outside the lab

Repeatable, generated VCF deployments matter well beyond a lab: a second site, a disaster-recovery instance, a new business unit, an environment per supported release. Generating the specification from a validated reference removes the class of errors that comes from editing hundreds of lines of JSON by hand, and running the installer’s own validation first turns “find out in hour two” into “find out in minute one”. It’s the difference between a VCF deployment being a project and being a procedure.

Rules learned

• Generate the spec from a validated export plus one number. Nobody hand-edits JSON at 2am.
• Run the installer’s own validation first, and make it a mode you can request on its own.
• Pre-create DNS. Enable SSH. Disable HCL enforcement on virtual NVMe.
• Respect the request lifetime: start, return a task id, re-attach. For a long chain, have the workflow re-run itself outside the catalog.
• Null-guard every input and fail loud; stale schemas are a fact of life after adding inputs.
• On a fresh instance, give NSX an hour before you expect DVS compatibility.

Part of The Lab Factory. Previously: porting the host script.

Lab environment; opinions my own. Bringup verified end-to-end on a rebuilt environment: 305/305 tasks, COMPLETED_WITH_SUCCESS.