That’s a state machine, and it’s built from three very boring mechanisms.

The three mechanisms

1. Master kill switch. If 100_build_complete.flag exists, exit immediately. A stray task run after completion does nothing.

2. Per-step flags. Every install step writes NN_<Step>_installed.flag on success and is skipped on subsequent runs.

3. Persisted state. data-state.json holds the provisioning start time and per-step timings, re-loaded on every boot. So the final report shows the true duration of every step across the whole build, not just the last boot — and a step re-observed as SKIPPED after a reboot never overwrites its real recorded duration.

every boot:
  if 100_build_complete.flag → exit
  load data-state.json
  for step in $softwarePayload:
      if NN_step_installed.flag → SKIP (keep recorded timing)
      run installer from local Staging (timeout ceiling)
      exit 0 or 3010 → write flag, record timing
      if step.RebootAfter → Restart-Computer; exit      ← next boot resumes here
  cleanup → validate → publish → self-destruct → final reboot

Phase 1: software, declaratively

The stack is a single array — the designed extension point:

$softwarePayload = @(
  @{ Name='MonitoringAgent'; Folder=$p.monPath;  File=$p.monInstallFile;  Timeout=20; Reboot=$false;
     Service='HealthService';       Path='...\Monitoring Agent\HealthService.exe' },
  @{ Name='InventoryAgent';  Folder=$p.invPath;  File=$p.invInstallFile;  Timeout=15; Reboot=$false;
     Service='InventoryAgent';      Path='...\Inventory\agent.exe' },
  @{ Name='EndpointSecurity';Folder=$p.epsPath;  File=$p.epsInstallFile;  Timeout=30; Reboot=$false;
     Service='masvc';               Path='...\Agent\masvc.exe' },
  @{ Name='LogAgent';        Folder=$p.logPath;  File=$p.logInstallFile;  Timeout=15; Reboot=$false;
     Service='LogAgentService';     Path='...\Log Agent\liwinsvc.exe' },
  @{ Name='WindowsUpdates';  Folder=$p.updPath;  File=$p.updInstallFile;  Timeout=45; Reboot=$true }
)

Adding a product is adding an element. Each installer runs from the local staging copy with a hung-installer timeout; exit codes 0 and 3010 (success, reboot required) count as success. When a Reboot=$true step succeeds, the script restarts the machine and exits; on the next boot the task re-runs it, completed steps skip via their flags, execution resumes at the next step.

Post-install health isn’t “installer said OK” — it’s flag present AND service running AND install path exists, with a 120-second wait for delayed-start services. Windows Updates is flag-only; there’s no service to check.

Phase 2: cleanup — leave nothing behind

• Eject the config-drive ISO. In Session 0 there’s no Explorer, so the usual shell ejection doesn’t work. A P/Invoke to winmm’s mciSendString("set cdaudio door open") does.
• Restore UAC. EnableLUA and FilterAdministratorToken were relaxed in the base image so the build could run unattended; re-enable both.
• Uninstall cloudbase-init. Stop and delete the service, kill its processes, run the uninstaller silently, remove the directory. Guarded by 99_cleanup_complete.flag.

The startup task is deleted before validation, so the health check can truthfully report “no automation task remains”.

Phase 3: collect the evidence

The script assembles data-validation.json — the single input to the report in part 3:

The verdict is strict: GuestStatus = Success only if every software item is healthy and the machine is domain-joined. Anything else renders the report banner red.

Phases 4 and 5: publish, then self-destruct

Remap the share with the write account (5 × 10 s retries — a different account from the read-only one used for pulls; a compromised build guest can’t tamper with the engine). Render the HTML report locally. Write 100_build_complete.flag — kill switch armed. Copy the report plus Data\, Flags\, Logs\ to \\share\Builds\<image>\<HOSTNAME>\. Stop the transcript before copying logs so the final log is complete and unlocked.

Then: delete the startup task; overwrite both share passwords in the on-disk payload with *** SCRUBBED ***; delete Data\ and Staging\ (and Flags\/Logs\ if the share copy succeeded); delete the sibling scripts and itself; remove Scripts\; reboot one final time.

The delivered server boots clean, domain-joined, agents running, and carries no credentials, payloads or tooling.

The reboot sequence of a nominal build

1. After static networking (00, exit 1003)
2. After engine pull (03, exit 1003) — ends the cloudbase-init phase
3. After Windows Updates (05, Reboot=$true)
4. Final, after publish and self-destruct

The report’s timeline chart finds the update reboot automatically: any gap over 30 seconds between steps is shaded and labelled REBOOT.

At completion the Flags\ folder is the whole history of the build in file names:

Flags  00_config-network.flag
  01_config-disks.flag
  02_join-domain.flag
  03_init-puller.flag
  04_stage-payloads.flag
  01_MonitoringAgent_installed.flag
  02_InventoryAgent_installed.flag
  03_EndpointSecurity_installed.flag
  04_LogAgent_installed.flag
  05_WindowsUpdates_installed.flag
  99_cleanup_complete.flag
  100_build_complete.flag        <- kill switch

And the persisted timings turn into this, in the report from part 3:

Every bar sits at its true wall-clock position across the reboots because the state file carried the timings. The amber band is the reboot after Windows Updates, found automatically from the gap.

Why this matters outside the lab

What customers get from a reboot-safe build is predictability: every server takes the same steps in the same order, survives the reboots Windows insists on, and finishes clean — with no tooling, no credentials and no leftover tasks on the delivered machine. That last part matters to security reviewers as much as the first part matters to operations. And because every step’s timing is recorded, “why did this build take twice as long?” has an answer instead of a guess.

Rules learned

• A reboot-safe build is kill switch + per-step flags + persisted timings. Nothing cleverer is needed.
• Treat exit 3010 as success. Let the step declare whether to reboot; the loop handles it.
• Health = flag and service and path. Installer exit codes lie.
• Read platform-injected metadata with retries — the injecting workflow may land after the guest starts looking.
• Two share accounts: read-only for pulls, write-only for publishing.
• Delete the task before validating, so “no task remains” is checkable.
• Stop the transcript before you copy the logs.
• Scrub, delete yourself, reboot. The customer gets a server, not a build environment.

Part 3: validation as a product, and the details that hurt.

Lab write-up of a production pattern; customer specifics removed. Opinions my own.

This three-part series is how that works. It’s a VM Apps build (classic Aria Automation cloud template plus vRO), and it’s a good example of the pattern: the platform does the allocation and the metadata; the guest does the guest. Part 1 is the architecture. Part 2 is the state machine that survives four reboots. Part 3 is the validation report and the details that hurt.

Customer-specific names, products and policies are generalised throughout. The patterns are the point.

Three layers, three reasons

┌──────────────────────────────────────────────────────────────────────┐
│ 1. Aria Automation + Orchestrator         (control plane)            │
│    cloud template · Event Broker → vRO at Allocation / Post-Provision │
│    hostname from AD · placement metadata → guestinfo · notifications  │
├──────────────────────────────────────────────────────────────────────┤
│ 2. cloudbase-init                          (first boot, template-owned)│
│    multipart userdata: network · disks · domain join · pull engine    │
├──────────────────────────────────────────────────────────────────────┤
│ 3. File-share build engine                 (guest state machine)      │
│    stage payloads · install across reboots · validate · report · self-│
│    destruct                                                           │
└──────────────────────────────────────────────────────────────────────┘

Each seam exists for a reason you can state in one sentence:

• Aria/vRO owns what needs platform credentials — querying AD for the next hostname, reading vCenter for where the VM landed. The guest never holds a vCenter or AD-admin credential.
• cloudbase-init owns what must happen before anything else — the network has to work before the share can be reached; the disk layout has to exist before installers land on it; and the domain join changes the security context everything after it runs in, which is the whole reason the next layer exists (more on that below).
• The engine is pulled, not embedded — because installers change, agents get new versions, and re-releasing a cloud template for every payload update is how automation dies. Update a script on the share; every subsequent build gets it.

vCenter guest customization is disabled (customizeGuestOs: false). One code path owns hostname, networking, disks and join; two would fight.

The control plane: two vRO hooks that matter

Compute Allocation → sequential hostname. The template composes a prefix from location/classification/environment/application inputs. A vRO workflow queries AD computer objects with that prefix and allocates highest existing suffix + 1. Gaps are never reused: with -001, -002, -003 and -005 present, the next is -006. A gap usually means a deleted machine whose name may still live in DNS, a backup catalogue or the CMDB; reusing it is how you restore the wrong server.

Compute Post Provision → placement metadata into the guest. A second workflow asks vCenter where the VM actually landed — vCenter, datacenter, cluster, host, folder, datastores, NIC→portgroup map — and writes it as JSON into the VM’s extraConfig as guestinfo.vra.infrastructure. Later, inside the guest, vmtoolsd.exe --cmd "info-get guestinfo.vra.infrastructure" reads it back. That’s the bridge that lets the build report say “this VM is on host X, datastore Y” with zero vCenter credentials in the guest.

Three more Post Provision workflows email the backup team, the SOC and the requester. (One observation for part 3: the requester’s “your deployment has completed” fires here — before the in-guest build has even started.)

First boot: cloudbase-init, four scripts, two reboots

The template’s cloudConfig is a MIME multipart: one cloud-config part (hostname + write the tags to disk) and four #ps1_sysnative scripts, executed in order. Three conventions make them idempotent:

• Flag files under Flags\ — a script exits immediately if its flag exists, so re-runs after a reboot are no-ops.
• Transcripts under Logs\, one per script.
• Exit 1003 = “reboot me and run this part again on next boot” (the flag then short-circuits it). Exit 0 = done. Exit 1 = failure.

The network script is the one with an assumption worth writing on the wall: it pairs OS adapters with the request’s NICs positionally, which is valid for freshly cloned VMs where NICs were added in PCI order. If anyone ever customises NIC order post-clone, revisit it.

Why not just cloudbase-init (or guest customization) all the way down?

The obvious design is the one we started with: let the platform do the guest. vCenter guest customization for hostname, IP and domain join, then one cloudbase-init userdata that installs the agents, patches, and reports back. No scheduled tasks, no pulled engine, no state machine. It works on a workgroup machine. It stops working the moment the machine joins a real domain, and it stops in a way that is easy to misread.

The domain join changes the rules mid-build. Everything up to the join runs as a fresh, local, un-managed Windows install: local Administrator, default execution policy, no central policy. At the join the machine lands in its target OU, and on the next policy refresh (which the reboot guarantees) Group Policy applies. In this environment the policy set for member servers includes the usual security baseline: script-execution controls, restrictions on what may run from where and under which accounts, and hardening of the local administrator context. None of that is negotiable, and none of it should be — it is the same policy every production server gets.

What that does to a first-boot pipeline. cloudbase-init runs its plugins as a service, as LocalSystem, executing scripts from its own directory. Before the join that context can do anything. After the join it is exactly the kind of context the baseline is designed to constrain, so the parts of the build that come after the join — the installer pulls, the agent installs, the reboots, the validation — either fail outright or, worse, quietly do nothing: cloudbase-init logs the plugin as executed, the script never ran anything, and the build “completes” with an unpatched server carrying no agents. The first few builds looked exactly like that.

Two ways out, one of them wrong. You can relax policy for the build (a staging OU with a weaker baseline, a GPO exemption for the cloudbase-init path, a delayed join) — which means the server is built under one set of rules and delivered under another, and the join becomes a late step that nothing after it exercises. Or you can accept the policy as the environment it is, and run the post-join work in a context the policy permits.

The permitted context is a scheduled task under an explicit identity. A task registered to run as a named account — a domain-joined local admin for the pull and the join verification, SYSTEM for the build master — at highest run level, with -ExecutionPolicy Bypass per invocation, from a staging path the policy allows, is an ordinary, auditable pattern that the baseline was written to accommodate. That is why the OS commands aren’t run by cloudbase-init but delegated to tasks: cloudbase-init’s job shrinks to “get the network up, lay out disks, join, hand off”. The design also picks up three things it now depends on:

• a per-task execution ceiling (1 h for the pull, 2 h for the build) that contains a hung installer instead of leaving a half-built VM;
• an at-startup trigger, which is what makes a multi-reboot state machine possible after cloudbase-init has been uninstalled;
• a clean security story: the identities that do the work are the ones the domain already governs, and they stop existing on the box when the build is done.

Why not Aria’s own in-guest mechanisms? Guest customization only covers hostname, IP and join, and having it and cloudbase-init own the same settings means two code paths fighting (hence customizeGuestOs: false). Driving the guest from outside — vRO calling into the VM for two hours — needs guest credentials held centrally and keeps a management path open for the whole build; ABX can’t reach inside the guest at all. The platform’s job is what needs platform credentials: the hostname from AD, the placement facts from vCenter. The guest does the guest, under the domain’s rules, from the first reboot after the join.

Even if the GPO were relaxed tomorrow, don’t simplify the wrappers away: the ceiling and the startup trigger are worth having on their own.

The hand-off: 03-init-puller

This is where template-embedded code stops and the centrally managed engine starts:

1. Create Flags\ Logs\ Data\ Scripts\ under the log folder.
2. Write data-payload.json — the one document everything after this reads: share path and accounts, image, project/deployment/requester, the installer folder/file pairs, the tags, the placement facts.
3. Encrypt the two share passwords with AES, key derived from the machine’s BIOS UUID. The payload is useless copied off-box.
4. Generate a runner, execute it via a one-shot task as local admin: map the share read-only, copy 04-stage-payloads, 05-build-master, 06-validate-build locally, unmap, launch staging.
5. Write the flag, exit 1003. Reboot 2. cloudbase-init’s job is done.

04-stage-payloads copies every installer to local disk (installers never run across SMB — immune to network blips and file locks mid-install) and registers the Build-Master startup task. From here on, every boot runs 05-build-master.ps1 until the build is complete.

What the requester actually sees is a short form. Stripped of the site-specific enum values, the inputs are:

inputs:
  location:        # site code -> hostname prefix
  classification:  # security zone -> hostname prefix, OU
  environment:     # prod / pre-prod / test -> hostname prefix, tags
  application:     # application code -> hostname prefix, folder
  image:           # Windows2025 (the template is image-versioned)
  flavor:          # Small / Medium / Large -> vCPU + RAM
  count:           # number of identical machines
  bootDiskSizeGB:  # C: (extended in-guest by 01-config-disks)
  primaryNetwork:  # required
  network2..4:     # optional; each becomes a static NIC
  additionalDisks: # [{number, name, letter, sizeGB}] -> D:, L:, ...
  tags:            # free-form key/value -> written to disk, shown in the report

Every one of those either shapes the hostname, lands in guestinfo, or is written to disk for the build engine to read. Nothing is entered twice.

Why this matters outside the lab

For an organisation, this pipeline turns a Windows server from something a person builds into something the platform delivers: a request in a catalog, a domain-joined and agent-loaded server out, with the security team’s controls (naming, join, hardening, monitoring agents) applied every time because they’re in the pipeline, not in a checklist. The separation of concerns is what makes it maintainable — the platform holds the credentials, the guest does the work, and a change to the software stack is a script update on a share rather than a template re-release.

Rules learned

• Split by who holds the credential: platform queries AD/vCenter, guest never does. guestinfo is the one-way bridge.
• Disable vCenter customization when cloudbase-init owns the guest. One owner.
• Pull the engine from a share; embed only what must run before the network exists.
• Flag files + transcripts + exit 1003 = idempotent, reboot-safe first-boot scripts.
• Anything after the domain join runs in a scheduled task under an explicit identity — for policy, for the execution ceiling, and for the startup trigger.
• Never reuse hostname gaps.

Part 2: the state machine that survives four reboots.

Lab write-up of a production pattern; customer specifics removed. Opinions my own.