This is the story of the least helpful error message in my recent memory, what it actually means, and the one-line ordering rule that would have saved an afternoon. If you’re doing self-service NSX VPCs on VCF 9 with the vSphere Supervisor, you will hit this. Bookmark accordingly.

The setup

Tenant-created VPC (via the VCF Automation CCI API), a supervisor namespace pinned to it, and a couple of VirtualMachineService objects of type LoadBalancer to publish SSH and HTTPS for the workloads inside. Standard stuff — the exact pattern that works out of the box in the org’s default VPC.

The k8s side looked perfect:

$ kubectl get endpoints -n pod-a
NAME           ENDPOINTS                        AGE
esx01-access   172.30.0.40:443,172.30.0.40:22   6m36s
esx02-access   172.30.0.41:443,172.30.0.41:22   6m35s

Endpoints resolved. VIPs: nothing. The only clue, a recurring event:

Warning  FailedRealizeNSXResource  service/esx01-access
Generic error occurred during realizing network for Service

“Generic error.” Wonderful.

Digging: what NCP actually wants

The supervisor’s network container plugin (NCP) logs told the real story:

nsx_ujo.ncp.nsx.policy.lb_layer4_service Lb Service not Found for Namespace pod-a
NCP00270 Failed to process virtual ip for service ...: Lbs pod-a is not found
Encountered retryable error ... : Lbs pod-a is not found

NCP wants an NSX LBService in the namespace’s VPC. In the org’s default VPC, one exists — the platform created it when the VPC was born. In my self-service VPC? Nobody had created one. Fair enough — that’s actually documented behaviour once you know where to look: a fresh VPC needs a LoadBalancer object (and before that, a VPCAttachment to a connectivity profile with the service gateway enabled, or the LB creation itself fails with a much better error message).

So I created the attachment, then the LBService. NSX: Realized=True. Problem solved?

Warning  FailedRealizeNSXResource  service/esx01-access
Generic error occurred during realizing network for Service

No.

The actual bug-shaped behaviour: a snapshot, not a lookup

Here’s the part that costs you the afternoon. That “retryable error” retries the lookup in NCP’s cache — not the discovery. NCP snapshots the VPC’s LB inventory when the namespace is created. An LBService that appears afterwards is never discovered, no matter how long you wait:

• Recreating the k8s services: no effect.
• Tagging the LBService with the nsx-op/* ownership tags the working ones carry: no effect — the cache doesn’t re-read NSX.
• Restarting NCP would force a full resync — but supervisor system pods are protected; even Administrator@vsphere.local gets a Forbidden.
• Mutating the namespace to nudge a re-sync: also blocked, by the supervisor’s namespace validation webhook.

As a tenant, there is exactly one fix: delete and recreate the namespace, now that its VPC has an LB. Fifteen minutes of rebuild for want of one ordering rule.

And the control experiment proves the rule: a namespace created after its VPC already had an LBService got its VIPs assigned without any drama —

esx01-access   VIP=192.168.144.34   22 OPEN · 443 OPEN
esx02-access   VIP=192.168.144.35   22 OPEN · 443 OPEN

And once the ordering is right, this is what “working” looks like — the pod’s state a couple of minutes after a correctly-ordered deployment:

What the requester sees once the order is right.

The ordering rule

For every self-service VPC that will publish LoadBalancer services, create — in this order, before the namespace:

1. VPC                                   (vpc.nsx.vmware.com/v1alpha1)
2. VPCAttachment                         (connectivity profile w/ service gateway
                                          — LB creation errors without it)
3. LoadBalancer   {regionName, vpcName}  (the step everyone misses)
4. ...and only THEN the Supervisor Namespace

Encode it in whatever provisions your VPCs — a script, a pipeline, an operator. It’s four API calls and it turns a silent, undiagnosable <pending> into a platform that just works.

Why this matters outside the lab

Nobody buys a platform for its ordering rules — but this is exactly the kind of edge that decides whether self-service provisioning feels reliable or flaky to the people using it. In a customer deployment the answer isn’t a blog post; it’s that the provisioning automation already does the four steps in the right order, every time, so a tenant never sees a VIP stuck at <pending>. Knowing where the sharp edges are — because you’ve been cut by them in a lab — is most of what an experienced delivery partner is for.

Rules learned

• In a self-service NSX VPC, the LBService must predate the namespace. NCP discovers LBs at namespace-add and never again.
• FailedRealizeNSXResource: Generic error on a Service = go read the NCP logs; the real message (Lbs <ns> is not found, NCP00270) is there.
• VPCAttachment (service gateway) is the prerequisite for the LB itself — that one at least fails loudly.
• Retro-tagging NSX objects to look “owned” doesn’t help a cache that never re-reads. Recreating the namespace is the only tenant-level fix.
• While you’re at it: new namespaces also reject VM creation until image status.disks syncs (~1–3 minutes after content library attach). Build the wait into your automation and both sharp edges disappear.

Previously in this series: nested ESXi inside an NSX VPC. Next: three datacenters, one IP plan — identical isolated pods.

Lab environment; opinions my own. Output captured live, trimmed for length, never edited for outcome.

1. ${input.x} is illegal inside a flow mapping

This looks like valid YAML and valid blueprint syntax:

- {key: guestinfo.hostname, value: {value: "esx01.${input.podName}.res.lab"}}

It fails content validation. The expression parser doesn’t reach into flow-style ({...}) mappings. Block style is fine:

- key: guestinfo.hostname
  value:
    value: esx01.${input.podName}.res.lab

Mixed style in the same list is fine too — only the entries that carry an expression need to be block-style. (This is why the blueprint’s vAppConfig list looks inconsistent; it’s deliberate.)

2. name vs generateName for a new namespace

A CCI.Supervisor.Namespace you’re creating must use generateName. metadata.name is rejected by the CCI API — the platform appends a random suffix, so pod-a- becomes pod-a-dgf5p. Everything downstream should reference ${resource.namespace.id}, never a literal name.

3. Zones and storage classes are flat

Early attempts wrapped them the way the raw CCI API does:

initialClassConfigOverrides:
  zones: [...]

In a blueprint they’re top-level properties of the namespace resource:

zones:
  - name: domain-c9
    cpuLimit: 40000M
    memoryLimit: 64000Mi
storageClasses:
  - name: vSAN Default Storage Policy
    limit: 400000Mi

And zones are required — omit them and the API says “Zone should be specified”, which at least is a clear message.

4. A new namespace has no content library

Deploy the namespace, deploy a VM, and get: no VirtualMachineImage found. A VCFA-created namespace attaches no content libraries by default. The fix is one block:

contentSources:
  - {name: ISO, type: ContentLibrary}
  - {name: f06-vks-lib01, type: ContentLibrary}

Without it you’re in the vSphere Client attaching libraries to a namespace by hand, which rather defeats the catalog.

5. Images sync after attach — wait for status.disks

Even with libraries attached at creation, the first VM create in a fresh namespace can be rejected:

no disks found in image ... status.disks

The image objects appear immediately; their disk metadata syncs over the next 1–3 minutes. The quota webhook checks status.disks and refuses until it’s populated. In a blueprint, put the hosts dependsOn something that takes a couple of minutes (the binding maps did the job here), or add an explicit wait. In a script, poll:

kubectl get virtualmachineimage -n <ns> <vmi> -o jsonpath='{.status.disks}'

6. Validation lives in status, not the HTTP code

Creating a BlueprintVersion returns 200 whether or not the content is valid. Read the object back:

status:
  contentValid: false
  validationMessages:
    - "... unexpected token ..."

If your pipeline checks the response code, it will happily publish a broken blueprint. Check status.contentValid and print the messages.

7. Only one published version — 409 on the second

Release 1.1.0 while 1.0.0 is released and you get a 409. It isn’t a transient conflict; it’s the rule. Unrelease the current version, then release the new one. Practically that means a publish step is unrelease old → release new, and there’s a short window where the catalog item has no released version. Do it when nobody’s requesting.

Bonus: the things that aren’t blueprint problems

Three prerequisites have no blueprint resource type and have to exist before the request — VPC, VPCAttachment, LoadBalancer, in that order. The blueprint’s vpcName input says “must exist and be Realized”, and it means it. Nothing in the blueprint fails if they’re missing; the deployment just never gets a VIP.

Why this matters outside the lab

VCF Automation’s All Apps model is new, and new platforms have edges. None of these seven are documented; all of them stall a first project by days if you meet them cold. The value of a delivery partner who has already built on the platform isn’t the YAML — it’s that a customer’s first blueprint publishes on day one instead of week two, and that the sharp edges are encoded into templates and provisioning scripts where users never meet them.

Rules learned

• Expressions need block-style YAML; flow mappings don’t get parsed.
• generateName, and reference the namespace by ${resource.x.id}.
• zones and storageClasses are flat and zones are required.
• contentSources on the namespace, or nothing can be deployed.
• Wait for image status.disks before the first VM (1–3 min).
• Check status.contentValid — the HTTP code lies by omission.
• One released version per blueprint: unrelease, then release.

Companion to a datacenter in a catalog tile.

Lab environment; opinions my own. Error text captured live.