kubedoom       192.168.144.20:5900   (VNC — yes, it kills pods)
kubeinvaders   192.168.144.21
kube-ops-view  192.168.144.22
pacman         192.168.144.23        (+ MongoDB on a PVC — persistent high scores)
podinfo        192.168.144.24
goldpinger     192.168.144.25        (DaemonSet incl. control plane)
prometheus     (in-cluster: KSM + node-exporter, 11/11 targets up)

Captured from the VIPs the platform handed out. KubeDoom is VNC-only and podinfo is an API, so they sit this one out; the Ops topology tile shows all seven by name.

The path that matters: tenanted, not shortcut

The first version of this stack was deployed against the supervisor — an admin kubeconfig, a vSphere namespace, kubectl apply. It worked and it was wrong, for the reason the previous post spells out: nothing about it was provided to anyone. So it was torn down (seven apps, cluster, VPC and VIPs gone in about seven minutes) and rebuilt the proper way:

dev-01 org → default-project → SupervisorNamespace (class large, region f06, VPC default-f06)
          → VKS cluster vks-demo01 → seven apps

All created through VCF Automation — the CCI API for the namespace and cluster, then the apps via the cluster’s own kubeconfig. Two auth facts worth writing down, because they cost a cycle each:

• A provider service account reaches the cloud API only. CCI (/cci/kubernetes/apis/...) rejects provider tokens with 401 — it needs an org-scoped service account.
• Device-flow login uses the service account’s own UUID as client_id (not its software ID), against the tenant endpoint /oauth/tenant/<org>/device_authorization.

What the platform does for free

Each app is an ordinary Deployment + Service of type LoadBalancer. The supervisor turns each service into an NSX VPC LB virtual server and hands back a VIP from the org’s external block. No ingress controller, no MetalLB, no port-forwarding — seven services, seven VIPs, done. Pac-Man’s MongoDB asks for a PVC and gets a vSAN-backed volume through the CSI the supervisor already installed. Goldpinger runs as a DaemonSet across every node including the control plane and draws the node-to-node mesh live.

That’s three platform services (LB, storage, networking) exercised by apps that know nothing about VCF.

The traps

Supervisor refuses cluster-scoped RBAC — even to admin. Demo apps that need ClusterRoles (kube-ops-view, Goldpinger, KubeDoom) must live on a guest cluster. You cannot run them as vSphere Pods on the supervisor.

PodSecurity restricted is the VKS 1.35 default. Half the demo set runs as root. Symptom: Deployment shows 0 UP-TO-DATE, ReplicaSet exists, zero pods, events say FailedCreate. Fix: label the namespace pod-security.kubernetes.io/enforce=privileged — and mention in the demo that you did, because it’s a teaching moment.

node-exporter without hostNetwork. The VPC fabric blocks pod → node IP scrapes, so the stock DaemonSet’s hostNetwork: true doesn’t help; run it as a normal pod and let Prometheus scrape it in-cluster.

Docker Hub is flaky from behind a proxy. TLS handshake timeouts put pods into kubelet’s image-pull backoff. Deleting the stuck pods bypasses the backoff; a Harbor proxy-cache project fixes it properly.

Pod CIDR shadowing. The stock 192.168.0.0/16 pod range hid the org’s 192.168.144.0/21 external block from inside the cluster — apps couldn’t reach their neighbours’ VIPs. Pod CIDR is now 172.16.0.0/16.

Automation notes

The whole stack is a checkbox on the lab’s catalog item that deploys a supervisor: installDemoApps creates the cluster (newest compatible Kubernetes release, newest built-in ClusterClass, auto-detected), applies the seven apps, and reports their URLs in the deployment summary. Run integrated on a fresh environment: 16.7 minutes to CREATE_SUCCESSFUL. An immediate re-run: 3.3 minutes, all steps idempotent — which is the number I actually care about, because it means a broken demo is a re-run, not a rebuild.

Why this matters outside the lab

A demo stack sounds like a toy. It’s actually the fastest way to make a platform legible to people who don’t read YAML: a customer watches a request become a cluster, watches seven services get their own addresses, opens one and plays it. Everything underneath — self-service Kubernetes, load balancing, persistent storage, isolation — is being exercised in a way a non-technical stakeholder can see working. The same stack is what we put in front of a new team on day one, and the same idempotent deploy is what makes it safe to demonstrate live: if it breaks on stage, it re-runs in three minutes.

Rules learned

• Demo apps that need cluster-scoped RBAC must run on a guest cluster; the supervisor won’t grant it, even to admin.
• Build the stack through the tenanted path (org SA → CCI → namespace → cluster → apps). Same apps, but now they’re provided, quota’d and visible in Ops.
• VKS 1.35: restricted PodSecurity by default. Label the namespace and say why.
• LoadBalancer per app = NSX VIP per app. No ingress needed for a demo.
• Pick a pod CIDR that doesn’t overlap the VPC external block.
• Make the deploy idempotent; a demo that re-runs in 3 minutes is one you can afford to break on stage.

Previously: one VKS cluster, two ways. The Pac-Man instance here is the one from What’s a VPC?.

Lab environment; opinions my own.

The lab has Pac-Man running twice on VCF 9.1 — once on a VKS cluster I built by hand with kubectl, once on a cluster deployed through VCF Automation’s catalog. Both live inside VPCs. Both were reachable when I started:

http://192.168.144.15/  ->  <title>Pacman in HTML 5 Canvas
http://192.168.144.23/  ->  <title>Pacman in HTML 5 Canvas

A VPC is a private universe with a door policy

Think of an NSX VPC as a tenant’s own routed network space: its own subnets, its own gateway, its own address plan — carved out by the tenant, not filed as a ticket with the network team. Three rules define it:

1. Private by default. A Private subnet is reachable only from inside the same VPC. Nobody outside can route to it — not other tenants, not other VPCs in the same org, not the corporate network.
2. Your addresses are your business. Because private subnets aren’t advertised anywhere, two VPCs can use identical CIDRs. (This is the superpower the rest of the series is built on.)
3. Every door out is deliberate. Traffic leaves via the transit gateway — SNAT’d — or arrives via a LoadBalancer VIP from an external block the provider allocated. Nothing is exposed by accident.

Pac-Man’s pods sit on a private subnet. The only reason 192.168.144.23 answers is a Kubernetes Service of type LoadBalancer, which NSX turns into a VIP on the VPC’s load balancer. So let’s remove the door.

Before: close the door

$ kubectl patch svc pacman -n pacman -p '{"spec":{"type":"ClusterIP"}}'
service/pacman patched
NAME     TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
pacman   ClusterIP   10.106.219.213   <none>        80/TCP    4d13h

$ curl -m 5 http://192.168.144.23/
curl: timed out / unreachable

The pods are running. The service exists. The game is fine — for anything inside the VPC. From my desk it’s simply gone. That’s the whole VPC model in one curl: the boundary isn’t a firewall rule somebody wrote, it’s the absence of a route.

After: open it again

$ kubectl patch svc pacman -n pacman -p '{"spec":{"type":"LoadBalancer"}}'
service/pacman patched
NAME     TYPE           CLUSTER-IP       EXTERNAL-IP      PORT(S)        AGE
pacman   LoadBalancer   10.106.219.213   192.168.144.23   80:31467/TCP   4d13h

Same VIP handed straight back. NSX programmed a virtual server and pool on the VPC LB; the supervisor stitched it to the cluster’s NodePort. Door open.

And then the game didn’t load.

The outage I gave myself (this is the useful bit)

Everything was green:

• kubectl get svc — LoadBalancer, VIP assigned
• kubectl get endpoints — pod IPs present
• NSX — virtual server up, pool members healthy
• iptables on the node — NodePort rules identical to a working neighbour service

Five layers, all green, and curl hung. The control experiment was the manually-built cluster’s Pac-Man at .15, untouched throughout, still playing.

The cause: my “harmless” ClusterIP patch earlier had included a ports list. kubectl patch with a merge patch replaces arrays, it doesn’t merge them — and my array said targetPort: 80. Pac-Man listens on 8080. Every layer above was faithfully forwarding traffic to a port nothing was listening on, and every layer reported success because its job was done.

$ kubectl patch svc pacman -n pacman --type=json \
    -p '[{"op":"replace","path":"/spec/ports/0/targetPort","value":8080}]'
service/pacman patched
$ curl -s http://192.168.144.23/ | grep -o '<title>.*</title>'
<title>Pacman in HTML 5 Canvas</title>

Instant recovery. The diagnosis walked the entire paravirtual chain — VIP → supervisor VirtualMachineService → NSX VS/pool → NodePort iptables → pod — and it’s exactly the walk you’ll need one day:

Bonus find on the way: kube-proxy and Antrea on that cluster had dropped their API watches days earlier (http2: client connection lost) and never re-established informers until restarted. It didn’t cause this outage, but it’s the kind of thing you only find when you’re forced to look.

Why this matters outside the lab

If you run a platform for more than one team, this is the feature you’ve been asking the network team for. A VPC gives each team, project or customer its own private network space — created by them, in minutes, with nothing reachable from outside until they publish it. Security teams like it for the same reason developers do: exposure is a deliberate, auditable act, not a side effect of plugging something in.

What organisations do with it once they have it:

• Per-team sandboxes that can’t see each other, provisioned without a ticket.
• Partner or supplier environments isolated from the corporate estate but hosted on the same platform.
• Multi-tenant hosting — service providers and internal IT alike — with isolation enforced by topology rather than a growing pile of firewall rules.

Rules learned

• A VPC’s boundary is the absence of a route, not a rule. Private subnets are unreachable from outside by construction — which is also why identical CIDRs across VPCs just work.
• A LoadBalancer service is the deliberate door: NSX VIP from the external block, programmed per service. Flip the type and the door closes with nothing else to clean up.
• kubectl patch (merge) replaces spec.ports, it doesn’t merge it. Patch a single field with --type=json, or don’t include the array.
• Five green layers can hide one wrong integer. Keep a working control (here: the untouched .15 instance) and compare layer by layer.
• Read kubectl get endpoints as IP:targetPort — the port is the part people skim.

Next in the Pod Papers: nested ESXi inside a VPC — where “private by default” meets a host that fakes its own MAC address.

Lab environment; opinions my own. Output captured live, trimmed for length, never edited for outcome — including the outage.

So: two clusters, one supervisor, one ClusterClass, two paths.

Path A: kubectl, the way we’ve always done it

kubectl vsphere login --server <supervisor> --tanzu-kubernetes-cluster-namespace demo
kubectl apply -f cluster.yaml

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
spec:
  clusterNetwork: { pods, services, serviceDomain }   # set all three — see below
  topology:
    class: builtin-generic-v3.6.0
    classNamespace: vmware-system-vks-public          # the gotcha
    version: v1.35.5+vmware.1
    controlPlane: {replicas: 1}
    workers: { one node pool, 2 replicas }
    variables: [ vmClass, storageClass ]

Fifteen minutes later: a cluster. Requires a vSphere namespace that somebody (an admin) created, with a content library attached, a VM class assigned and quota set — all in the vSphere Client, by hand.

Path B: the same manifest, as a catalog request

In All Apps the cluster is a CCI.Supervisor.Resource inside a blueprint, sitting next to a CCI.Supervisor.Namespace:

resources:
  namespace:
    type: CCI.Supervisor.Namespace
    properties:
      generateName: ${input.name}-
      className: large                      # quota comes from the class
      regionName: f06
      vpcName: ${input.vpc}
      contentSources: [{name: f06-vks-lib01, type: ContentLibrary}]
  cluster:
    type: CCI.Supervisor.Resource
    properties:
      context: ${resource.namespace.id}
      manifest: <the SAME Cluster object as above>

Publish it, and a tenant user requests it from a tile:

Fifteen minutes later: a cluster. Byte-identical Cluster object.

What path B adds for free

That last row is the one operations teams care about:

The VCFA-deployed cluster appears in Ops’ object model — supervisor → namespace → cluster → nodes → the workloads running on it — and its demo apps show up by name in the topology tab. The kubectl-built cluster is also visible to Ops (it’s the same supervisor, after all), but it has no deployment, no owner, no request history and no quota lineage. It’s a thing that exists, not a thing that was provided.

The four gotchas, in order of how much time they cost

Both paths share the same four traps on VKS 1.35 / VCF 9.1:

1. A VCFA-created namespace has no content library. Zero VirtualMachineImages → no cluster possible. contentSources in the blueprint, or attach by hand.
2. ClusterClass lives in vmware-system-vks-public. It 404s from the workload namespace unless the spec sets topology.classNamespace.
3. The default Quick Start namespace is 1000M CPU / 1000Mi. Unusable for a cluster. Use a real class: small 10000M/10000Mi, medium 20000M, large 40000M.
4. VKS 1.35 enforces PodSecurity restricted by default. Demo apps that run as root get a ReplicaSet and no pods; the events say FailedCreate. Label the app namespace pod-security.kubernetes.io/enforce=privileged (or fix the apps).

Plus one that only shows up later: set clusterNetwork.serviceDomain explicitly. It’s immutable after create, and a cluster without it produces a service DNS name that some add-ons build wrongly (....svc. with no domain). And pick a pod CIDR that doesn’t shadow your VPC’s external range — the stock 192.168.0.0/16 hid the org’s 192.168.144.0/21 from inside the cluster.

So which one?

Use kubectl when you’re the platform team proving something on a supervisor, or debugging. Use All Apps the moment a second person needs a cluster: the request form is the interface, the namespace class is the guardrail, the deployment is the audit trail, and Ops sees it as a provided service rather than a stray object.

The cluster’s the same either way. The service isn’t.

Why this matters outside the lab

The business case for the second path is governance without friction. Development teams get Kubernetes clusters on request; the platform team gets quotas, ownership, RBAC and a monitoring view of every cluster for free. That’s the difference between a managed Kubernetes service and a collection of clusters nobody can account for — and it’s typically the gap that stops organisations offering Kubernetes broadly at all. Everything the developers touch stays standard Kubernetes; the control lands around it, not on it.

Rules learned

• The Cluster object is identical across paths — All Apps wraps it, it doesn’t change it.
• What All Apps adds: catalog RBAC, class-based quota, library attach, VPC pinning, deployment history, and a place in the Ops object model.
• Four traps on 1.35: no default library, classNamespace, tiny default quota, PodSecurity restricted.
• Set serviceDomain and a non-shadowing pod CIDR at create; both are immutable.

Next in All Apps in Practice: the demo apps that live on this cluster.

Lab environment; opinions my own.