<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/">
  <channel>
    <title>Oci on The Nested Lab</title>
    <link>https://thenestedlab.com/tags/oci/</link>
    <description>Recent content in Oci on The Nested Lab</description>
    <generator>Hugo</generator>
    <language>en-gb</language>
    <lastBuildDate>Wed, 14 Oct 2026 00:00:00 +0100</lastBuildDate>
    <atom:link href="https://thenestedlab.com/tags/oci/index.xml" rel="self" type="application/rss+xml" />
    <item>
      <title>imgpkg tar vs OCI archive: why your air-gap tarballs won&#39;t load</title>
      <link>https://thenestedlab.com/posts/imgpkg-tar-vs-oci-archive/</link>
      <pubDate>Wed, 14 Oct 2026 00:00:00 +0100</pubDate>
      <guid>https://thenestedlab.com/posts/imgpkg-tar-vs-oci-archive/</guid>
      <description>Two tarballs full of container images, both bound for the same Harbor. imgpkg loads one and rejects the other. The layout difference nobody explains, what survives a copy, and three ways across.</description>
      <content:encoded><![CDATA[<p>You&rsquo;ve staged 40 container images for a dark site. The VKS bundle loads
into Harbor first time with <code>imgpkg copy --tar</code>. The GPU Operator images,
same tool and same command, fail. Both are tarballs. Both contain the
images. What&rsquo;s different?</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-fallback" data-lang="fallback"><span class="line"><span class="cl">$ tar -tf images/nvidia_gpu-operator__v26.3.3.tar | grep -v ^blobs/
</span></span><span class="line"><span class="cl">oci-layout
</span></span><span class="line"><span class="cl">index.json
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">$ tar -tf supervisor/vks-3.7.1.tar | head -2
</span></span><span class="line"><span class="cl">manifest.json
</span></span><span class="line"><span class="cl">sha256-05e0b744....tar.gz
</span></span></code></pre></div><p>That&rsquo;s the whole answer. It took me an afternoon to find, which is a
generous amount of afternoon for a couple of filenames.</p>
<h2 id="two-layouts-that-look-the-same-from-the-outside">Two layouts that look the same from the outside</h2>
<table>
	<thead>
			<tr>
					<th>Archive</th>
					<th>Layout</th>
					<th><code>imgpkg copy --tar</code>?</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>Written by a registry-API puller / <code>crane pull --format=oci</code> / skopeo</td>
					<td><strong>OCI image layout</strong>: <code>oci-layout</code>, <code>index.json</code>, <code>blobs/sha256/*</code></td>
					<td><strong>No</strong></td>
			</tr>
			<tr>
					<td>Written by <code>imgpkg copy --to-tar</code></td>
					<td><strong>imgpkg tar</strong>: <code>manifest.json</code>, <code>sha256-*.tar.gz</code></td>
					<td>Yes</td>
			</tr>
	</tbody>
</table>
<p>imgpkg reads <em>only</em> the tar format imgpkg itself writes. There&rsquo;s no
converter: <code>imgpkg push -f &lt;dir&gt;</code> would wrap the OCI directory as a new
image <em>layer</em>, not replicate the original image. If your staging pipeline
pulled with anything other than imgpkg, imgpkg won&rsquo;t load it. Full stop.</p>
<p><img alt="Two tarballs: an imgpkg tar (manifest.json, sha256-*.tar.gz) loads with imgpkg copy &ndash;tar; an OCI image layout (oci-layout, index.json, blobs) does not, and loads with crane push or skopeo instead. imgpkg&rsquo;s copy keeps tags, digests and platforms, and drops cosign signatures unless both ends ask for them" loading="lazy" src="/images/diagrams/imgpkg-which-tool.svg">
<em>Same tool, same command, different archive. <code>tar -tf</code> tells you which one you&rsquo;ve got.</em></p>
<h2 id="three-ways-out">Three ways out</h2>
<p><strong>1. Re-pull with imgpkg</strong> (chosen, when you still have internet somewhere):</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-fallback" data-lang="fallback"><span class="line"><span class="cl">imgpkg copy -i nvcr.io/nvidia/gpu-operator:v26.3.3 --to-tar gpu-operator.tar
</span></span><span class="line"><span class="cl"># at the dark site:
</span></span><span class="line"><span class="cl">imgpkg copy --tar gpu-operator.tar --to-repo harbor02/nvidia/gpu-operator
</span></span></code></pre></div><p>The cost: a second download, and it&rsquo;s <strong>bigger</strong>. <code>imgpkg copy</code> has no
platform filter, so you get every architecture in the manifest list
(amd64 and arm64, sometimes s390x and ppc64le). For the GPU Operator set,
1.1 GiB became 2.3 GiB. The upside: your arm64 node pool, if you ever have
one, won&rsquo;t send you back to the internet.</p>
<p><strong>2. crane, no re-pull.</strong> <code>crane</code> is a static Go binary (no install, no
root), and it reads OCI layouts natively:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-fallback" data-lang="fallback"><span class="line"><span class="cl">crane push images/nvidia_gpu-operator__v26.3.3.tar harbor02/nvidia/gpu-operator:v26.3.3
</span></span></code></pre></div><p>It&rsquo;s the cheapest in bytes. It does mean getting one more binary through
the airlock.</p>
<p><strong>3. skopeo</strong>, if it happens to be there. <code>skopeo copy oci-archive:... docker://...</code>.
Usually it isn&rsquo;t.</p>
<p>(There&rsquo;s a fourth: stand up a local <code>registry:2</code>, crane-push into it, then
<code>imgpkg copy -i localhost:5000/... --to-repo harbor02/...</code>. It&rsquo;s strictly
more work than 1 or 2. Don&rsquo;t.)</p>
<h2 id="what-survives-the-copy-verified-not-assumed">What survives the copy (verified, not assumed)</h2>
<p>I tested this against a throwaway <code>crane registry serve</code> with a real
multi-arch image, because &ldquo;the digest should be preserved&rdquo; is the kind of
sentence that precedes a bad week.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-fallback" data-lang="fallback"><span class="line"><span class="cl">imgpkg copy -i nvcr.io/nvidia/cloud-native/k8s-mig-manager:v0.13.1 --to-tar probe.tar
</span></span><span class="line"><span class="cl">imgpkg copy --tar probe.tar --to-repo 127.0.0.1:5099/real/k8s-mig-manager
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">imgpkg tag list -i 127.0.0.1:5099/real/k8s-mig-manager --digests
</span></span><span class="line"><span class="cl">  v0.13.1   sha256:8e0803d2...           # manifest-list digest
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">crane digest nvcr.io/nvidia/cloud-native/k8s-mig-manager:v0.13.1
</span></span><span class="line"><span class="cl">  sha256:8e0803d2...                     # identical
</span></span><span class="line"><span class="cl">crane digest --platform linux/amd64 127.0.0.1:5099/real/k8s-mig-manager:v0.13.1
</span></span><span class="line"><span class="cl">  sha256:69250353...                     # identical to the OCI-archive .digest sidecar
</span></span></code></pre></div><p>That settles three things:</p>
<ol>
<li><strong>The human tag survives.</strong> The destination gets <code>:v0.13.1</code> <em>and</em>
imgpkg&rsquo;s <code>:sha256-&lt;hex&gt;.imgpkg</code>. This matters, because Helm values
reference images by tag. (Only <code>--repo-based-tags</code> changes the tag
shape; don&rsquo;t pass it.)</li>
<li><strong>Digests survive</strong> at both the manifest-list and per-platform level.</li>
<li><strong>The amd64 bits are identical</strong> to the OCI archive. A re-pull is a
re-packaging, not different content.</li>
</ol>
<p>One trap: pinning by digest (<code>imgpkg copy -i repo@sha256:...</code>) <em>would</em>
keep it single-arch and small. But then there&rsquo;s no tag to carry across.
The destination only gets <code>:sha256-&lt;hex&gt;.imgpkg</code>, and your Helm values
break. Don&rsquo;t do that either.</p>
<h2 id="the-one-that-bites-later-signatures">The one that bites later: signatures</h2>
<p>imgpkg <strong>drops cosign signatures by default</strong> at every copy. That matters
when the thing you&rsquo;re loading is a platform bundle whose consumer checks
signatures. VKS supervisor services on VCF 9.0.1+ do, for versions above
3.4.0 from a private registry.</p>
<p>Every content check passes and the activation proceeds, so for a while
everyone is happy. Then every new control-plane pod is denied by a
webhook, for running &ldquo;in an untrusted namespace&rdquo;.</p>
<p>Pull and push both need <code>--cosign-signatures</code>. The <code>.sig</code> artifacts are
separate tags (<code>sha256-&lt;digest&gt;.sig</code>), about 100 KB for a whole bundle. So
if the content is already in the registry, you can ship just the
signatures.</p>
<p>That one gets <a href="/series/dark-site-notes/">its own post</a>.</p>
<h2 id="why-this-matters-outside-the-lab">Why this matters outside the lab</h2>
<p>Air-gapped customers pay for every failed transfer twice: once in the
change window that slipped, and once in the import/export cycle to try
again.</p>
<p>Staging discipline is the whole game. Know which tool wrote each archive,
and what survives a copy. Prove the load against a throwaway registry
<em>before</em> anything crosses the airlock. That rehearsal is a fixed part of
how we prepare dark-site deliveries, and it&rsquo;s why the loading step is the
boring part on the day. Boring is underrated at a dark site.</p>
<h2 id="rules-learned">Rules learned</h2>
<ul>
<li>Run <code>tar -tf x.tar | head</code> before anything else. <code>manifest.json</code> means
imgpkg; <code>oci-layout</code> means an OCI archive. They are not interchangeable.</li>
<li>imgpkg loads <strong>only</strong> imgpkg tars. No converter exists.</li>
<li>OCI archives load with <code>crane push</code> (static binary) or skopeo.</li>
<li><code>imgpkg copy</code> copies <strong>all platforms</strong> (bigger) and preserves tags and
digests (verified). Pin by tag, never by digest, when Helm values
reference tags.</li>
<li>Add <code>--cosign-signatures</code> on both ends when the consumer verifies
bundles.</li>
<li>Stage <code>imgpkg</code> <em>and</em> <code>crane</code> binaries with the tarballs. One of them
will be the wrong tool for something.</li>
</ul>
<h2 id="broadcom-documentation">Broadcom documentation</h2>
<ul>
<li><a href="https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-service-administration-and-development/9-0/managing-vsphere-kuberenetes-service-clusters-and-workloads/using-private-registries-with-tkg-service-clusters/push-standard-packages-to-a-private-harbor-registry.html">Push Standard Packages to a Private Harbor Registry</a>: <code>imgpkg copy --to-tar</code> on the connected side, <code>--tar ... --to-repo</code> into Harbor</li>
<li><a href="https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-service-administration-and-development/9-1/vcf-services-development-guide/supervisor-services-carvel-private-registry.html">Supporting Private Registries</a>: relocating Carvel bundles, and why omitting <code>--cosign-signatures</code> leaves a bundle untrusted</li>
<li><a href="https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-service-administration-and-development/9-0/managing-vsphere-kubernetes-service/installing-and-upgrading-the-tkg-service/upgrade-tkg-service-from-a-private-registry.html">Upgrade VKS from a Private Registry</a>: the VKS bundle copied with <code>--cosign-signatures</code> on both copies</li>
<li><a href="https://knowledge.broadcom.com/external/article/431379/vsphere-kubernetes-service-upgrade-faile.html">vSphere kubernetes service upgrade failed with &ldquo;kapp: error waiting on reconcile packageinstall&rdquo;</a>: the untrusted-namespace denial when the signatures were dropped</li>
<li><a href="https://knowledge.broadcom.com/external/article/388470">Script to mirror resources for VPAIF-N AI workloads in air-gapped environment</a>: Broadcom&rsquo;s Docker-based script for mirroring Private AI container images, Helm charts and models into Harbor</li>
</ul>
<hr>
<p><em>Lab environment; opinions my own. Verified against a local registry with
real multi-arch images; digests shown are truncated.</em></p>
]]></content:encoded>
    </item>
  </channel>
</rss>
