<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/">
  <channel>
    <title>Networking on The Nested Lab</title>
    <link>https://thenestedlab.com/tags/networking/</link>
    <description>Recent content in Networking on The Nested Lab</description>
    <generator>Hugo</generator>
    <language>en-gb</language>
    <lastBuildDate>Wed, 16 Sep 2026 07:30:00 +0100</lastBuildDate>
    <atom:link href="https://thenestedlab.com/tags/networking/index.xml" rel="self" type="application/rss+xml" />
    <item>
      <title>Dual-NIC nested hosts: what redundancy means when the fabric is virtual</title>
      <link>https://thenestedlab.com/posts/dual-nic-nested-hosts/</link>
      <pubDate>Wed, 16 Sep 2026 07:30:00 +0100</pubDate>
      <guid>https://thenestedlab.com/posts/dual-nic-nested-hosts/</guid>
      <description>VCF wants two pNICs per host. In a nested lab the second vNIC adds no physical redundancy — so why add it? Because bringup validation and uplink teaming expect it, and because the failover test tells you something real about the trunk. vmnic0 down, 0% loss, and the SSH session watching it never dropped.</description>
      <content:encoded><![CDATA[<p>&ldquo;Naturally, a VCF host has at least two NICs. Are we testing that, or have
you virtualised it away?&rdquo;</p>
<p>Fair question, and the honest answer has two halves. In a nested lab the
<em>physical</em> redundancy is provided by the outer host — its vDS, its NSX
uplinks — and a second vNIC on the nested VM adds precisely none. But VCF
doesn&rsquo;t know it&rsquo;s nested. Bringup&rsquo;s host validation and the vDS uplink
teaming it configures <strong>expect two vmnics</strong>, and a host with one gets
flagged. So the nested hosts get two vNICs, both on the trunk subnet, and
the question becomes: does failover between them actually work inside a
VPC?</p>
<h2 id="the-setup">The setup</h2>
<p>Both vNICs attach to the same <code>sn-trunk</code> subnet — <a href="/posts/nested-esxi-nsx-vpc/">the trunk from part
1</a> — and ESXi sees them as two 10G vmnics:</p>
<p><img alt="Host Client: vmnic0 and vmnic1, both 10 Gbit/s on vSwitch0" loading="lazy" src="/images/ui/u13-hostclient-dual-nics.jpg"></p>
<p>vSwitch0 teams them active/active with the default originating-port-ID
policy; every portgroup (Management 1610, vMotion 1611, vSAN 1612) inherits
it. Nothing you wouldn&rsquo;t do on metal.</p>
<h2 id="the-test-pull-a-nic-while-watching-from-inside">The test: pull a NIC while watching from inside</h2>
<p>The interesting bit isn&rsquo;t whether pings continue — it&rsquo;s <em>which</em> session
I&rsquo;m watching from. I&rsquo;m SSH&rsquo;d to <code>esx01</code> <strong>through its public VIP</strong>, which
means my session traverses the NSX LB → the VPC → the trunk port → whichever
vmnic happens to carry vmk0. If failover breaks anything, it breaks the
terminal I&rsquo;m typing in.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-fallback" data-lang="fallback"><span class="line"><span class="cl">[root@esx01-a:~] esxcli network nic list
</span></span><span class="line"><span class="cl">Name    ...  Admin Status  Link Status  Speed  MAC Address
</span></span><span class="line"><span class="cl">vmnic0  ...  Up            Up           10000  04:50:56:00:5c:03
</span></span><span class="line"><span class="cl">vmnic1  ...  Up            Up           10000  04:50:56:00:68:00
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">[root@esx01-a:~] esxcli network nic down -n vmnic0    # FAIL THE FIRST NIC
</span></span><span class="line"><span class="cl">vmnic0  ...  Down          Down             0  04:50:56:00:5c:03
</span></span><span class="line"><span class="cl">vmnic1  ...  Up            Up           10000  04:50:56:00:68:00
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">[root@esx01-a:~] vmkping -I vmk1 172.30.0.71          # vMotion VLAN, now over vmnic1
</span></span><span class="line"><span class="cl">3 packets transmitted, 3 packets received, 0% packet loss
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">[root@esx01-a:~] vmkping -I vmk2 172.30.0.101         # vSAN VLAN, now over vmnic1
</span></span><span class="line"><span class="cl">3 packets transmitted, 3 packets received, 0% packet loss
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">[root@esx01-a:~] esxcli network nic up -n vmnic0      # restore
</span></span><span class="line"><span class="cl"># session never dropped.
</span></span></code></pre></div><p><img alt="Before / after: vmnic0 down, every VLAN still passing" loading="lazy" src="/images/c6-nic-failover-beforeafter.gif"></p>
<p><img alt="Full failover transcript" loading="lazy" src="/images/demo-c6-nic-failover.jpg"></p>
<p>Every VLAN moved to vmnic1. Zero loss on vMotion and vSAN. And the
management session — the one <em>most</em> likely to notice — never blinked.</p>
<h2 id="why-this-is-a-real-result-not-a-party-trick">Why this is a real result, not a party trick</h2>
<p>Think about what just happened at the NSX layer. vmk0&rsquo;s MAC — a MAC ESXi
synthesised, not the vNIC&rsquo;s — was being learned on trunk port A. When
vmnic0 went down, the same MAC appeared on trunk port B mid-flow, with an
established TCP session riding on it.</p>
<p>On a <strong>standard</strong> VPC subnet port that is exactly the scenario SpoofGuard
exists to stop: the port&rsquo;s address bindings pin one MAC, and a frame from a
different MAC — or the <em>same</em> MAC arriving on a different port — is dropped.
Part 1 showed that killing the host on a standard subnet before it ever
spoke. This test shows the trunk subnet tolerating the live migration of a
foreign MAC between two of its ports, which is the property nested vSphere
(and anything else with a vSwitch inside a VM) fundamentally needs.</p>
<p>So the second vNIC buys three things, none of them physical redundancy:</p>
<ol>
<li><strong>Bringup and vLCM stop complaining</strong> about a single-uplink host.</li>
<li><strong>The teaming policy you&rsquo;ll configure in production gets exercised</strong> —
uplink failover, active/standby for vSAN, whatever you&rsquo;re rehearsing.</li>
<li><strong>A live proof that the trunk carries MAC mobility</strong>, which is the
real assurance that the design isn&rsquo;t relying on a quiet network.</li>
</ol>
<h2 id="what-it-does-not-buy-and-how-to-say-so">What it does <em>not</em> buy, and how to say so</h2>
<p>If someone asks &ldquo;is this host redundant?&rdquo;, the answer is &ldquo;the nested host
believes it is; actual redundancy lives one layer down.&rdquo; In a training pod
that&rsquo;s the correct and useful answer — students configure and test failover
exactly as they would on metal, and the outer platform does the real work.
In a reproduction lab for a customer NIC-teaming issue, it&rsquo;s usually enough
too: most teaming bugs are in ESXi&rsquo;s policy handling, not in the copper.</p>
<p>Where it&rsquo;s genuinely insufficient: anything about physical link behaviour —
LACP negotiation, LLDP, flapping, MTU mismatch on one uplink. The virtual
fabric never fails asymmetrically, so it can&rsquo;t reproduce those.</p>
<h2 id="why-this-matters-outside-the-lab">Why this matters outside the lab</h2>
<p>The practical value here is knowing <em>what a nested environment can and
can&rsquo;t prove</em> — which is what lets you decide when a virtual lab is enough
and when it isn&rsquo;t. For training, upgrade rehearsals, configuration and
policy testing and the vast majority of &ldquo;how does it behave when…&rdquo;
questions, nested is enough and dramatically cheaper. For physical link
behaviour — LACP, optics, asymmetric faults — you still want metal. Being
able to make that call confidently is worth more than the test itself.</p>
<h2 id="rules-learned">Rules learned</h2>
<ul>
<li>Give nested VCF hosts <strong>two vNICs on the same trunk subnet</strong>. Bringup,
vLCM and vDS teaming expect ≥ 2 vmnics; humouring them costs nothing.</li>
<li>Test failover <strong>from a session that depends on it</strong> (SSH via the VIP).
Pings passing while your terminal dies is not success.</li>
<li>The trunk subnet tolerates a <strong>vmk MAC moving between ports mid-flow</strong>
— that&rsquo;s the property standard subnets lack and nested vSphere needs.</li>
<li>Be precise in the write-up: nested dual-NIC gives <em>policy</em> realism, not
<em>physical</em> redundancy. Physical link faults can&rsquo;t be reproduced here.</li>
<li>Rebuilds re-run the vmk config: the appliance creates vmk0 only;
vmk1/vmk2, their VLANs and override gateways are applied post-boot.</li>
</ul>
<p><em>Companion to <a href="/posts/nested-esxi-nsx-vpc/">nested ESXi inside an NSX VPC</a>.</em></p>
<hr>
<p><em>Lab environment; opinions my own. Output captured live, trimmed for length,
never edited for outcome.</em></p>
]]></content:encoded>
    </item>
  </channel>
</rss>
