<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/">
  <channel>
    <title>Vcf-Operations-for-Logs on The Nested Lab</title>
    <link>https://thenestedlab.com/tags/vcf-operations-for-logs/</link>
    <description>Recent content in Vcf-Operations-for-Logs on The Nested Lab</description>
    <generator>Hugo</generator>
    <language>en-gb</language>
    <lastBuildDate>Wed, 16 Sep 2026 12:00:00 +0100</lastBuildDate>
    <atom:link href="https://thenestedlab.com/tags/vcf-operations-for-logs/index.xml" rel="self" type="application/rss+xml" />
    <item>
      <title>fluent-bit two ways: VKS add-on and Windows agent, one log endpoint</title>
      <link>https://thenestedlab.com/posts/fluent-bit-two-ways/</link>
      <pubDate>Wed, 16 Sep 2026 12:00:00 +0100</pubDate>
      <guid>https://thenestedlab.com/posts/fluent-bit-two-ways/</guid>
      <description>The same log pipeline for two very different worlds: fluent-bit as a VKS package (values secret, CFAPI output, verified 200s per batch) and fluent-bit as a Windows service on Server 2025 shipping the event log to the same VCF Operations for Logs endpoint. One backend, two configs, and what each side taught me.</description>
      <content:encoded><![CDATA[<p>Logs from a Kubernetes cluster and logs from a Windows server end up in
the same place — VCF Operations for Logs — but the paths there couldn&rsquo;t
look more different. On VKS, fluent-bit is a <em>package</em>: declare a values
secret, reconcile, done. On Windows it&rsquo;s a <em>service</em>: install, write a
config by hand, restart, tail the debug log. Same binary, same output
plugin, same endpoint. This post is both, side by side, so the shared
shape is obvious.</p>
<h2 id="the-endpoint">The endpoint</h2>
<p>VCF Operations for Logs ingests over its <strong>CFAPI</strong> on port 9543:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-fallback" data-lang="fallback"><span class="line"><span class="cl">https://&lt;ops-logs&gt;:9543/api/v2/events
</span></span></code></pre></div><p>Auth is Ops-side (the ingestion endpoint accepts from configured sources);
what matters for fluent-bit is the URI, the port, TLS, and a JSON body
shaped as <code>{&quot;events&quot;:[...]}</code>. Both shippers below produce exactly that.</p>
<h2 id="way-1-vks-package">Way 1: VKS package</h2>
<p>VKS ships fluent-bit in its standard package repository. On a cluster with
the repo registered:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-fallback" data-lang="fallback"><span class="line"><span class="cl">kubectl get packages -A | grep fluent-bit
</span></span><span class="line"><span class="cl">  fluent-bit.fluent-bit.tanzu.vmware.com   4.0.5+vmware.1-vks.1
</span></span></code></pre></div><p>A values secret configures the output — the package&rsquo;s own default output
shape targets CFAPI, so the values are short:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">fluent_bit</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">config</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">outputs</span><span class="p">:</span><span class="w"> </span><span class="p">|</span><span class="sd">
</span></span></span><span class="line"><span class="cl"><span class="sd">      [OUTPUT]
</span></span></span><span class="line"><span class="cl"><span class="sd">        Name    http
</span></span></span><span class="line"><span class="cl"><span class="sd">        Match   *
</span></span></span><span class="line"><span class="cl"><span class="sd">        Host    f06-flt-log01.res.lab
</span></span></span><span class="line"><span class="cl"><span class="sd">        Port    9543
</span></span></span><span class="line"><span class="cl"><span class="sd">        URI     /api/v2/events
</span></span></span><span class="line"><span class="cl"><span class="sd">        Format  json
</span></span></span><span class="line"><span class="cl"><span class="sd">        tls     On
</span></span></span><span class="line"><span class="cl"><span class="sd">        tls.verify Off
</span></span></span><span class="line"><span class="cl"><span class="sd">        json_date_key    timestamp
</span></span></span><span class="line"><span class="cl"><span class="sd">        json_date_format iso8601</span><span class="w">
</span></span></span></code></pre></div><p>Then a <code>PackageInstall</code> referencing it:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="l">packaging.carvel.dev/v1alpha1</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l">PackageInstall</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">metadata</span><span class="p">:</span><span class="w"> </span>{<span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="nt">fluent-bit, namespace</span><span class="p">:</span><span class="w"> </span><span class="l">tanzu-packages}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">spec</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">serviceAccountName</span><span class="p">:</span><span class="w"> </span><span class="l">tanzu-packages-sa</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">packageRef</span><span class="p">:</span><span class="w"> </span>{<span class="nt">refName</span><span class="p">:</span><span class="w"> </span><span class="nt">fluent-bit.fluent-bit.tanzu.vmware.com, versionSelection</span><span class="p">:</span><span class="w"> </span>{<span class="nt">constraints</span><span class="p">:</span><span class="w"> </span><span class="m">4.0.5</span><span class="l">+vmware.1-vks.1}}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">values</span><span class="p">:</span><span class="w"> </span><span class="p">[</span>{<span class="nt">secretRef</span><span class="p">:</span><span class="w"> </span>{<span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">fluent-bit-values}}]</span><span class="w">
</span></span></span></code></pre></div><p><code>Reconcile succeeded</code>, and the proof is in the pod logs, not the Ops UI —
Ops for Logs&rsquo; API auth is identity-broker federated, so scripted
verification from the shipper side is far easier:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-fallback" data-lang="fallback"><span class="line"><span class="cl">[ info] [output:http:http.0] f06-flt-log01.res.lab:9543, HTTP status=200
</span></span><span class="line"><span class="cl">[ info] [output:http:http.0] f06-flt-log01.res.lab:9543, HTTP status=200
</span></span></code></pre></div><p>One 200 per batch. That&rsquo;s the whole VKS side, and it&rsquo;s why the packaged
path is the one to recommend: the inputs (container logs, kubelet,
systemd) are pre-wired, the parsers are right, and the only decision you
make is the output.</p>
<p><strong>One trap</strong> (shared with Telegraf, <a href="/series/observability-on-vcf/">next post</a>):
a cluster created without <code>clusterNetwork.serviceDomain</code> produces in-cluster
service names of the form <code>...svc.</code> with no domain, which some add-ons
build into unresolvable URLs. It&rsquo;s immutable after create; set it
explicitly on every new cluster.</p>
<p><img alt="VCF Operations — Logs: text contains vks-demo01, last 24 h, 1.74K events" loading="lazy" src="/images/ui/l2-ops-logs-vks-demo01-query.jpg">
<em>The receiving end. One filter — <code>text contains vks-demo01</code> — and a day of the cluster&rsquo;s logs, one bar per hour.</em></p>
<p><img alt="The stream: container logs arriving with Kubernetes metadata — app, cluster, container, kubernetes_namespace, node, pod" loading="lazy" src="/images/ui/l1-ops-logs-vks-demo01-24h.jpg">
<em>Every event carries the fields the package&rsquo;s filters add — <code>cluster</code>, <code>kubernetes_namespace</code>, <code>pod</code>, <code>container</code> — which is what makes the Windows-vs-VKS comparison in one explorer possible later.</em></p>
<h2 id="way-2-windows-server-2025-service">Way 2: Windows Server 2025 service</h2>
<p>fluent-bit ships a Windows build with a <code>winevtlog</code> input. On the
<a href="/series/the-windows-build-pipeline/">pipeline-built W2025 server</a> I used the
portable zip rather than the MSI — no installer, a folder under <code>C:\</code>, a
service registered with <code>sc.exe</code>. The config is short and every line of it
turned out to matter:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-ini" data-lang="ini"><span class="line"><span class="cl"><span class="k">[INPUT]</span>
</span></span><span class="line"><span class="cl">    <span class="na">Name          winevtlog</span>
</span></span><span class="line"><span class="cl">    <span class="na">Channels      System,Application,Security</span>
</span></span><span class="line"><span class="cl">    <span class="na">DB            C:\fluent-bit\winevt.db        # bookmark: no replay after restart</span>
</span></span><span class="line"><span class="cl">    <span class="na">String_Inserts On</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">[FILTER]</span>
</span></span><span class="line"><span class="cl">    <span class="na">Name    modify</span>
</span></span><span class="line"><span class="cl">    <span class="na">Match   *</span>
</span></span><span class="line"><span class="cl">    <span class="na">Rename  Message  text                     # &lt;- the field Ops actually indexes</span>
</span></span><span class="line"><span class="cl">    <span class="na">Add     hostname TEST-W2025</span>
</span></span><span class="line"><span class="cl">    <span class="na">Add     appname  v-windows</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">[OUTPUT]</span>
</span></span><span class="line"><span class="cl">    <span class="na">Name    http</span>
</span></span><span class="line"><span class="cl">    <span class="na">Host    f06-flt-log01.res.lab             # FQDN, never the IP</span>
</span></span><span class="line"><span class="cl">    <span class="na">Port    9543</span>
</span></span><span class="line"><span class="cl">    <span class="na">URI     /api/v2/events</span>
</span></span><span class="line"><span class="cl">    <span class="na">Format  json</span>
</span></span><span class="line"><span class="cl">    <span class="na">json_date_key    timestamp</span>
</span></span><span class="line"><span class="cl">    <span class="na">json_date_format epoch_ms                 # milliseconds, not ISO, not seconds</span>
</span></span><span class="line"><span class="cl">    <span class="na">tls     On</span>
</span></span><span class="line"><span class="cl">    <span class="na">net.dns.resolver LEGACY                   # Windows: c-ares can&#39;t resolve</span>
</span></span></code></pre></div><p>The service came up first time. Getting a single event to <em>land</em> took four
attempts, and each failure taught something the documentation doesn&rsquo;t say:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-fallback" data-lang="fallback"><span class="line"><span class="cl">[error] [output:http:http.0] no upstream connections available to f06-flt-log01.res.lab:9543
</span></span></code></pre></div><p><strong>1. fluent-bit couldn&rsquo;t resolve a name Windows could.</strong> <code>Resolve-DnsName</code>
worked; <code>Test-NetConnection … -Port 9543</code> worked; fluent-bit&rsquo;s own async
resolver didn&rsquo;t. <code>net.dns.resolver LEGACY</code> hands DNS back to the OS.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-fallback" data-lang="fallback"><span class="line"><span class="cl">[error] [output:http:http.0] 10.26.5.25:9543, HTTP status=404
</span></span></code></pre></div><p><strong>2. The appliance routes on the Host header.</strong> Sidestepping DNS by using
the IP gets a 404 for <em>every</em> path. Address it by FQDN.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-fallback" data-lang="fallback"><span class="line"><span class="cl">{&#34;errorMessage&#34;:&#34;Cannot deserialize value of type `java.lang.Long` from String \&#34;2026-09-16T10:55:00.000000Z\&#34;&#34;}
</span></span></code></pre></div><p><strong>3. Timestamps must be numeric.</strong> fluent-bit&rsquo;s <code>iso8601</code> output is a
string; the ingest API wants a Long. Worse: it reads that number as
<strong>milliseconds</strong>, so the default <code>double</code> (epoch <em>seconds</em>) is accepted with
a 200 and files your events in January 1970. <code>epoch_ms</code>.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-fallback" data-lang="fallback"><span class="line"><span class="cl">{&#34;received&#34;:0,&#34;message&#34;:&#34;events ingested&#34;,&#34;status&#34;:&#34;ok&#34;}
</span></span></code></pre></div><p><strong>4. The message field must be called <code>text</code>.</strong> Anything else — <code>Message</code>
as <code>winevtlog</code> emits it, <code>message</code>, <code>log</code> — returns 200 and
<code>received: 0</code>. Silently dropped. Hence the <code>Rename</code> filter. Broadcom&rsquo;s own
reference config for Windows carries exactly this line; I found it the hard
way first.</p>
<p>Then, finally:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-fallback" data-lang="fallback"><span class="line"><span class="cl">[ info] [output:http:http.0] f06-flt-log01.res.lab:9543, HTTP status=200
</span></span><span class="line"><span class="cl">[ info] [output:http:http.0] f06-flt-log01.res.lab:9543, HTTP status=200
</span></span></code></pre></div><p><img alt="Ops for Logs: hostname contains TEST-W2025 — Windows System, Application and Security events, with channel, computer, eventid and appname as fields" loading="lazy" src="/images/ui/w3-ops-logs-test-w2025-events.jpg">
<em>Fifty events in the first five minutes: service state changes, a &ldquo;system time was changed&rdquo; security event with its full subject block, all searchable with the same fields as everything else.</em></p>
<p><img alt="The same explorer with both hostnames in one filter: TEST-W2025 and vks-demo01" loading="lazy" src="/images/ui/w4-ops-logs-windows-and-vks-filter.jpg">
<em>One filter, two worlds. A Windows server and a Kubernetes cluster, in the same query, in the same store.</em></p>
<p>One more Windows-specific note: a bare <code>fluent-bit.exe</code> registered with
<code>sc.exe</code> isn&rsquo;t a proper service binary — it ignores the stop signal and
Windows sits at &ldquo;waiting for service to stop&rdquo; until you kill the process.
The MSI installs a real service wrapper; use it for anything that isn&rsquo;t a
lab.</p>
<h2 id="the-shape-they-share">The shape they share</h2>
<table>
	<thead>
			<tr>
					<th></th>
					<th>VKS package</th>
					<th>Windows service</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>Install</td>
					<td><code>PackageInstall</code></td>
					<td>MSI / <code>sc.exe create</code></td>
			</tr>
			<tr>
					<td>Config</td>
					<td>values Secret</td>
					<td><code>fluent-bit.conf</code></td>
			</tr>
			<tr>
					<td>Inputs</td>
					<td>pre-wired (containers, kubelet, systemd)</td>
					<td><code>winevtlog</code> channels you choose</td>
			</tr>
			<tr>
					<td>Output</td>
					<td><code>http</code> → CFAPI :9543 <code>/api/v2/events</code></td>
					<td>identical</td>
			</tr>
			<tr>
					<td>Verify</td>
					<td>pod log <code>HTTP status=200</code></td>
					<td>service log <code>HTTP status=200</code> — and check the <em>date</em> on what arrived</td>
			</tr>
			<tr>
					<td>Restart safety</td>
					<td>Kubernetes</td>
					<td><code>DB</code> bookmark file</td>
			</tr>
	</tbody>
</table>
<p>The output stanza is byte-identical. That&rsquo;s the lesson: standardise the
<em>sink</em> and let each platform own its <em>source</em>.</p>
<h2 id="why-this-matters-outside-the-lab">Why this matters outside the lab</h2>
<p>The business outcome is one place to look. Windows servers, Kubernetes
clusters and the platform itself all ship logs to the same VCF Operations
for Logs, with the same fields, searchable in one query — so an incident
that spans a Windows service and a container gets investigated in one
screen instead of three tools. Standardising the <em>destination</em> while
letting each platform keep its native shipper is also what keeps the
estate maintainable: one endpoint to secure and retain, no bespoke agent
per team.</p>
<h2 id="rules-learned">Rules learned</h2>
<ul>
<li>Ops for Logs ingestion = CFAPI <code>:9543/api/v2/events</code>, JSON. One output
stanza serves every shipper.</li>
<li>On VKS, use the <strong>package</strong>; decide only the output. Verify from the
pod log — the Ops API is awkward to script against.</li>
<li>Set <code>serviceDomain</code> on every new VKS cluster; it&rsquo;s immutable.</li>
<li>On Windows: FQDN not IP (Host-header routing), <code>net.dns.resolver LEGACY</code>,
<code>Rename Message text</code>, <code>json_date_format epoch_ms</code>. Each one fails
differently and two of them fail <em>silently</em>.</li>
<li>A <code>200</code> is not proof. <code>received: 0</code> is a drop; a seconds timestamp is a
200 filed in 1970. Look for the event in the explorer before you call it done.</li>
<li>Prove &ldquo;one endpoint&rdquo; with one explorer view showing both sources.</li>
</ul>
<p><em>Next: <a href="/series/observability-on-vcf/">Telegraf on Windows Server 2025 — unsupported, works anyway</a>.</em></p>
<hr>
<p><em>Lab environment; opinions my own.</em></p>
]]></content:encoded>
    </item>
  </channel>
</rss>
