Compress fluent-bit log batches sent to Linseed - #5224
Open
hjiawei wants to merge 2 commits into
Open
Conversation
marvin-tigera
added
docs-pr-required
release-note-required
and removed
release-note-required
docs-not-required
labels
The linseed http outputs post plain NDJSON. Setting compress makes fluent-bit zstd-encode each batch, which cuts ingest bandwidth by roughly 12x on real traffic. Both codecs are already compiled into the fluent-bit image, so this is config only. Requires the Linseed-side decompression from calico-private (EV-6973); those must ship together, since fluent-bit drops a chunk on a 4xx rather than retrying it. EV-6973 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Compression for the fluent-bit to Linseed ingestion path was hardcoded to zstd in the rendered config, with no API to turn it off. Both resources gain ingestionCompression: None | Gzip | Zstd, defaulting to Zstd. LogCollector drives the in-cluster outputs: the rendered compress option follows the field, and None omits it so out_http posts plain NDJSON. NonClusterHost drives non-cluster hosts, which read the resource directly and hot-reload the value. Hosts treat an unset field as plain, so both reconcilers materialize the default with a write-back — the CRD default only applies on write, and CRs from before the field would otherwise leave hosts uncompressed forever. The field is named for the ingestion path and "in transit" deliberately: felix already zstd-compresses rotated log files on disk, and the additional stores (S3, Syslog, Splunk) and OpenTelemetry exporters are unaffected. EV-6973. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
hjiawei
force-pushed
the
linseed-ingest-compression
branch
from
August 20, 2026 17:28
1cb24e7 to
da81c9f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Config-only change: the fluent-bit outputs that ship logs to Linseed now set
compress: zstd, so each batch is zstd-encoded on the wire. Measured on a kubeadm cluster, this cuts log ingestion bandwidth by roughly 12x (about 92%).Both zstd and gzip are already compiled into the fluent-bit image, so nothing about the build changes. zstd is the default because it beat gzip on both axes on real Calico log data — 43x vs 28x compression on 2MB flow-log samples, and 0.03s vs 0.31s of CPU per 100MB.
compressis added only inlinseedHTTPOutput, so it applies to the Linseedhttpoutputs and nothing else. The additional stores (S3, Splunk, Syslog, OpenTelemetry) are untouched — note that many rendered-config golden files are named after a scenario (s3.yaml,splunk-http.yaml) but also contain the standard Linseed outputs, which is where the added lines appear. S3's pre-existingcompression: gzipisout_s3's own option for gzipping archived objects and is unrelated.Affects the log-collector rendering only.
Requires the Linseed-side change
Depends on tigera/calico-private#13367, which teaches Linseed to inflate gzip and zstd request bodies. That must ship first or in the same release: fluent-bit's
out_httpdrops a chunk on a 4xx rather than retrying it, so a compressing sender pointed at a Linseed without decompression would lose logs. Since the operator renders both the fluent-bit config and the Linseed deployment from the same release, they move together.CPU cost
Compression costs Linseed roughly 5-20% more CPU per request (less for bigger batches), measured on both a dev laptop and a cluster node (
e2-standard-2). Decompression runs at 488 MB/s of logs per node vCPU against about 30 MB/s for the NDJSON parsing that follows it, so it is ~15x cheaper per byte than work Linseed already does. Sender-side cost is distributed across nodes at 254us per 10KB batch, under 1% of a vCPU. Full numbers in tigera/calico-private#13367 and EV-6973.Release Note
For PR author
make gen-files— n/a, no API changemake gen-versions— n/aRendered-config golden files regenerated with
UPDATE_RENDERED_CONFIGS=1; every changed line is the addedcompresskey. A behavioural assertion was added tofluentbit_test.goso the intent is stated in a test rather than only in golden output.pkg/render/logcollectorandpkg/controller/logcollectorsuites pass.Validated end to end on a real cluster: zstd and gzip each exercised against the patched Linseed, 254 bulk batches with
"failed":0on every one, documents landing in Elasticsearch throughout.