logtail: always zstd compress with FastestCompression and LowMemory (#11583)

This is based on empirical testing using actual logs data.

FastestCompression only incurs a marginal <1% compression ratio hit
for a 2.25x reduction in memory use for small payloads
(which are common if log uploads happen at a decently high frequency).
The memory savings for large payloads is much lower
(less than 1.1x reduction).

LowMemory only incurs a marginal <5% hit on performance
for a 1.6-2.0x reduction in memory use for small or large payloads.

The memory gains for both settings justifies the loss of benefits,
which are arguably minimal.

tailscale/corp#18514

Signed-off-by: Joe Tsai <joetsai@digital-static.net>
pull/11584/head
Joe Tsai 8 months ago committed by GitHub
parent 7558a1d594
commit e2586bc674
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194

@ -384,11 +384,9 @@ func (l *Logger) uploading(ctx context.Context) {
switch { switch {
case l.zstdEncoder != nil: case l.zstdEncoder != nil:
zbody = l.zstdEncoder.EncodeAll(body, nil) zbody = l.zstdEncoder.EncodeAll(body, nil)
case l.lowMem: default:
zbody = zstdframe.AppendEncode(nil, body, zbody = zstdframe.AppendEncode(nil, body,
zstdframe.FastestCompression, zstdframe.LowMemory(true)) zstdframe.FastestCompression, zstdframe.LowMemory(true))
default:
zbody = zstdframe.AppendEncode(nil, body)
} }
// Only send it compressed if the bandwidth savings are sufficient. // Only send it compressed if the bandwidth savings are sufficient.

Loading…
Cancel
Save