You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Joe Tsai d4bfe34ba7 util/zstdframe: add package for stateless zstd compression (#11481) ... The Go zstd package is not friendly for stateless zstd compression. Passing around multiple zstd.Encoder just for stateless compression is a waste of memory since the memory is never freed and seldom used if no compression operations are happening. For performance, we pool the relevant Encoder/Decoder with the specific options set. Functionally, this package is a wrapper over the Go zstd package with a more ergonomic API for stateless operations. This package can be used to cleanup various pre-existing zstd.Encoder pools or one-off handlers spread throughout our codebases. Performance: BenchmarkEncode/Best 1690 610926 ns/op 25.78 MB/s 1 B/op 0 allocs/op zstd_test.go:137: memory: 50.336 MiB zstd_test.go:138: ratio: 3.269x BenchmarkEncode/Better 10000 100939 ns/op 156.04 MB/s 0 B/op 0 allocs/op zstd_test.go:137: memory: 20.399 MiB zstd_test.go:138: ratio: 3.131x BenchmarkEncode/Default 15775 74976 ns/op 210.08 MB/s 105 B/op 0 allocs/op zstd_test.go:137: memory: 1.586 MiB zstd_test.go:138: ratio: 3.064x BenchmarkEncode/Fastest 23222 53977 ns/op 291.81 MB/s 26 B/op 0 allocs/op zstd_test.go:137: memory: 599.458 KiB zstd_test.go:138: ratio: 2.898x BenchmarkEncode/FastestLowMemory 23361 50789 ns/op 310.13 MB/s 15 B/op 0 allocs/op zstd_test.go:137: memory: 334.458 KiB zstd_test.go:138: ratio: 2.898x BenchmarkEncode/FastestNoChecksum 23086 50253 ns/op 313.44 MB/s 26 B/op 0 allocs/op zstd_test.go:137: memory: 599.458 KiB zstd_test.go:138: ratio: 2.900x BenchmarkDecode/Checksum 70794 17082 ns/op 300.96 MB/s 4 B/op 0 allocs/op zstd_test.go:163: memory: 316.438 KiB BenchmarkDecode/NoChecksum 74935 15990 ns/op 321.51 MB/s 4 B/op 0 allocs/op zstd_test.go:163: memory: 316.438 KiB BenchmarkDecode/LowMemory 71043 16739 ns/op 307.13 MB/s 0 B/op 0 allocs/op zstd_test.go:163: memory: 79.347 KiB We can see that the options are taking effect where compression ratio improves with higher levels and compression speed diminishes. We can also see that LowMemory takes effect where the pooled coder object references less memory than other cases. We can see that the pooling is taking effect as there are 0 amortized allocations. Additional performance: BenchmarkEncodeParallel/zstd-24 1857 619264 ns/op 1796 B/op 49 allocs/op BenchmarkEncodeParallel/zstdframe-24 1954 532023 ns/op 4293 B/op 49 allocs/op BenchmarkDecodeParallel/zstd-24 5288 197281 ns/op 2516 B/op 49 allocs/op BenchmarkDecodeParallel/zstdframe-24 6441 196254 ns/op 2513 B/op 49 allocs/op In concurrent usage, handling the pooling in this package has a marginal benefit over the zstd package, which relies on a Go channel as the pooling mechanism. In particular, coders can be freed by the GC when not in use. Coders can be shared throughout the program if they use this package instead of multiple independent pools doing the same thing. The allocations are unrelated to pooling as they're caused by the spawning of goroutines. Updates #cleanup Updates tailscale/corp#18514 Updates tailscale/corp#17653 Updates tailscale/corp#18005 Signed-off-by: Joe Tsai <joetsai@digital-static.net>		3 months ago
..
cache	util/cache: fix missing interface methods (#11275)	4 months ago
cibuild	all: update copyright and license headers	1 year ago
clientmetric	util/clientmetric: allow client metric values to be provided by a function	1 year ago
cloudenv	util/cloudenv: add support for DigitalOcean	5 months ago
cmpver	util/cmpver: add Less/LessEq helper funcs	4 months ago
codegen	util/codegen: support embedded fields	1 year ago
cstruct	all: update copyright and license headers	1 year ago
ctxkey	all: use reflect.TypeFor now available in Go 1.22 (#11078)	5 months ago
deephash	all: use reflect.TypeFor now available in Go 1.22 (#11078)	5 months ago
dirwalk	all: use tstest.Replace more	1 year ago
dnsname	cmd/tailscale: extend hostname validation (#7678)	1 year ago
execqueue	control/controlclient,util/execqueue: extract execqueue into a package	5 months ago
expvarx	util/expvarx: add a time and concurrency limiting expvar.Func wrapper	5 months ago
goroutines	util/goroutines: let ScrubbedGoroutineDump get only current stack	1 year ago
groupmember	util/groupmember: fail earlier if group doesn't exist, use slices.Contains	8 months ago
hashx	all: update copyright and license headers	1 year ago
httphdr	util/httphdr: add new package for parsing HTTP headers (#9797)	9 months ago
httpm	util/httpm: don't run test if .git doesn't exist	7 months ago
jsonutil	all: update copyright and license headers	1 year ago
limiter	util/lru, util/limiter: add debug helper to dump state as HTML	10 months ago
lineread	all: update copyright and license headers	1 year ago
linuxfw	util/linuxfw: fix support for containers without IPv6 iptables filters (#11381)	4 months ago
lru	util/lru: update c.head when deleting the most recently used entry	9 months ago
mak	various: add golangci-lint, fix issues (#7905)	1 year ago
multierr	all: use Go 1.21 slices, maps instead of x/exp/{slices,maps}	10 months ago
must	all: update copyright and license headers	1 year ago
nocasemaps	util/nocasemaps: add AppendSliceElem method to nocasemaps (#10871)	5 months ago
osdiag	util/osdiag, util/winutil: expose Windows policy key	7 months ago
osshare	all: update copyright and license headers	1 year ago
osuser	ssh/tailssh,util: extract new osuser package from ssh code (#10170)	8 months ago
pidowner	all: update copyright and license headers	1 year ago
precompress	all: update copyright and license headers	1 year ago
quarantine	all: update copyright and license headers	1 year ago
race	util/race: add test to confirm we don't leak goroutines	9 months ago
racebuild	all: update copyright and license headers	1 year ago
rands	util/rands: add Shuffle and Perm functions with on-stack RNG state	5 months ago
reload	util/reload: add new package to handle periodic value loading	3 months ago
ringbuffer	wgengine/magicsock: don't use endpoint debug ringbuffer on mobile.	3 months ago
set	util/set: implement json.Marshaler/Unmarshaler (#10308)	7 months ago
singleflight	all: update copyright and license headers	1 year ago
slicesx	util/slicesx: add Filter function	5 months ago
syspolicy	util/syspolicy: add 'ResetToDefaults' (#11194)	4 months ago
sysresources	util/sysresources, magicsock: scale DERP buffer based on system memory	1 year ago
systemd	all: update copyright and license headers	1 year ago
testenv	util/testenv: add new package to hold InTest	11 months ago
topk	util/topk: add package containing a probabilistic top-K tracker	4 months ago
truncate	util: add truncate package (#7490)	1 year ago
uniq	all: update copyright and license headers	1 year ago
vizerror	util/vizerror: add As function to get wrapped Error	1 year ago
winutil	util/winutil: add some missing docs to restartmgr errors	6 months ago
zstdframe	util/zstdframe: add package for stateless zstd compression (#11481)	3 months ago