You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
tailscale/cmd/tailscaled
Joe Tsai 84c99fe0d9
logtail: be less aggressive about re-uploads (#8117)
The retry logic was pathological in the following ways:

* If we restarted the logging service, any pending uploads
would be placed in a retry-loop where it depended on backoff.Backoff,
which was too aggresive. It would retry failures within milliseconds,
taking at least 10 retries to hit a delay of 1 second.

* In the event where a logstream was rate limited,
the aggressive retry logic would severely exacerbate the problem
since each retry would also log an error message.
It is by chance that the rate of log error spam
does not happen to exceed the rate limit itself.

We modify the retry logic in the following ways:

* We now respect the "Retry-After" header sent by the logging service.

* Lacking a "Retry-After" header, we retry after a hard-coded period of
30 to 60 seconds. This avoids the thundering-herd effect when all nodes
try reconnecting to the logging service at the same time after a restart.

* We do not treat a status 400 as having been uploaded.
This is simply not the behavior of the logging service.

Updates #tailscale/corp#11213

Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2 years ago
..
childproc all: update copyright and license headers 2 years ago
debug.go all: avoid repeated default interface lookups 2 years ago
depaware.txt logtail: be less aggressive about re-uploads (#8117) 2 years ago
generate.go cmd/mkmanifest, cmd/tailscale, cmd/tailscaled: remove Windows arm32 resources from OSS 2 years ago
install_darwin.go all: update copyright and license headers 2 years ago
install_windows.go all: update copyright and license headers 2 years ago
manifest_windows_386.syso cmd/{tailscale,tailscaled}: embed manifest into Windows binaries 2 years ago
manifest_windows_amd64.syso cmd/{tailscale,tailscaled}: embed manifest into Windows binaries 2 years ago
manifest_windows_arm64.syso cmd/{tailscale,tailscaled}: embed manifest into Windows binaries 2 years ago
proxy.go all: update copyright and license headers 2 years ago
required_version.go all: update to Go 1.20, use strings.CutPrefix/Suffix instead of our fork 2 years ago
ssh.go all: update copyright and license headers 2 years ago
taildrop.go various: add detection and Taildrop for Unraid 2 years ago
tailscaled.defaults cmd/tailscaled: rename relaynode reference in defaults file comment 5 years ago
tailscaled.go fix sys.Set(router) issue will crash the daemon in some OSs 2 years ago
tailscaled.openrc tstest/integration/vms: use an in-process logcatcher (#2360) 3 years ago
tailscaled.service cmd/tailscaled: use explicit equal sign in --port=$PORT in tailscaled.service 2 years ago
tailscaled_bird.go all: update copyright and license headers 2 years ago
tailscaled_notwindows.go all: update copyright and license headers 2 years ago
tailscaled_test.go all: update copyright and license headers 2 years ago
tailscaled_windows.go tsd: add package with System type to unify subsystem init, discovery 2 years ago
windows-manifest.xml cmd/{tailscale,tailscaled}: embed manifest into Windows binaries 2 years ago
with_cli.go all: update copyright and license headers 2 years ago