You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
tailscale/cmd
Joe Tsai 84c99fe0d9
logtail: be less aggressive about re-uploads (#8117)
The retry logic was pathological in the following ways:

* If we restarted the logging service, any pending uploads
would be placed in a retry-loop where it depended on backoff.Backoff,
which was too aggresive. It would retry failures within milliseconds,
taking at least 10 retries to hit a delay of 1 second.

* In the event where a logstream was rate limited,
the aggressive retry logic would severely exacerbate the problem
since each retry would also log an error message.
It is by chance that the rate of log error spam
does not happen to exceed the rate limit itself.

We modify the retry logic in the following ways:

* We now respect the "Retry-After" header sent by the logging service.

* Lacking a "Retry-After" header, we retry after a hard-coded period of
30 to 60 seconds. This avoids the thundering-herd effect when all nodes
try reconnecting to the logging service at the same time after a restart.

* We do not treat a status 400 as having been uploaded.
This is simply not the behavior of the logging service.

Updates #tailscale/corp#11213

Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2 years ago
..
addlicense all: update tools that manage copyright headers 2 years ago
cloner all: update tools that manage copyright headers 2 years ago
containerboot cmd/containerboot,kube: consolidate the two kube clients 2 years ago
derper go.mod: bump all deps possible that don't break the build 2 years ago
derpprobe prober: migrate to Prometheus metric library 2 years ago
dist release/dist: add forgotten license headers 2 years ago
get-authkey cmd/tailscale/cli: [up] add experimental oauth2 authkey support 2 years ago
gitops-pusher Add support for OAuth tokens #7394 (#7393) 2 years ago
hello all: update copyright and license headers 2 years ago
k8s-operator cmd/k8s-operator: print version in startup logs 2 years ago
mkmanifest cmd/mkmanifest, cmd/tailscale, cmd/tailscaled: remove Windows arm32 resources from OSS 2 years ago
mkpkg all: update copyright and license headers 2 years ago
mkversion version/mkversion: open-source version generation logic 2 years ago
nardump all: update copyright and license headers 2 years ago
netlogfmt all: replace logtail.{Public,Private}ID with logid.{Public,Private}ID (#7404) 2 years ago
nginx-auth tailcfg,all: add and use Node.IsTagged() 2 years ago
pgproxy various: add golangci-lint, fix issues (#7905) 2 years ago
printdep cmd/printdep: print correct toolchain URL 2 years ago
proxy-to-grafana tailcfg,all: add and use Node.IsTagged() 2 years ago
sniproxy cmd/sniproxy: Set App name in tsnet hostinfo 2 years ago
speedtest all: update copyright and license headers 2 years ago
ssh-auth-none-demo all: update copyright and license headers 2 years ago
stunc all: update copyright and license headers 2 years ago
sync-containers cmd/sync-containers: update latest and stable tags 2 years ago
tailscale various: add detection and Taildrop for Unraid 2 years ago
tailscaled logtail: be less aggressive about re-uploads (#8117) 2 years ago
testcontrol all: update copyright and license headers 2 years ago
testwrapper all: update copyright and license headers 2 years ago
tsconnect cmd/tsconnect: fix forgotten API change for wasm 2 years ago
tsshd all: update copyright and license headers 2 years ago
viewer util/codegen: support embedded fields 2 years ago