You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
tailscale/logtail
Joe Tsai 84c99fe0d9
logtail: be less aggressive about re-uploads (#8117)
The retry logic was pathological in the following ways:

* If we restarted the logging service, any pending uploads
would be placed in a retry-loop where it depended on backoff.Backoff,
which was too aggresive. It would retry failures within milliseconds,
taking at least 10 retries to hit a delay of 1 second.

* In the event where a logstream was rate limited,
the aggressive retry logic would severely exacerbate the problem
since each retry would also log an error message.
It is by chance that the rate of log error spam
does not happen to exceed the rate limit itself.

We modify the retry logic in the following ways:

* We now respect the "Retry-After" header sent by the logging service.

* Lacking a "Retry-After" header, we retry after a hard-coded period of
30 to 60 seconds. This avoids the thundering-herd effect when all nodes
try reconnecting to the logging service at the same time after a restart.

* We do not treat a status 400 as having been uploaded.
This is simply not the behavior of the logging service.

Updates #tailscale/corp#11213

Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2 years ago
..
backoff
example
filch all: use tstest.Replace more 2 years ago
.gitignore
README.md
api.md
buffer.go
logtail.go logtail: be less aggressive about re-uploads (#8117) 2 years ago
logtail_test.go

README.md

Tailscale Logs Service

This github repository contains libraries, documentation, and examples for working with the public API of the tailscale logs service.

For a very quick introduction to the core features, read the API docs and peruse the logs reprocessing example.

For more information, write to info@tailscale.io.