tailscale

Commit Graph

Author	SHA1	Message	Date
Brad Fitzpatrick	7c1d6e35a5	all: use Go 1.22 range-over-int Updates #11058 Change-Id: I35e7ef9b90e83cac04ca93fd964ad00ed5b48430 Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>	7 months ago
Joe Tsai	7a77a2edf1	logtail: optimize JSON processing (#11671 ) Changes made: * Avoid "encoding/json" for JSON processing, and instead use "github.com/go-json-experiment/json/jsontext". Use jsontext.Value.IsValid for validation, which is much faster. Use jsontext.AppendQuote instead of our own JSON escaping. * In drainPending, use a different maxLen depending on lowMem. In lowMem mode, it is better to perform multiple uploads than it is to construct a large body that OOMs the process. * In drainPending, if an error is encountered draining, construct an error message in the logtail JSON format rather than something that is invalid JSON. * In appendTextOrJSONLocked, use jsontext.Decoder to check whether the input is a valid JSON object. This is faster than the previous approach of unmarshaling into map[string]any and then re-marshaling that data structure. This is especially beneficial for network flow logging, which produces relatively large JSON objects. * In appendTextOrJSONLocked, enforce maxSize on the input. If too large, then we may end up in a situation where the logs can never be uploaded because it exceeds the maximum body size that the Tailscale logs service accepts. * Use "tailscale.com/util/truncate" to properly truncate a string on valid UTF-8 boundaries. * In general, remove unnecessary spaces in JSON output. Performance: name old time/op new time/op delta WriteText 776ns ± 2% 596ns ± 1% -23.24% (p=0.000 n=10+10) WriteJSON 110µs ± 0% 9µs ± 0% -91.77% (p=0.000 n=8+8) name old alloc/op new alloc/op delta WriteText 448B ± 0% 0B -100.00% (p=0.000 n=10+10) WriteJSON 37.9kB ± 0% 0.0kB ± 0% -99.87% (p=0.000 n=10+10) name old allocs/op new allocs/op delta WriteText 1.00 ± 0% 0.00 -100.00% (p=0.000 n=10+10) WriteJSON 1.08k ± 0% 0.00k ± 0% -99.91% (p=0.000 n=10+10) For text payloads, this is 1.30x faster. For JSON payloads, this is 12.2x faster. Updates #cleanup Updates tailscale/corp#18514 Signed-off-by: Joe Tsai <joetsai@digital-static.net>	7 months ago
as2643	3fb6ee7fdb	tailscale/logtail: redact public ipv6 and ipv4 ip addresses within tailscaled. (#10531 ) Updates #15664 Signed-off-by: Anishka Singh <anishkasingh66@gmail.com>	11 months ago
Brad Fitzpatrick	d852c616c6	logtail: fix Logger.Write return result io.Writer says you need to write completely on err=nil. (the result int should be the same as the input buffer length) We weren't doing that. We used to, but at some point the verbose filtering was modifying buf before the final return of len(buf). We've been getting lucky probably, that callers haven't looked at our results and turned us into a short write error. Updates #cleanup Updates tailscale/corp#15664 Change-Id: I01e765ba35b86b759819e38e0072eceb9d10d75c Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>	1 year ago
Claire Wang	e1bcecc393	logtail: use tstime (#8607 ) Updates #8587 Signed-off-by: Claire Wang <claire@tailscale.com>	1 year ago
Joe Tsai	283a84724f	types/logid: simplify implementation (#7415 ) Share the same underlying implementation for both PrivateID and PublicID. For the shared methods, declare them in the same order. Only keep documentation on methods without obvious meaning. Signed-off-by: Joe Tsai <joetsai@digital-static.net>	2 years ago
Joe Tsai	7e4788e383	logtail: delete ID types and functions (#7412 ) These have been moved to the types/logid package. Signed-off-by: Joe Tsai <joetsai@digital-static.net>	2 years ago
Mihai Parparita	f0f2b2e22b	logtail: increase maximum log line size in low memory mode The 255 byte limit was chosen more than 3 years ago (tailscale/corp@929635c9d9), when iOS was operating under much more significant memory constraints. With iOS 15 the network extension has an increased limit, so increasing it to 4K should be fine. The motivating factor was that the network interfaces being logged by linkChange in wgengine/userspace.go were getting truncated, and it would be useful to know why in some cases we're choosing the pdp_ip1 cell interface instead of the pdp_ip0 one. Updates #7184 Updates #7188 Signed-off-by: Mihai Parparita <mihai@tailscale.com>	2 years ago
Will Norris	71029cea2d	all: update copyright and license headers This updates all source files to use a new standard header for copyright and license declaration. Notably, copyright no longer includes a date, and we now use the standard SPDX-License-Identifier header. This commit was done almost entirely mechanically with perl, and then some minimal manual fixes. Updates #6865 Signed-off-by: Will Norris <will@tailscale.com>	2 years ago
Eng Zer Jun	f0347e841f	refactor: move from io/ioutil to io and os packages The io/ioutil package has been deprecated as of Go 1.16 [1]. This commit replaces the existing io/ioutil functions with their new definitions in io and os packages. Reference: https://golang.org/doc/go1.16#ioutil Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>	2 years ago
Mihai Parparita	3222bce02d	logtail: add instance metadata to the entry logtail Allows instances that are running with the same machine ID (due to cloning) to be distinguished. Also adds sequence numbers to detect duplicates. For tailscale/corp#5244 Signed-off-by: Mihai Parparita <mihai@tailscale.com>	3 years ago
Josh Bleecher Snyder	0868329936	all: use any instead of interface{} My favorite part of generics. Signed-off-by: Josh Bleecher Snyder <josh@tailscale.com>	3 years ago
Brad Fitzpatrick	84138450a4	types/logger, logtail: add mechanism to do structured JSON logs e.g. the change to ipnlocal in this commit ultimately logs out: {"logtail":{"client_time":"2022-02-17T20:40:30.511381153-08:00","server_time":"2022-02-18T04:40:31.057771504Z"},"type":"Hostinfo","val":{"GoArch":"amd64","Hostname":"tsdev","IPNVersion":"1.21.0-date.20220107","OS":"linux","OSVersion":"Debian 11.2 (bullseye); kernel=5.10.0-10-amd64"},"v":1} Change-Id: I668646b19aeae4a2fed05170d7b279456829c844 Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>	3 years ago
Brad Fitzpatrick	5d9ab502f3	logtail: don't strip verbose level on upload For analysis of log spam. Bandwidth is ~unchanged from had we not stripped the "[vN] " from text; it just gets restructed intot he new "v":N, field. I guess it adds one byte. Updates #1548 Change-Id: Ie00a4e0d511066a33d10dc38d765d92b0b044697 Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>	3 years ago
Josh Bleecher Snyder	9fe5ece833	logtail: cap the buffer size in encodeText This started as an attempt to placate GitHub's code scanner, but it's also probably generally a good idea. Signed-off-by: Josh Bleecher Snyder <josh@tailscale.com>	3 years ago
Josh Bleecher Snyder	94fb42d4b2	all: use testingutil.MinAllocsPerRun There are a few remaining uses of testing.AllocsPerRun: Two in which we only log the number of allocations, and one in which dynamically calculate the allocations target based on a different AllocsPerRun run. This also allows us to tighten the "no allocs" test in wgengine/filter. Signed-off-by: Josh Bleecher Snyder <josh@tailscale.com>	3 years ago
Josh Bleecher Snyder	278e7de9c9	logtail: always send a json array The code goes to some effort to send a single JSON object when there's only a single line and a JSON array when there are multiple lines. It makes the code more complex and more expensive; when we add a second line, we have to use a second buffer to duplicate the first one after adding a leading square brackets. The savings come to two bytes. Instead, always send an array. Signed-off-by: Josh Bleecher Snyder <josh@tailscale.com>	3 years ago
Brad Fitzpatrick	b34fbb24e8	logtail: reduce PublicID.UnmarshalText from 2 allocs to 0 Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>	4 years ago
Denton Gentry	013da6660e	logtail: add tests + add a test for parseAndRemoveLogLevel() + add a test for drainPendingMessages() + test JSON log encoding including several special cases Other tests frequently send logs but a) don't check the result and b) do so by happenstance, such that the code in encode() was not consistently being exercised and leading to spurious changes in code coverage. These tests attempt to more systematically test the logging function. This is the second attempt to add these tests, the first attempt (in https://github.com/tailscale/tailscale/pull/1114) had two issues: 1. httptest.NewServer creates multiple goroutine handlers, and logtail uses goroutines to upload, but the first version had no locking in the server to guard this. Moved data handling into channels to get synchronization. 2. The channel to notify the test of the arrival of data had a depth of 1, in cases where the Logger sent multiple uploads it would block the server. This resulted in the first iteration of these tests being flaky, and we reverted it. This new version of the tests has passed with go test -race -count=10000 and seems solid. Signed-off-by: Denton Gentry <dgentry@tailscale.com>	4 years ago
Denton Gentry	ce058c8280	Revert "Add logtail tests (#1114 )" (#1116 ) This reverts commit `e4f53e9b6f`. At least two of these tests are flakey, reverting until they can be made more robust. Signed-off-by: Denton Gentry <dgentry@tailscale.com>	4 years ago
Denton Gentry	e4f53e9b6f	Add logtail tests (#1114 ) * logtail: test parseAndRemoveLogLevel() Signed-off-by: Denton Gentry <dgentry@tailscale.com> * logtail: test JSON log encoding. Expand TestUploadMessages to also exercise the encoding functions in logtail, like JSON logging and timestamps. Other tests frequently send logs but a) don't check the result and b) do so by happenstance, such that the lines in encode() were not consistently being exercised and leading to spurious changes in code coverage. Signed-off-by: Denton Gentry <dgentry@tailscale.com> * logtail: add a test for drainPendingMessages Make the client buffer some messages before the upload server becomes available. Signed-off-by: Denton Gentry <dgentry@tailscale.com> * logtail: use %q, raw strings, and io.WriteString %q escapes binary characters for us. raw strings avoid so much backslash escaping Signed-off-by: Denton Gentry <dgentry@tailscale.com>	4 years ago
Denton Gentry	b771a1363b	logtail: start a local server for TestFastShutdown Right now TestFastShutdown tries to upload logs to localhost:1234, which will most likely respond with an error. However if one has an actual service running on port 1234, it would receive a connection attempting to POST every time the unit test runs. Start a local server and direct the upload there instead. Signed-off-by: Denton Gentry <dgentry@tailscale.com>	4 years ago
Denton Gentry	2c328da094	logtail: add a test to upload logs to local server Start an HTTP server to accept POST requests, and upload some logs to it. Check that uploaded logs were received. Code in logtail:drainPending was not being reliably exercised by other tests. This shows up in code coverage reports, as lines of code in drainPending are alternately added and subtracted from code coverage. This test will reliably exercise and verify this code. Signed-off-by: Denton Gentry <dgentry@tailscale.com>	4 years ago
Brad Fitzpatrick	d97ee12179	logtail, logpolicy: remove an unidiomatic use of an interface	4 years ago
Elias Naur	bca9fe35ba	logtail: return correct write size from logger.Write Signed-off-by: Elias Naur <mail@eliasnaur.com>	4 years ago
Avery Pennarun	08acb502e5	Add tstest.PanicOnLog(), and fix various problems detected by this. If a test calls log.Printf, 'go test' horrifyingly rearranges the output to no longer be in chronological order, which makes debugging virtually impossible. Let's stop that from happening by making log.Printf panic if called from any module, no matter how deep, during tests. This required us to change the default error handler in at least one http.Server, as well as plumbing a bunch of logf functions around, especially in magicsock and wgengine, but also in logtail and backoff. To add insult to injury, 'go test' also rearranges the output when a parent test has multiple sub-tests (all the sub-test's t.Logf is always printed after all the parent tests t.Logf), so we need to screw around with a special Logf that can point at the "current" t (current_t.Logf) in some places. Probably our entire way of using subtests is wrong, since 'go test' would probably like to run them all in parallel if you called t.Parallel(), but it definitely can't because the're all manipulating the shared state created by the parent test. They should probably all be separate toplevel tests instead, with common setup/teardown logic. But that's a job for another time. Signed-off-by: Avery Pennarun <apenwarr@tailscale.com>	5 years ago
Brad Fitzpatrick	fcb6a34f4b	logtail: reduce allocations encoding text Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>	5 years ago
Earl Lee	a8d8b8719a	Move Linux client & common packages into a public repo.	5 years ago

28 Commits (1016aa045fb3f31b40d4038828114a5f1fb3547b)