Commit Graph

24 Commits (b42b9817b0fa7c209c495949e40a3337d0570f2f)

Author SHA1 Message Date
Percy Wegmann 91a1019ee2 cmd/testwrapper: apply results of all unit tests to coverage for all packages
This allows coverage from tests that hit multiple packages at once
to be reflected in all those packages' coverage.

Updates #cleanup

Signed-off-by: Percy Wegmann <percy@tailscale.com>
9 months ago
Andrew Lytvynov 2716250ee8
all: cleanup unused code, part 2 (#10670)
And enable U1000 check in staticcheck.

Updates #cleanup

Signed-off-by: Andrew Lytvynov <awly@tailscale.com>
11 months ago
Percy Wegmann 17501ea31a ci: report test coverage to coveralls.io
This records test coverage for the amd64 no race tests and uploads the
results to coveralls.io.

Updates #cleanup

Signed-off-by: Ox Cart <ox.to.a.cart@gmail.com>
1 year ago
Brad Fitzpatrick 4dec0c6eb9 tstest, tstest/integration, github/workflows: shard integration tests
Over four jobs for now.

Updates #cleanup

Change-Id: Ic2b1a739a454916893945a3f9efc480d6fcbd70b
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
1 year ago
Maisem Ali e6ab7d3c14 cmd/testwrapper: parse args better
Previously we were just smushing together args and not trying
to parse the values at all. This resulted in the args to testwrapper
being limited and confusing.

This makes it so that testwrapper parses flags in the exact format as `go test`
command and passes them down in the provided order. It uses tesing.Init to
register flags that `go test` understands, however those are not the only
flags understood by `go test` (such as `-exec`) so we register these separately.

Updates tailscale/corp#14975

Signed-off-by: Maisem Ali <maisem@tailscale.com>
1 year ago
Brad Fitzpatrick c363b9055d tstest/integration: add tests for tun mode (requiring root)
Updates #7894

Change-Id: Iff0b07b21ae28c712dd665b12918fa28d6f601d0
Co-authored-by: Maisem Ali <maisem@tailscale.com>
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
1 year ago
Brad Fitzpatrick 93c6e1d53b tstest/deptest: add check that x/exp/{maps,slices} imported as xfoo
Updates #cleanup

Change-Id: I4cbb5e477c739deddf7a46b66f286c9fdb106279
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
1 year ago
Brad Fitzpatrick 856d32b4a9 cmd/testwrapper: include flake URL in JSON metadata
Updates tailscale/corp#14975

Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
1 year ago
Brad Fitzpatrick 2c92f94e2a cmd/testwrapper: output machine-readable JSON on test flakes
For parsing by other tools.

Updates tailscale/corp#14975

Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
1 year ago
Paul Scott 683ba62f3e
cmd/testwrapper: fix exit deflake (#9342)
Sometimes `go test` would exit and close its stdout before we started reading
it, and we would return that "file closed" error then forget to os.Exit(1).
Fixed to prefer the go test subprocess error and exit regardless of the type of
error.

Fixes #9334

Signed-off-by: Paul Scott <paul@tailscale.com>
1 year ago
Brad Fitzpatrick 0396366aae cmd/testwrapper/flakytest: don't spam stderr in Mark when not under wrapper
If the user's running "go test" by hand, no need to spam stderr with
the sentinel marker. It already calls t.Logf (which only gets output
on actual failure, or verbose mode) which is enough to tell users it's
known flaky. Stderr OTOH always prints out and is distracting to
manual "go test" users.

Updates #cleanup

Change-Id: Ie5e6881bae291787c30f75924fa132f4a28abbb2
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
1 year ago
Paul Scott 96094cc07e
cmd/testwrapper: exit code 1 when go build fails (#9276)
Fixes #9275
Fixes #8586
Fixes tailscale/corp#13115

Signed-off-by: Paul Scott <paul@tailscale.com>
1 year ago
Maisem Ali 0f397baf77 cmd/testwrapper: emit logs of failed tests on timeout
It would just fail the entire pkg, but would not print any
logs. It was already tracking all the logs, so have it emit
them when the pkg fails/times out.

Updates #9231

Signed-off-by: Maisem Ali <maisem@tailscale.com>
1 year ago
Maisem Ali ae747a2e48 cmd/testwrapper: handle timeouts as test failures
While investigating the fix in 7538f38671,
I was curious why the testwrapper didn't fail. Turns out if the test
times out and there was no explicit failure, the only message we get
is that the overall pkg failed and no failure information about the
individual test. This resulted in a 0 exit code.

This fixes that by failing the explicit case of the pkg failing when
there is nothing to retry for that pkg.

Updates #8493

Signed-off-by: Maisem Ali <maisem@tailscale.com>
1 year ago
Maisem Ali ff7f4b4224 cmd/testwrapper: fix off-by-one error in maxAttempts check
It was checking if `>= maxAttempts` which meant that the third
attempt would never run.

Updates #8493

Signed-off-by: Maisem Ali <maisem@tailscale.com>
1 year ago
Brad Fitzpatrick bc0eb6b914 all: import x/exp/maps as xmaps to distinguish from Go 1.21 "maps"
Updates #8419

Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
1 year ago
Maisem Ali 8cda647a0f cmd/testwrapper: handle build failures
`go test -json` outputs invalid JSON when a build fails.
Handle that case by reseting the json.Decode and continuing to read.

Updates #8493

Signed-off-by: Maisem Ali <maisem@tailscale.com>
1 year ago
Maisem Ali 1ca5dcce15 cmd/testwrapper: stream output results
Previously it would wait for all tests to run before printing anything,
instead stream the results over a channel so that they can be emitted
immediately.

Updates #8493

Signed-off-by: Maisem Ali <maisem@tailscale.com>
1 year ago
Maisem Ali 2e4e7d6b9d cmd/testwrapper: output packages tested
Previously it would only print the failures without providing
more information on which package the failures from.

This commit makes it so that it prints out the package information
as well as the attempt numbers.

```
➜  tailscale.com git:(main) ✗ go run ./cmd/testwrapper ./cmd/...
ok       tailscale.com/cmd/derper
ok       tailscale.com/cmd/k8s-operator
ok       tailscale.com/cmd/tailscale/cli
ok       tailscale.com/cmd/tailscaled
=== RUN   TestFlakeRun
    flakytest.go:38: flakytest: issue tracking this flaky test: https://github.com/tailscale/tailscale/issues/0
    flakytest_test.go:41: First run in testwrapper, failing so that test is retried. This is expected.
--- FAIL: TestFlakeRun (0.00s)
FAIL     tailscale.com/cmd/testwrapper/flakytest

Attempt #2: Retrying flaky tests:

ok       tailscale.com/cmd/testwrapper/flakytest
```

Updates #8493

Signed-off-by: Maisem Ali <maisem@tailscale.com>
1 year ago
Maisem Ali 12a2221db2 cmd/testwrapper/flakytest: clearly describe why TestFlakeRun fails
Fixes #8474

Signed-off-by: Maisem Ali <maisem@tailscale.com>
1 year ago
Maisem Ali 8e840489ed cmd/testwrapper: only retry flaky failed tests
Redo the testwrapper to track and only retry flaky tests instead
of retrying the entire pkg. It also fails early if a non-flaky test fails.

This also makes it so that the go test caches are used.

Fixes #7975

Signed-off-by: Maisem Ali <maisem@tailscale.com>
1 year ago
Will Norris 71029cea2d all: update copyright and license headers
This updates all source files to use a new standard header for copyright
and license declaration.  Notably, copyright no longer includes a date,
and we now use the standard SPDX-License-Identifier header.

This commit was done almost entirely mechanically with perl, and then
some minimal manual fixes.

Updates #6865

Signed-off-by: Will Norris <will@tailscale.com>
2 years ago
Andrew Dunham b74db24149 tstest/integration: mark all integration tests as flaky
Updates #7036

Change-Id: I3aec5ad680078199ba984bf8afc20b2f2eb37257
Signed-off-by: Andrew Dunham <andrew@du.nham.ca>
2 years ago
Andrew Dunham aea251d42a cmd/testwrapper: move from corp; mark magicsock test as flaky
Signed-off-by: Andrew Dunham <andrew@du.nham.ca>
Change-Id: Ibab5860f5797b3db151d3c27855333e43a9088a4
2 years ago