As a fallback to package managers, allow updating tailscale that was
self-installed in some way. There are some tricky bits around updating
the systemd unit (should we stick to local binary paths or to the ones
in tailscaled.service?), so leaving that out for now.
Updates #6995
Signed-off-by: Andrew Lytvynov <awly@tailscale.com>
They were entirely redundant and 1:1 with the status field
so this turns them into methods instead.
Updates #cleanup
Updates #1909
Change-Id: I7d939750749edf7dae4c97566bbeb99f2f75adbc
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Instead of confusing users, emit an event that explicitly tells the
user that HTTPS is disabled on the tailnet and that ingress may not
work until they enable it.
Updates #9141
Signed-off-by: Maisem Ali <maisem@tailscale.com>
Port 8080 is routinely used for HTTP services, make it easier to
use --forwards=tcp/8080/... by moving the metrics port out of the
way.
Updates #1748
Signed-off-by: Denton Gentry <dgentry@tailscale.com>
Ensures that Statefulset reconciler config has only one of Cluster target IP or tailnet target IP.
Adds a test case for containerboot egress proxy mode.
Updates tailscale/tailscale#8184
Signed-off-by: irbekrm <irbekrm@gmail.com>
First part of work for the functionality that allows users to create an egress
proxy to access Tailnet services from within Kubernetes cluster workloads.
This PR allows creating an egress proxy that can access Tailscale services over HTTP only.
Updates tailscale/tailscale#8184
Signed-off-by: irbekrm <irbekrm@gmail.com>
Co-authored-by: Maisem Ali <maisem@tailscale.com>
Co-authored-by: Rhea Ghosh <rhea@tailscale.com>
While investigating the fix in 7538f38671,
I was curious why the testwrapper didn't fail. Turns out if the test
times out and there was no explicit failure, the only message we get
is that the overall pkg failed and no failure information about the
individual test. This resulted in a 0 exit code.
This fixes that by failing the explicit case of the pkg failing when
there is nothing to retry for that pkg.
Updates #8493
Signed-off-by: Maisem Ali <maisem@tailscale.com>
This makes wsconn.Conns somewhat present reasonably when they are
the client of an http.Request, rather than just put a placeholder
in that field.
Updates tailscale/corp#13777
Signed-off-by: David Anderson <danderson@tailscale.com>
We would call Update on the secret, but that was racey and would occasionaly
fail. Instead use patch whenever we can.
Fixes errors like
```
boot: 2023/08/29 01:03:53 failed to set serve config: sending serve config: updating config: writing ServeConfig to StateStore: Operation cannot be fulfilled on secrets "ts-webdav-kfrzv-0": the object has been modified; please apply your changes to the latest version and try again
{"level":"error","ts":"2023-08-29T01:03:48Z","msg":"Reconciler error","controller":"ingress","controllerGroup":"networking.k8s.io","controllerKind":"Ingress","Ingress":{"name":"webdav","namespace":"default"},"namespace":"default","name":"webdav","reconcileID":"96f5cfed-7782-4834-9b75-b0950fd563ed","error":"failed to provision: failed to create or get API key secret: Operation cannot be fulfilled on secrets \"ts-webdav-kfrzv-0\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:324\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226"}
```
Updates #502
Updates #7895
Signed-off-by: Maisem Ali <maisem@tailscale.com>
Printing out JSON representation things in log output is pretty common.
Updates #cleanup
Change-Id: Ife2d2e321a18e6e1185efa8b699a23061ac5e5a4
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Reimplement `downloadURLToFile` using `distsign.Download` and move all
of the progress reporting logic over there.
Updates #6995
Updates #755
Signed-off-by: Andrew Lytvynov <awly@tailscale.com>
This was added in 3451b89e5f, but
resulted in the v6 Tailscale address being added to status when
when the forwarding only happened on the v4 address.
Updates #502
Signed-off-by: Maisem Ali <maisem@tailscale.com>
The new ingress reconcile raises events on failure, but I forgot to
add the updated permission.
Updates #502
Signed-off-by: Maisem Ali <maisem@tailscale.com>
This uses the new react-based web client for all builds, not just with
the --dev flag.
If the web client assets have not been built, the client will serve a
message that Tailscale was built without the web client, and link to
build instructions. Because we will include the web client in all of our
builds, this should only be seen by developers or users building from
source. (And eventually this will be replaced by attempting to download
needed assets as runtime.)
We do now checkin the build/index.html file, which serves the error
message when assets are unavailable. This will also eventually be used
to trigger in CI when new assets should be built and uploaded to a
well-known location.
Updates tailscale/corp#13775
Signed-off-by: Will Norris <will@tailscale.com>
This option allows logging the raw HTTP requests and responses that the
portmapper Client makes when using UPnP. This can be extremely helpful
when debugging strange UPnP issues with users' devices, and might allow
us to avoid having to instruct users to perform a packet capture.
Updates #8992
Signed-off-by: Andrew Dunham <andrew@du.nham.ca>
Change-Id: I2c3cf6930b09717028deaff31738484cc9b008e4
On k8s the serve-config secret mount is symlinked so checking against
the Name makes us miss the events.
Updates #7895
Signed-off-by: Maisem Ali <maisem@tailscale.com>
Previously, the operator would only monitor Services and create
a Tailscale StatefulSet which acted as a L3 proxy which proxied
traffic inbound to the Tailscale IP onto the services ClusterIP.
This extends that functionality to also monitor Ingress resources
where the `ingressClassName=tailscale` and similarly creates a
Tailscale StatefulSet, acting as a L7 proxy instead.
Users can override the desired hostname by setting:
```
- tls
hosts:
- "foo"
```
Hostnames specified under `rules` are ignored as we only create a single
host. This is emitted as an event for users to see.
Fixes#7895
Signed-off-by: Maisem Ali <maisem@tailscale.com>
This watches the provided path for a JSON encoded ipn.ServeConfig.
Everytime the file changes, or the nodes FQDN changes it reapplies
the ServeConfig.
At boot time, it nils out any previous ServeConfig just like tsnet does.
As the ServeConfig requires pre-existing knowledge of the nodes FQDN to do
SNI matching, it introduces a special `${TS_CERT_DOMAIN}` value in the JSON
file which is replaced with the known CertDomain before it is applied.
Updates #502
Updates #7895
Signed-off-by: Maisem Ali <maisem@tailscale.com>
Previously users would have to unexpose/expose the service in order to
change Hostname/TargetIP. This now applies those changes by causing a
StatefulSet rollout now that a61a9ab087 is in.
Updates #502
Signed-off-by: Maisem Ali <maisem@tailscale.com>
1. Add TCP port forwarding.
For example: ./sniproxy -forwards=tcp/22/github.com
will forward SSH to github.
% ssh -i ~/.ssh/id_ecdsa.pem -T git@github.com
Hi GitHubUser! You've successfully authenticated, but GitHub does not
provide shell access.
% ssh -i ~/.ssh/id_ecdsa.pem -T git@100.65.x.y
Hi GitHubUser! You've successfully authenticated, but GitHub does not
provide shell access.
2. Additionally export clientmetrics as prometheus metrics for local
scraping over the tailnet: http://sniproxy-hostname:8080/debug/varz
Updates https://github.com/tailscale/tailscale/issues/1748
Signed-off-by: Denton Gentry <dgentry@tailscale.com>
I'm not saying it works, but it compiles.
Updates #5794
Change-Id: I2f3c99732e67fe57a05edb25b758d083417f083e
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Adds a cached self node to the web client Server struct, which will
be used from the web client api to verify that request came from the
node's own machine (i.e. came from the web client frontend). We'll
be using when we switch the web client api over to acting as a proxy
to the localapi, to protect against DNS rebinding attacks.
Updates tailscale/corp#13775
Signed-off-by: Sonia Appasamy <sonia@tailscale.com>
Add `dist.Signer` hook which can arbitrarily sign linux/synology
artifacts. Plumb it through in `cmd/dist` and remove existing tarball
signing key. Distsign signing will happen on a remote machine, not using
a local key.
Updates #755
Updates #8760
Signed-off-by: Andrew Lytvynov <awly@tailscale.com>
Previously we would not reapply changes to TS_HOSTNAME etc when
then the container restarted and TS_AUTH_ONCE was enabled.
This splits those into two steps login and set, allowing us to
only rerun the set step on restarts.
Updates #502
Signed-off-by: Maisem Ali <maisem@tailscale.com>
The Windows Security Center is a component that manages the registration of
security products on a Windows system. Only products that have obtained a
special cert from Microsoft may register themselves using the WSC API.
Practically speaking, most vendors do in fact sign up for the program as it
enhances their legitimacy.
From our perspective, this is useful because it gives us a high-signal
source of information to query for the security products installed on the
system. I've tied this query into the osdiag package and is run during
bugreports.
It uses COM bindings that were automatically generated by my prototype
metadata processor, however that program still has a few bugs, so I had
to make a few manual tweaks. I dropped those binding into an internal
package because (for the moment, at least) they are effectively
purpose-built for the osdiag use case.
We also update the wingoes dependency to pick up BSTR.
Fixes#10646
Signed-off-by: Aaron Klotz <aaron@tailscale.com>
Indicate to the web client when it is running in CGI mode, and if it is
then cache the csrf key between requests.
Updates tailscale/corp#13775
Signed-off-by: Will Norris <will@tailscale.com>
Can write "wasm" instead of js || wasi1p, since there's only two:
$ go tool dist list | grep wasm
js/wasm
wasip1/wasm
Plus, if GOOS=wasip2 is added later, we're already set.
Updates #5794
Change-Id: Ifcfb187c3775c17c9141bc721512dc4577ac4434
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
We have a fancy package for doing TLS cert validation even if the machine
doesn't have TLS certs (for LetsEncrypt only) but the CLI's netcheck command
wasn't using it.
Also, update the tlsdial's outdated package docs while here.
Updates #cleanup
Change-Id: I74b3cb645d07af4d8ae230fb39a60c809ec129ad
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
This PR removes calls to ioutil library and replaces them
with their new locations in the io and os packages.
Fixes#9034
Updates #5210
Signed-off-by: Marwan Sulaiman <marwan@tailscale.com>
I forgot to move the defer out of the func, so the tsnet.Server
immediately closed after starting.
Updates #502
Signed-off-by: Maisem Ali <maisem@tailscale.com>
It was jumbled doing a lot of things, this breaks it up into
the svc reconciliation and the tailscale sts reconciliation.
Prep for future commit.
Updates #502
Signed-off-by: Maisem Ali <maisem@tailscale.com>
In order for the installer to restart the GUI correctly post-upgrade, we
need the GUI to be able to register its restart preferences.
This PR adds API support for doing so. I'm adding it to OSS so that it
is available should we need to do any such registrations on OSS binaries
in the future.
Updates https://github.com/tailscale/corp/issues/13998
Signed-off-by: Aaron Klotz <aaron@tailscale.com>
Adds ability to start Funnel in the foreground and stream incoming
connections. When foreground process is stopped, Funnel is turned
back off for the port.
Exampe usage:
```
TAILSCALE_FUNNEL_V2=on tailscale funnel 8080
```
Updates #8489
Signed-off-by: Marwan Sulaiman <marwan@tailscale.com>
Adds csrf protection and hooks up an initial POST request from
the React web client.
Updates tailscale/corp#13775
Signed-off-by: Sonia Appasamy <sonia@tailscale.com>
In b987b2ab18 (2021-01-12) when we introduced sharing we mapped
the sharer to the userid at a low layer, mostly to fix the display of
"tailscale status" and the client UIs, but also some tests.
The commit earlier today, 7dec09d169, removed the 2.5yo option
to let clients disable that automatic mapping, as clearly we were never
getting around to it.
This plumbs the Sharer UserID all the way to ipnstatus so the CLI
itself can choose to print out the Sharer's identity over the node's
original owner.
Then we stop mangling Node.User and let clients decide how they want
to render things.
To ease the migration for the Windows GUI (which currently operates on
tailcfg.Node via the NetMap from WatchIPNBus, instead of PeerStatus),
a new method Node.SharerOrUser is added to do the mapping of
Sharer-else-User.
Updates #1909
Updates tailscale/corp#1183
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Make it just a views.Slice[netip.Prefix] instead of its own named type.
Having the special case led to circular dependencies in another WIP PR
of mine.
Updates #8948
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
The tailscale serve|funnel commands frequently call the LocalBackend's Status
but they never need the peers to be included. This PR changes the call to be
StatusWithoutPeers which should gain a noticeable speed improvement
Updates #8489
Signed-off-by: Marwan Sulaiman <marwan@tailscale.com>
Previously we would use the Impersonate-Group header to pass through
tags to the k8s api server. However, we would do nothing for non-tagged
nodes. Now that we have a way to specify these via peerCaps respect those
and send down groups for non-tagged nodes as well.
For tagged nodes, it defaults to sending down the tags as groups to retain
legacy behavior if there are no caps set. Otherwise, the tags are omitted.
Updates #5055
Signed-off-by: Maisem Ali <maisem@tailscale.com>
If a node is flapping or otherwise generating lots of STUN endpoints, we
can end up caching a ton of useless values and sending them to peers.
Instead, let's apply a fixed per-Addr limit of endpoints that we cache,
so that we're only sending peers up to the N most recent.
Updates tailscale/corp#13890
Signed-off-by: Andrew Dunham <andrew@du.nham.ca>
Change-Id: I8079a05b44220c46da55016c0e5fc96dd2135ef8
When trying to use serve with https, send users through https cert
provisioning enablement before editing the ServeConfig.
Updates tailscale/corp#10577
Signed-off-by: Sonia Appasamy <sonia@tailscale.com>
1. Add metrics to funnel flow.
2. Stop blocking users from turning off funnels when no longer in
their node capabilities.
3. Rename LocalClient.IncrementMetric to IncrementCounter to better
callout its usage is only for counter clientmetrics.
Updates tailscale/corp#10577
Signed-off-by: Sonia Appasamy <sonia@tailscale.com>
This removes the unsafe/linkname and only uses the standard library.
It's a bit slower, for now, but https://go.dev/cl/518336 should get us
back.
On darwin/arm64, without https://go.dev/cl/518336
pkg: tailscale.com/tstime/mono
│ before │ after │
│ sec/op │ sec/op vs base │
MonoNow-8 16.20n ± 0% 19.75n ± 0% +21.92% (p=0.000 n=10)
TimeNow-8 39.46n ± 0% 39.40n ± 0% -0.16% (p=0.002 n=10)
geomean 25.28n 27.89n +10.33%
And with it,
MonoNow-8 16.34n ± 1% 16.93n ± 0% +3.67% (p=0.001 n=10)
TimeNow-8 39.55n ± 15% 38.46n ± 1% -2.76% (p=0.000 n=10)
geomean 25.42n 25.52n +0.41%
Updates #8839
Updates tailscale/go#70
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Netcheck no longer performs I/O itself, instead it makes requests via
SendPacket and expects users to route reply traffic to
ReceiveSTUNPacket.
Netcheck gains a Standalone function that stands up sockets and
goroutines to implement I/O when used in a standalone fashion.
Magicsock now unconditionally routes STUN traffic to the netcheck.Client
that it hosts, and plumbs the send packet sink.
The CLI is updated to make use of the Standalone mode.
Fixes#8723
Signed-off-by: James Tucker <james@tailscale.com>
Currently just serving a "Hello world" page when running the web
cli in --dev mode.
Updates tailscale/corp#13775
Co-authored-by: Will Norris <will@tailscale.com>
Signed-off-by: Sonia Appasamy <sonia@tailscale.com>
Extract the self-update logic from cmd/tailscale/cli into a standalone
package that could be used from tailscaled later.
Updates #6995
Signed-off-by: Andrew Lytvynov <awly@tailscale.com>
move the tailscale web client out of the cmd/tailscale/cli package, into
a new client/web package. The remaining cli/web.go file is still
responsible for parsing CLI flags and such, and then calls into
client/web. This will allow the web client to be hooked into from other
contexts (for example, from a tsnet server), and provide a dedicated
space to add more functionality to this client.
Updates tailscale/corp#13775
Signed-off-by: Will Norris <will@tailscale.com>
Refactor two shared functions used by the tailscale cli,
calcAdvertiseRoutes and licensesURL. These are used by the web client as
well as other tailscale subcommands. The web client is being moved out
of the cli package, so move these two functions to new locations.
Updates tailscale/corp#13775
Signed-off-by: Will Norris <will@tailscale.com>
This adds the capability to pad disco ping message payloads to reach a
specified size. It also plumbs it through to the tailscale ping -size
flag.
Disco pings used for actual endpoint discovery do not use this yet.
Updates #311.
Signed-off-by: salman <salman@tailscale.com>
Co-authored-by: Val <valerie@tailscale.com>
Plumb a signing callback function to `unixpkgs.rpmTarget` to allow
signing RPMs. This callback is optional and RPMs will build unsigned if
not set, just as before.
Updates https://github.com/tailscale/tailscale/issues/1882
Signed-off-by: Andrew Lytvynov <awly@tailscale.com>
Upgrade the nfpm package to the latest version to pick up
24a43c5ad7.
The upgrade is from v0 to v2, so there was some breakage to fix.
Generated packages should have the same contents as before.
Updates https://github.com/tailscale/tailscale/issues/1882
Signed-off-by: Andrew Lytvynov <awly@tailscale.com>
* We update wingoes to pick up new version information functionality
(See pe/version.go in the https://github.com/dblohm7/wingoes repo);
* We move the existing LogSupportInfo code (including necessary syscall
stubs) out of util/winutil into a new package, util/osdiag, and implement
the public LogSupportInfo function may be implemented for other platforms
as needed;
* We add a new reason argument to LogSupportInfo and wire that into
localapi's bugreport implementation;
* We add module information to the Windows implementation of LogSupportInfo
when reason indicates a bugreport. We enumerate all loaded modules in our
process, and for each one we gather debug, authenticode signature, and
version information.
Fixes#7802
Signed-off-by: Aaron Klotz <aaron@tailscale.com>
Add optional `--upstream` flag to `tailscale version` to fetch the
latest upstream release version from `pkgs.tailscale.com`. This is
useful to diagnose `tailscale update` behavior or write other tooling.
Example output:
$ tailscale version --upstream --json
{
"majorMinorPatch": "1.47.35",
"short": "1.47.35",
"long": "1.47.35-t6afffece8",
"unstableBranch": true,
"gitCommit": "6afffece8a32509aa7a4dc2972415ec58d8316de",
"cap": 66,
"upstream": "1.45.61"
}
Fixes#8669
RELNOTE=adds "tailscale version --upstream"
Signed-off-by: Andrew Lytvynov <awly@tailscale.com>
The revoke-keys command allows nodes with tailnet lock keys
to collaborate to erase the use of a compromised key, and remove trust
in it.
Signed-off-by: Tom DNetto <tom@tailscale.com>
Updates ENG-1848
Previously, tailscale upgrade was doing the bare minimum for checking
authenticode signatures via `WinVerifyTrustEx`. This is fine, but we can do
better:
* WinVerifyTrustEx verifies that the binary's signature is valid, but it doesn't
determine *whose* signature is valid; tailscale upgrade should also ensure that
the binary is actually signed *by us*.
* I added the ability to check the signatures of MSI files.
* In future PRs I will be adding diagnostic logging that lists details about
every module (ie, DLL) loaded into our process. As part of that metadata, I
want to be able to extract information about who signed the binaries.
This code is modelled on some C++ I wrote for Firefox back in the day. See
https://searchfox.org/mozilla-central/rev/27e4816536c891d85d63695025f2549fd7976392/toolkit/xre/dllservices/mozglue/Authenticode.cpp
for reference.
Fixes#8284
Signed-off-by: Aaron Klotz <aaron@tailscale.com>
Count number of sessions, number of DNS queries answered
successfully and in error, and number of http->https redirects.
Updates #1748
Signed-off-by: Denton Gentry <dgentry@tailscale.com>
Pass an optional PEM-encoded ECDSA key to `cmd/dist` to sign all built
tarballs. The signature is stored next to the tarball with a `.sig`
extension.
Tested this with an `openssl`-generated key pair and verified the
resulting signature.
Updates #8760
Signed-off-by: Andrew Lytvynov <awly@tailscale.com>
This change introduces a new subcommand, `exit-node`, along with a
subsubcommand of `list` and a `--filter` flag.
Exit nodes without location data will continue to be displayed when
`status` is used. Exit nodes with location data will only be displayed
behind `exit-node list`, and in status if they are the active exit node.
The `filter` flag can be used to filter exit nodes with location data by
country.
Exit nodes with Location.Priority data will have only the highest
priority option for each country and city listed. For countries with
multiple cities, a <Country> <Any> option will be displayed, indicating
the highest priority node within that country.
Updates tailscale/corp#13025
Signed-off-by: Charlotte Brandhorst-Satzkorn <charlotte@tailscale.com>
Implement `tailscale update` on FreeBSD. This is much simpler than other
platforms because `pkg rquery` lets us get the version in their repos
without any extra parsing.
Updates #6995
Signed-off-by: Andrew Lytvynov <awly@tailscale.com>
Define PeerCapabilty and PeerCapMap as the new way of sending down
inter-peer capability information.
Previously, this was unstructured and you could only send down strings
which got too limiting for certain usecases. Instead add the ability
to send down raw JSON messages that are opaque to Tailscale but provide
the applications to define them however they wish.
Also update accessors to use the new values.
Updates #4217
Signed-off-by: Maisem Ali <maisem@tailscale.com>
Similar to Arch support, use the latest version info from the official
`apk` repo and don't offer explicit track or version switching.
Add detection for Alpine Linux in version/distro along the way.
Updates #6995
Signed-off-by: Andrew Lytvynov <awly@tailscale.com>
This is the Fedora family of distros, including CentOS, RHEL and others.
Tested in `fedora:latest` and `centos:7` containers.
Updates #6995
Signed-off-by: Andrew Lytvynov <awly@tailscale.com>
The util/linuxfw/iptables.go had a bunch of code that wasn't yet used
(in prep for future work) but because of its imports, ended up
initializing code deep within gvisor that panicked on init on arm64
systems not using 4KB pages.
This deletes the unused code to delete the imports and remove the
panic. We can then cherry-pick this back to the branch and restore it
later in a different way.
A new test makes sure we don't regress in the future by depending on
the panicking package in question.
Fixes#8658
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Arch version of tailscale is not maintained by us, but is generally
up-to-date with our releases. Therefore "tailscale update" is just a
thin wrapper around "pacman -Sy tailscale" with different flags.
Updates #6995
Signed-off-by: Andrew Lytvynov <awly@tailscale.com>
* cmd/tailscale/cli: make `tailscale update` query `softwareupdate`
Even on macOS when Tailscale was installed via the App Store, we can check for
and even install new versions if people ask explicitly. Also, warn if App Store
AutoUpdate is not turned on.
Updates #6995
This changes the ACLTestError type to reuse the existing/identical
types from the ACL implementation, to avoid issues in the future if
the two types fall out of sync.
Updates #8645
Signed-off-by: Jenny Zhang <jz@tailscale.com>
When using a custom http port like 8080, this was resulting in a
constructed hostname of `host.tailnet.ts.net:8080.tailnet.ts.net` when
looking up the serve handler. Instead, strip off the port before adding
the MagicDNS suffix.
Also use the actual hostname in `serve status` rather than the literal
string "host".
Fixes#8635
Signed-off-by: Will Norris <will@tailscale.com>
Add a few helper functions in tsweb to add common security headers to handlers. Use those functions for all non-tailscaled-facing endpoints in derper.
Signed-off-by: Andrew Lytvynov <awly@tailscale.com>
`go test -json` outputs invalid JSON when a build fails.
Handle that case by reseting the json.Decode and continuing to read.
Updates #8493
Signed-off-by: Maisem Ali <maisem@tailscale.com>
This allows sending logs from the "logpolicy" package (and associated
callees) to something other than the log package. The behaviour for
tailscaled remains the same, passing in log.Printf
Updates #8249
Signed-off-by: Andrew Dunham <andrew@du.nham.ca>
Change-Id: Ie1d43b75fa7281933d9225bffd388462c08a5f31
When performing a fallback DNS query, run the recursive resolver in a
separate goroutine and compare the results returned by the recursive
resolver with the results we get from "regular" bootstrap DNS. This will
allow us to gather data about whether the recursive DNS resolver works
better, worse, or about the same as "regular" bootstrap DNS.
Updates #5853
Signed-off-by: Andrew Dunham <andrew@du.nham.ca>
Change-Id: Ifa0b0cc9eeb0dccd6f7a3d91675fe44b3b34bd48
Previously it would wait for all tests to run before printing anything,
instead stream the results over a channel so that they can be emitted
immediately.
Updates #8493
Signed-off-by: Maisem Ali <maisem@tailscale.com>
Previously it would only print the failures without providing
more information on which package the failures from.
This commit makes it so that it prints out the package information
as well as the attempt numbers.
```
➜ tailscale.com git:(main) ✗ go run ./cmd/testwrapper ./cmd/...
ok tailscale.com/cmd/derper
ok tailscale.com/cmd/k8s-operator
ok tailscale.com/cmd/tailscale/cli
ok tailscale.com/cmd/tailscaled
=== RUN TestFlakeRun
flakytest.go:38: flakytest: issue tracking this flaky test: https://github.com/tailscale/tailscale/issues/0
flakytest_test.go:41: First run in testwrapper, failing so that test is retried. This is expected.
--- FAIL: TestFlakeRun (0.00s)
FAIL tailscale.com/cmd/testwrapper/flakytest
Attempt #2: Retrying flaky tests:
ok tailscale.com/cmd/testwrapper/flakytest
```
Updates #8493
Signed-off-by: Maisem Ali <maisem@tailscale.com>
This change is introducing new netfilterRunner interface and moving iptables manipulation to a lower leveled iptables runner.
For #391
Signed-off-by: KevinLiang10 <kevinliang@tailscale.com>
Redo the testwrapper to track and only retry flaky tests instead
of retrying the entire pkg. It also fails early if a non-flaky test fails.
This also makes it so that the go test caches are used.
Fixes#7975
Signed-off-by: Maisem Ali <maisem@tailscale.com>
In order to improve our ability to understand the state of policies and
registry settings when troubleshooting, we enumerate all values in all subkeys.
x/sys/windows does not already offer this, so we need to call RegEnumValue
directly.
For now we're just logging this during startup, however in a future PR I plan to
also trigger this code during a bugreport. I also want to log more than just
registry.
Fixes#8141
Signed-off-by: Aaron Klotz <aaron@tailscale.com>
The client/tailscale is a stable-ish API we try not to break. Revert
the Client.CreateKey method as it was and add a new
CreateKeyWithExpiry method to do the new thing. And document the
expiry field and enforce that the time.Duration can't be between in
range greater than 0 and less than a second.
Updates #7143
Updates #8124 (reverts it, effectively)
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Adds a parameter for create key that allows a number of seconds
(less than 90) to be specified for new keys.
Fixes https://github.com/tailscale/tailscale/issues/7965
Signed-off-by: Matthew Brown <matthew@bargrove.com>
getSingleObject can return `nil, nil`, getDeviceInfo was not handling
that case which resulted in panics.
Fixes#7303
Signed-off-by: Maisem Ali <maisem@tailscale.com>
The retry logic was pathological in the following ways:
* If we restarted the logging service, any pending uploads
would be placed in a retry-loop where it depended on backoff.Backoff,
which was too aggresive. It would retry failures within milliseconds,
taking at least 10 retries to hit a delay of 1 second.
* In the event where a logstream was rate limited,
the aggressive retry logic would severely exacerbate the problem
since each retry would also log an error message.
It is by chance that the rate of log error spam
does not happen to exceed the rate limit itself.
We modify the retry logic in the following ways:
* We now respect the "Retry-After" header sent by the logging service.
* Lacking a "Retry-After" header, we retry after a hard-coded period of
30 to 60 seconds. This avoids the thundering-herd effect when all nodes
try reconnecting to the logging service at the same time after a restart.
* We do not treat a status 400 as having been uploaded.
This is simply not the behavior of the logging service.
Updates #tailscale/corp#11213
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
I noticed cmd/{cloner,viewer} didn't support structs with embedded
fields while working on a change in another repo. This adds support.
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Signed-off-by: Chenyang Gao <gps949@outlook.com>
in commit 6e96744, the tsd system type has been added.
Which will cause the daemon will crash on some OSs (Windows, darwin and so on).
The root cause is that on those OSs, handleSubnetsInNetstack() will return true and set the conf.Router with a wrapper.
Later in NewUserspaceEngine() it will do subsystem set and found that early set router mismatch to current value, then panic.
This is part of an effort to clean up tailscaled initialization between
tailscaled, tailscaled Windows service, tsnet, and the mac GUI.
Updates #8036
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
This holds back gvisor, kubernetes, goreleaser, and esbuild, which all
had breaking API changes.
Updates #8043
Updates #7381
Updates #8042 (updates u-root which adds deps)
Change-Id: I889759bea057cd3963037d41f608c99eb7466a5b
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
This change introduces address selection for wireguard only endpoints.
If a endpoint has not been used before, an address is randomly selected
to be used based on information we know about, such as if they are able
to use IPv4 or IPv6. When an address is initially selected, we also
initiate a new ICMP ping to the endpoints addresses to determine which
endpoint offers the best latency. This information is then used to
update which endpoint we should be using based on the best possible
route. If the latency is the same for a IPv4 and an IPv6 address, IPv6
will be used.
Updates #7826
Signed-off-by: Charlotte Brandhorst-Satzkorn <charlotte@tailscale.com>
DERP doesn't support HTTP/2. If an HTTP/2 proxy was placed in front of
a DERP server requests would fail because the connection would
be initialized with HTTP/2, which the DERP client doesn't support.
Signed-off-by: Kyle Carberry <kyle@carberry.com>
This change adds a v6conn to the pinger to enable sending pings to v6
addrs.
Updates #7826
Signed-off-by: Charlotte Brandhorst-Satzkorn <charlotte@tailscale.com>
We need to always specify tags when creating an AuthKey from an OAuth key.
Check for that, and reuse the `--advertise-tags` param.
Updates #7982
Signed-off-by: Maisem Ali <maisem@tailscale.com>
On some platforms (notably macOS and iOS) we look up the default
interface to bind outgoing connections to. This is both duplicated
work and results in logspam when the default interface is not available
(i.e. when a phone has no connectivity, we log an error and thus cause
more things that we will try to upload and fail).
Fixed by passing around a netmon.Monitor to more places, so that we can
use its cached interface state.
Fixes#7850
Updates #7621
Signed-off-by: Mihai Parparita <mihai@tailscale.com>
We're using it in more and more places, and it's not really specific to
our use of Wireguard (and does more just link/interface monitoring).
Also removes the separate interface we had for it in sockstats -- it's
a small enough package (we already pull in all of its dependencies
via other paths) that it's not worth the extra complexity.
Updates #7621
Updates #7850
Signed-off-by: Mihai Parparita <mihai@tailscale.com>
This is a follow-up to #7905 that adds two more linters and fixes the corresponding findings. As per the previous PR, this only flags things that are "obviously" wrong, and fixes the issues found.
Signed-off-by: Andrew Dunham <andrew@du.nham.ca>
Change-Id: I8739bdb7bc4f75666a7385a7a26d56ec13741b7c
This adds an initial and intentionally minimal configuration for
golang-ci, fixes the issues reported, and adds a GitHub Action to check
new pull requests against this linter configuration.
Signed-off-by: Andrew Dunham <andrew@du.nham.ca>
Change-Id: I8f38fbc315836a19a094d0d3e986758b9313f163
Redoes the approach from #5550 and #7539 to explicitly pass in the logf
function, instead of having global state that can be overridden.
Signed-off-by: Mihai Parparita <mihai@tailscale.com>
This also adds a bunch of tests for this function to ensure that we're
returning the proper IP(s) in all cases.
Signed-off-by: Andrew Dunham <andrew@du.nham.ca>
Change-Id: I0d9d57170dbab5f2bf07abdf78ecd17e0e635399
This splits Prometheus metric handlers exposed by tsweb into two
modules:
- `varz.Handler` exposes Prometheus metrics generated by our expvar
converter;
- `promvarz.Handler` combines our expvar-converted metrics and native
Prometheus metrics.
By default, tsweb will use the promvarz handler, however users can keep
using only the expvar converter. Specifically, `tailscaled` now uses
`varz.Handler` explicitly, which avoids a dependency on the
(heavyweight) Prometheus client.
Updates https://github.com/tailscale/corp/issues/10205
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
This provides an example of using native Prometheus metrics with tsweb.
Prober library seems to be the only user of PrometheusVar, so I am
removing support for it in tsweb.
Updates https://github.com/tailscale/corp/issues/10205
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
The handler will expose built-in process and Go metrics by default,
which currently duplicate some of the expvar-proxied metrics
(`goroutines` vs `go_goroutines`, `memstats` vs `go_memstats`), but as
long as their names are different, Prometheus server will just scrape
both.
This will change /debug/varz behaviour for most tsweb binaries, but
notably not for control, which configures a `tsweb.VarzHandler`
[explicitly](a5b5d5167f/cmd/tailcontrol/tailcontrol.go (L779))
Updates https://github.com/tailscale/corp/issues/10205
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
Make developing derp easier by:
1. Creating an envknob telling clients to use HTTP to connect to derp
servers, so devs don't have to acquire a valid TLS cert.
2. Creating an envknob telling clients which derp server to connect
to, so devs don't have to edit the ACLs in the admin console to add a
custom DERP map.
3. Explaining how the -dev and -a command lines args to derper
interact.
To use this:
1. Run derper with -dev.
2. Run tailscaled with TS_DEBUG_USE_DERP_HTTP=1 and
TS_DEBUG_USE_DERP_ADDR=localhost
This will result in the client connecting to derp via HTTP on port
3340.
Fixes#7700
Signed-off-by: Val <valerie@tailscale.com>
I realized that a lot of the problems that we're seeing around migration and
LocalBackend state can be avoided if we drive Windows pref migration entirely
from within tailscaled. By doing it this way, tailscaled can automatically
perform the migration as soon as the connection with the client frontend is
established.
Since tailscaled is already running as LocalSystem, it already has access to
the user's local AppData directory. The profile manager already knows which
user is connected, so we simply need to resolve the user's prefs file and read
it from there.
Of course, to properly migrate this information we need to also check system
policies. I moved a bunch of policy resolution code out of the GUI and into
a new package in util/winutil/policy.
Updates #7626
Signed-off-by: Aaron Klotz <aaron@tailscale.com>
This adds the util/sysresources package, which currently only contains a
function to return the total memory size of the current system.
Then, we modify magicsock to scale the number of buffered DERP messages
based on the system's available memory, ensuring that we never use a
value lower than the previous constant of 32.
Signed-off-by: Andrew Dunham <andrew@du.nham.ca>
Change-Id: Ib763c877de4d0d4ee88869078e7d512f6a3a148d
#7339 changed the root directory logic to find the ancestor of the cwd
with a go.mod file. This works when running the the binary from this
repo directly, but breaks when we're a dependency in another repo.
Allow the directory to be passed in via a -rootdir flag (the repo that
depends on it can then use `go list -m -f '{{.Dir}}' tailscale.com`
or similar to pass in the value).
Updates tailscale/corp#10165
Signed-off-by: Mihai Parparita <mihai@tailscale.com>
When running a SOCKS or HTTP proxy, configure the tshttpproxy package to
drop those addresses from any HTTP_PROXY or HTTPS_PROXY environment
variables.
Fixes#7407
Signed-off-by: Andrew Dunham <andrew@du.nham.ca>
Change-Id: I6cd7cad7a609c639780484bad521c7514841764b
This adds support to make exit nodes and subnet routers work
when in scenarios where NAT is required.
It also updates the NATConfig to be generated from a `wgcfg.Config` as
that handles merging prefs with the netmap, so it has the required information
about whether an exit node is already configured and whether routes are accepted.
Updates tailscale/corp#8020
Signed-off-by: Maisem Ali <maisem@tailscale.com>
Since users can run tailscaled in a variety of ways (root, non-root,
non-root with process capabilities on Linux), this check will print the
current process permissions to the log to aid in debugging.
Signed-off-by: Andrew Dunham <andrew@du.nham.ca>
Change-Id: Ida93a206123f98271a0c664775d0baba98b330c7
In addition to checking the total hostname length, validate characters used in each DNS label and label length.
Updates https://github.com/tailscale/corp/issues/10012
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
* wgengine/magicsock: add envknob to send CallMeMaybe to non-existent peer
For testing older client version responses to the PeerGone packet format change.
Updates #4326
Signed-off-by: Val <valerie@tailscale.com>
* derp: remove dead sclient struct member replaceLimiter
Leftover from an previous solution to the duplicate client problem.
Updates #2751
Signed-off-by: Val <valerie@tailscale.com>
* derp, derp/derphttp, wgengine/magicsock: add new PeerGone message type Not Here
Extend the PeerGone message type by adding a reason byte. Send a
PeerGone "Not Here" message when an endpoint sends a disco message to
a peer that this server has no record of.
Fixes#4326
Signed-off-by: Val <valerie@tailscale.com>
---------
Signed-off-by: Val <valerie@tailscale.com>
We were checking against the wrong directory, instead if we
have a custom store configured just use that.
Fixes#7588Fixes#7665
Signed-off-by: Maisem Ali <maisem@tailscale.com>
Kubernetes uses SPDY/3.1 which is incompatible with HTTP/2, disable it
in the transport and server.
Fixes#7645Fixes#7646
Signed-off-by: Maisem Ali <maisem@tailscale.com>
This change focuses on the backend log ID, which is the mostly commonly
used in the client. Tests which don't seem to make use of the log ID
just use the zero value.
Signed-off-by: Will Norris <will@tailscale.com>
This allows disabling spread mode, which is helpful if you are manually
running derpprobe in `--once` mode against a small number of DERP
machines.
Updates https://github.com/tailscale/corp/issues/9916
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
We were not handling tags at all, pass them through as Impersonate-Group headers.
And use the FQDN for tagged nodes as Impersonate-User.
Updates #5055
Signed-off-by: Maisem Ali <maisem@tailscale.com>
This lets a tsnet binary share a server out over Tailscale Funnel.
Signed-off-by: David Crawshaw <crawshaw@tailscale.com>
Signed-off-by: Maisem Ali <maisem@tailscale.com>
Signed-off-by: Shayne Sweeney <shayne@tailscale.com>
This reverts commit 6eca47b16c and fixes forward.
Previously the first ever streaming MapRequest that a client sent would also
set ReadOnly to true as it didn't have any endpoints and expected/relied on the
map poll to restart as soon as it got endpoints. However with 48f6c1eba4,
we would no longer restart MapRequests as frequently as we used to, so control
would only ever get the first streaming MapRequest which had ReadOnly=true.
Control would treat this as an uninteresting request and would not send it
any further netmaps, while the client would happily stay in the map poll forever
while litemap updates happened in parallel.
This makes it so that we never set `ReadOnly=true` when we are doing a streaming
MapRequest. This is no longer necessary either as most endpoint discovery happens
over disco anyway.
Co-authored-by: Andrew Dunham <andrew@du.nham.ca>
Signed-off-by: Maisem Ali <maisem@tailscale.com>
This change adds a ringbuffer to each magicsock endpoint that keeps a
fixed set of "changes"–debug information about what updates have been
made to that endpoint.
Additionally, this adds a LocalAPI endpoint and associated
"debug peer-status" CLI subcommand to fetch the set of changes for a given
IP or hostname.
Updates tailscale/corp#9364
Signed-off-by: Andrew Dunham <andrew@du.nham.ca>
Change-Id: I34f726a71bddd0dfa36ec05ebafffb24f6e0516a
Add a DNS server which always responds as its own IP addresses.
Additionally add a tsnet TailscaleIPs() function to return the
IP addresses, both IPv4 and IPv6.
Updates https://github.com/tailscale/tailscale/issues/1748
Signed-off-by: Denton Gentry <dgentry@tailscale.com>
No ListenPacket support yet, but Listen with a udp network type fit
easier into netstack's model to start.
Then added an example of using it to cmd/sniproxy with a little udp
:53 handler.
No tests in tsnet yet because we don't have support for dialing over
UDP in tsnet yet. When that's done, a new test can test both sides.
Updates #5871
Updates #1748
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
We have many function pointers that we replace for the duration of test and
restore it on test completion, add method to do that.
Signed-off-by: Maisem Ali <maisem@tailscale.com>
Now that we're using rand.Shuffle in a few locations, create a generic
shuffle function and use it instead. While we're at it, move the
interleaveSlices function to the same package for use.
Signed-off-by: Andrew Dunham <andrew@du.nham.ca>
Change-Id: I0b00920e5b3eea846b6cedc30bd34d978a049fd3
The debug flag on tailscaled isn't available in the macOS App Store
build, since we don't have a tailscaled binary; move it to the
'tailscale debug' CLI that is available on all platforms instead,
accessed over LocalAPI.
Updates #7377
Signed-off-by: Andrew Dunham <andrew@du.nham.ca>
Change-Id: I47bffe4461e036fab577c2e51e173f4003592ff7
Followup to #7177 to avoid adding extra dependencies to the CLI. We
instead declare an interface for the link monitor.
Signed-off-by: Mihai Parparita <mihai@tailscale.com>
This is to address a possible DNS failure on startup. Before this
change IPv6 addresses would be listed first, and the client dialer would
fail for hosts without IPv6 connectivity.
We had two implemenetations of the kube client, merge them.
containerboot was also using a raw http.Transport, this also has
the side effect of making it use a http.Client
Signed-off-by: Maisem Ali <maisem@tailscale.com>
Given recent changes in corp, I originally thought we could remove all of the
syso files, but then I realized that we still need them so that binaries built
purely from OSS (without going through corp) will still receive a manifest.
We can remove the arm32 one though, since we don't support 32-bit ARM on Windows.
Updates https://github.com/tailscale/corp/issues/9576
Signed-off-by: Aaron Klotz <aaron@tailscale.com>
"Device Authorization" was recently renamed to "Device Approval"
on the control side. This change updates the k8s operator to match.
Signed-off-by: Sonia Appasamy <sonia@tailscale.com>
"Device Authorization" was recently renamed to "Device Approval"
on the control side. This change updates tsconnect to match.
Signed-off-by: Sonia Appasamy <sonia@tailscale.com>
Uses the hooks added by tailscale/go#45 to instrument the reads and
writes on the major code paths that do network I/O in the client. The
convention is to use "<package>.<type>:<label>" as the annotation for
the responsible code path.
Enabled on iOS, macOS and Android only, since mobile platforms are the
ones we're most interested in, and we are less sensitive to any
throughput degradation due to the per-I/O callback overhead (macOS is
also enabled for ease of testing during development).
For now just exposed as counters on a /v0/sockstats PeerAPI endpoint.
We also keep track of the current interface so that we can break out
the stats by interface.
Updates tailscale/corp#9230
Updates #3363
Signed-off-by: Mihai Parparita <mihai@tailscale.com>
"Device Authorization" was recently renamed to "Device Approval"
on the control side. This change updates the linux cli to match.
Signed-off-by: Sonia Appasamy <sonia@tailscale.com>
The log ID types were moved to a separate package so that
code that only depend on log ID types do not need to link
in the logic for the logtail client itself.
Not all code need the logtail client.
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
This allows us to differentiate between the various tsnet apps that
we have like `golinks` and `k8s-operator`.
Signed-off-by: Maisem Ali <maisem@tailscale.com>
trimmed builds don't have absolute path information in executable
metadata, which leads the runtime.Caller approach failing
mysteriously in yarn with complaints about relative package paths.
So, instead of using embedded package metadata to find paths,
expect that we're being invoked within the tailscale repo, and
locate the tsconnect directory that way.
Signed-off-by: David Anderson <danderson@tailscale.com>
This ensures that we put the kubeconfig in the correct directory from within the macOS Sandbox when
paired with tailscale/corp@3035ef7
Updates #7220
Signed-off-by: Maisem Ali <maisem@tailscale.com>
In the switch to static toolchains, we removed a legacy oddity from the
toolchain URL structure, but forgot to update printdep.
Signed-off-by: David Anderson <danderson@tailscale.com>
With #6566 we added an external mechanism for getting the default
interface, and used it on macOS and iOS (see tailscale/corp#8201).
The goal was to be able to get the default physical interface even when
using an exit node (in which case the routing table would say that the
Tailscale utun* interface is the default).
However, the external mechanism turns out to be unreliable in some
cases, e.g. when multiple cellular interfaces are present/toggled (I
have occasionally gotten my phone into a state where it reports the pdp_ip1
interface as the default, even though it can't actually route traffic).
It was observed that `ifconfig -v` on macOS reports an "effective interface"
for the Tailscale utn* interface, which seems promising. By examining
the ifconfig source code, it turns out that this is done via a
SIOCGIFDELEGATE ioctl syscall. Though this is a private API, it appears
to have been around for a long time (e.g. it's in the 10.13 xnu release
at https://opensource.apple.com/source/xnu/xnu-4570.41.2/bsd/net/if_types.h.auto.html)
and thus is unlikely to go away.
We can thus use this ioctl if the routing table says that a utun*
interface is the default, and go back to the simpler mechanism that
we had before #6566.
Updates #7184
Updates #7188
Signed-off-by: Mihai Parparita <mihai@tailscale.com>
Tailnet-owned auth keys (which all OAuth-created keys are) must include tags, since there is no user to own the registered devices.
Signed-off-by: Will Norris <will@tailscale.com>
Also removes the toolchain builds from flake.nix. For now the flake
build uses upstream Go 1.20, a followup change will switch it back to
our custom toolchain.
Updates tailscale/corp#9005
Signed-off-by: David Anderson <danderson@tailscale.com>
When we make a connection to a server, we previously would verify with
the system roots, and then fall back to verifying with our baked-in
Let's Encrypt root if the system root cert verification failed.
We now explicitly check for, and log a health error on, self-signed
certificates. Additionally, we now always verify against our baked-in
Let's Encrypt root certificate and log an error if that isn't
successful. We don't consider this a health failure, since if we ever
change our server certificate issuer in the future older non-updated
versions of Tailscale will no longer be healthy despite being able to
connect.
Updates #3198
Change-Id: I00be5ceb8afee544ee795e3c7a2815476abc4abf
Signed-off-by: Andrew Dunham <andrew@du.nham.ca>
It's since been rewritten in Swift.
#cleanup
Change-Id: I0860d681e8728697804ce565f63c5613b8b1088c
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
It includes xtermjs/xterm.js#4216, which improves handling of some
escape sequences. Unfortunately it's not enough to fix the issue
with `ponysay`, but it does not hurt to be up to date.
Updates #6090
Signed-off-by: Mihai Parparita <mihai@tailscale.com>
`prober.DERP` was created in #5988 based on derpprobe. Having used it
instead of derpprobe for a few months, I think we have enough confidence
that it works and can now migrate derpprobe to use the prober framework
and get rid of code duplication.
A few notable changes in behaviour:
- results of STUN probes over IPv4 and IPv6 are now reported separately;
- TLS probing now includes OCSP verification;
- probe names in the output have changed;
- ability to send Slack notification from the prober has been removed.
Instead, the prober now exports metrics in Expvar (/debug/vars) and
Prometheus (/debug/varz) formats.
Fixes https://github.com/tailscale/corp/issues/8497
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
Maisem Ali