diff --git a/docs/k8s/operator-architecture.md b/docs/k8s/operator-architecture.md new file mode 100644 index 000000000..9df2a048c --- /dev/null +++ b/docs/k8s/operator-architecture.md @@ -0,0 +1,254 @@ +# Operator architecture diagrams + +The Tailscale Kubernetes operator has a collection of use-cases that can be +mixed and matched as required. The following diagrams illustrate how the +operator implements each use-case. + +In each diagram, the "tailscale" namespace is entirely managed by the operator +once the operator itself has been deployed. + +Tailscale devices are highlighted as black nodes. The salient devices for each +use-case are marked as "src" or "dst" to denote which node is a source or a +destination in the context of ACL rules that will apply to network traffic. + +## API server proxy + +[Documentation][kb-operator-proxy] + +The operator runs the API server proxy in-process. If the proxy is running in +"noauth" mode, it forwards HTTP requests unmodified. If the proxy is running in +"auth" mode, it deletes any existing auth headers and adds impersonation +headers to the request before forwarding to the API server. + +```mermaid +%%{ init: { 'theme':'neutral' } }%% +flowchart LR + classDef tsnode color:#fff,fill:#000; + classDef pod fill:#fff; + + subgraph Key + ts[Tailscale device]:::tsnode + end + + subgraph k8s[Kubernetes cluster] + subgraph tailscale-ns[tailscale] + operator["operator (dst)"]:::tsnode + end + + subgraph controlplane["Control plane"] + api[kube-apiserver] + end + end + client["client (src)"]:::tsnode --> operator + operator-->|proxies requests| api + +``` + +## L3 ingress + +[Documentation][kb-operator-l3-ingress] + +The user deploys an app to the default namespace, and creates a normal Service +that selects the app's pods. Add the annotation `tailscale.com/expose: "true"` +to the Service, and the operator will create an ingress proxy that allows +devices anywhere on the tailnet to access the Service. + +```mermaid +%%{ init: { 'theme':'neutral' } }%% +flowchart TD + classDef tsnode color:#fff,fill:#000; + classDef pod fill:#fff; + + subgraph Key + ts[Tailscale device]:::tsnode + pod(Pod):::pod + end + + subgraph k8s[Kubernetes cluster] + subgraph tailscale-ns[tailscale] + operator(operator):::tsnode + ingress("ingress proxy (dst)"):::tsnode + secret + end + + subgraph defaultns[default] + svc[annotated svc] + svc --> app1(app) + svc --> app2(app) + end + end + client["client (src)"]:::tsnode --> ingress + ingress -->|forwards traffic| svc + operator -.->|deploys| ingress + operator -.->|reads| svc + operator -.->|creates| secret + secret -.->|mounted| ingress + +``` + +## L7 ingress + +[Documentation][kb-operator-l7-ingress] + +## L3 egress + +[Documentation][kb-operator-l3-egress] + +1. The user deploys a Service named `db` with `type: ExternalName` and an annotation + `tailscale.com/tailnet-fqdn: db.tails-scales.ts.net`. +1. The operator creates a proxy Pod managed by a single replica StatefulSet, and a headless Service pointing at the proxy Pod. +1. The operator updates the `db` Service's `spec.externalName` field to point + at the headless Service it created in the previous step. + +(Optional) If the user also adds the `tailscale.com/proxy-group: egress-proxies` +annotation to their `db` Service, the operator will skip creating a proxy Pod and +instead point the headless Service at the existing ProxyGroup's pods. In this +case, ports are also required in the `db` Service spec. + +Note, in some cases, the config and the state Secret may be the same Kubernetes Secret. + +```mermaid +%%{ init: { 'theme':'neutral' } }%% + +flowchart TD + classDef tsnode color:#fff,fill:#000; + classDef pod fill:#fff; + + subgraph Key + ts[Tailscale device]:::tsnode + pod(Pod):::pod + end + + subgraph k8s[Kubernetes cluster] + subgraph tailscale-ns[tailscale] + operator(operator):::tsnode + egress("egress proxy (src)"):::tsnode + headless-svc[headless svc] + cfg-secret["config Secret"] + state-secret["state Secret"] + end + + subgraph defaultns[default] + svc[db ExternalName svc] + app1(app) --> svc + app2(app) --> svc + end + end + node["db.tails-scales.ts.net (dst)"]:::tsnode + svc -->|DNS points to| headless-svc + headless-svc -->|forwards traffic| egress + egress -->|forwards traffic| node + operator -.->|deploys| egress + operator -.->|deploys| headless-svc + operator -.->|creates| cfg-secret + operator -.->|watches & updates| svc + cfg-secret -.->|mounted| egress + egress -.->|stores state| state-secret + +``` + +## `ProxyGroup` + +[Documentation][kb-operator-l3-egress-proxygroup] + +The `ProxyGroup` custom resource manages a collection of proxy Pods that can be +configured to egress traffic out of the cluster via ExternalName Services defined +elsewhere. They will also support ingress in the future. In this diagram, the +`ProxyGroup` is named `pg`, and the operator creates proxy pods, via a StatefulSet +but they don't yet serve any traffic. + +`ProxyGroups` currently only support egress (see above). + +```mermaid +%%{ init: { 'theme':'neutral' } }%% + +flowchart TD + classDef tsnode color:#fff,fill:#000; + classDef pod fill:#fff; + + subgraph Key + ts[Tailscale device]:::tsnode + pod(Pod):::pod + end + + subgraph k8s[Kubernetes cluster] + subgraph tailscale-ns[tailscale] + operator(operator):::tsnode + pg-sts[pg StatefulSet] + pg-0("pg-0 (src)"):::tsnode + pg-1("pg-1 (src)"):::tsnode + cfg-secret-0["pg-0-config Secret"] + cfg-secret-1["pg-1-config Secret"] + state-secret-0["pg-0 Secret"] + state-secret-1["pg-1 Secret"] + end + + subgraph cluster-scope["Cluster scoped resources"] + pg["pg ProxyGroup"] + end + end + operator-.->|watches| pg + operator -.->|deploys| pg-sts + pg-sts -.->|manages| pg-0 + pg-sts -.->|manages| pg-1 + operator -.->|creates| cfg-secret-0 + operator -.->|creates| cfg-secret-1 + cfg-secret-0 -.->|mounted| pg-0 + cfg-secret-1 -.->|mounted| pg-1 + pg-0 -.->|stores state| state-secret-0 + pg-1 -.->|stores state| state-secret-1 + +``` + +## Subnet routers and exit nodes + +[Documentation][kb-operator-connector] + +## Recorder nodes + +[Documentation][kb-operator-recorder] + +The `Recorder` custom resource makes it easier to deploy `tsrecorder` to a cluster. +It currently only supports a single replica. + +```mermaid +%%{ init: { 'theme':'neutral' } }%% + +flowchart TD + classDef tsnode color:#fff,fill:#000; + classDef pod fill:#fff; + + subgraph Key + ts[Tailscale device]:::tsnode + pod(Pod):::pod + end + + subgraph k8s[Kubernetes cluster] + subgraph tailscale-ns[tailscale] + operator(operator):::tsnode + rec-sts[rec StatefulSet] + rec-0("rec-0 Pod (tsrecorder)"):::tsnode + cfg-secret-0["rec-0-config Secret"] + state-secret-0["rec-0 Secret"] + end + + subgraph cluster-scope["Cluster scoped resources"] + rec["rec Recorder"] + end + end + operator-.->|watches| rec + operator -.->|deploys| rec-sts + rec-sts -.->|manages| rec-0 + operator -.->|creates| cfg-secret-0 + cfg-secret-0 -.->|mounted| rec-0 + rec-0 -.->|stores state| state-secret-0 + +``` + +[kb-operator-proxy]: https://tailscale.com/kb/1437/kubernetes-operator-api-server-proxy +[kb-operator-l3-ingress]: https://tailscale.com/kb/1439/kubernetes-operator-cluster-ingress#exposing-a-cluster-workload-using-a-kubernetes-service +[kb-operator-l7-ingress]: https://tailscale.com/kb/1439/kubernetes-operator-cluster-ingress#exposing-cluster-workloads-using-a-kubernetes-ingress +[kb-operator-l3-egress]: https://tailscale.com/kb/1438/kubernetes-operator-cluster-egress +[kb-operator-l3-egress-proxygroup]: TODO +[kb-operator-connector]: https://tailscale.com/kb/1441/kubernetes-operator-connector +[kb-operator-recorder]: TODO