diff --git a/docs/k8s/README.md b/docs/k8s/README.md index cdade968b..a37f4b267 100644 --- a/docs/k8s/README.md +++ b/docs/k8s/README.md @@ -1,6 +1,10 @@ # Overview -There are quite a few ways of running Tailscale inside a Kubernetes Cluster, some of the common ones are covered in this doc. +There are quite a few ways of running Tailscale inside a Kubernetes Cluster. +This doc covers creating and managing your own Tailscale node deployments in cluster. +If you want a higher level of automation, easier configuration, automated cleanup of stopped Tailscale devices, or a mechanism for exposing the [Kubernetes API](https://kubernetes.io/docs/concepts/overview/kubernetes-api/) server to the tailnet, take a look at [Tailscale Kubernetes operator](https://tailscale.com/kb/1236/kubernetes-operator). + +:warning: Note that the manifests generated by the following commands are not intended for production use, and you will need to tweak them based on your environment and use case. For example, the commands to generate a standalone proxy manifest, will create a standalone `Pod`- this will not persist across cluster upgrades etc. :warning: ## Instructions @@ -153,3 +157,74 @@ the entire Kubernetes cluster network (assuming NetworkPolicies allow) over Tail INTERNAL_PORT=8080 curl http://$INTERNAL_IP:$INTERNAL_PORT ``` + +## Multiple replicas + +Note that if you want to use the `Pod` manifests generated by the commands above in a multi-replica setup (i.e a multi-replica `StatefulSet`) you will need to change the mechanism for storing tailscale state to ensure that multiple replicas are not attemting to use a single Kubernetes `Secret` to store their individual states. + +To avoid proxy state clashes you could either store the state in memory or an `emptyDir` volume, or you could change the provided state `Secret` name to ensure that a unique name gets generated for each replica. + +### Option 1: storing in an `emptyDir` + +You can mount an [`emptyDir` volume](https://kubernetes.io/docs/concepts/storage/volumes/#emptydir) and configure the mount as the tailscale state store via `TS_STATE_DIR` env var. +You must also set `TS_KUBE_SECRET` to an empty string. + +An example: + +```yaml +kind: StatefulSet +metadata: + name: subnetrouter +spec: + replicas: 2 + ... + template: + ... + spec: + ... + volumes: + - name: tsstate + emptyDir: {} + containers: + - name: tailscale + env: + - name: TS_STATE_DIR + value: /tsstate + - name: TS_KUBE_SECRET + value: "" + volumeMounts: + - name: tsstate + mountPath: /tsstate +``` + +The downside of this approach is that the state will be lost when a `Pod` is +deleted. In practice this means that when you, for example, upgrade proxy +versions you will get a new set of Tailscale devices with different hostnames. + +### Option 2: dynamically generating unique `Secret` names + +If you run the proxy as a `StatefulSet`, the `Pod`s get [stable identifiers](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#stable-network-id). +You can use that to pass an individual, static state `Secret` name to each proxy: + +```yaml +kind: StatefulSet +metadata: + name: subnetrouter +spec: + replicas: 2 + ... + template: + ... + spec: + ... + containers: + - name: tailscale + env: + - name: TS_KUBE_SECRET + valueFrom: + fieldRef: + apiVersion: v1 + fieldPath: metadata.name +``` + +In this case, each replica will store its state in a `Secret` named the same as the `Pod` and as `Pod` names for a `StatefulSet` do not change if `Pod`s get recreated, proxy state will persist across cluster and proxy version updates etc. diff --git a/docs/k8s/proxy.yaml b/docs/k8s/proxy.yaml index bff4efd6f..2ab7ed334 100644 --- a/docs/k8s/proxy.yaml +++ b/docs/k8s/proxy.yaml @@ -32,6 +32,8 @@ spec: value: "{{TS_KUBE_SECRET}}" - name: TS_USERSPACE value: "false" + - name: TS_DEBUG_FIREWALL_MODE + value: auto - name: TS_AUTHKEY valueFrom: secretKeyRef: diff --git a/docs/k8s/sidecar.yaml b/docs/k8s/sidecar.yaml index e35d78e3e..7efd32a38 100644 --- a/docs/k8s/sidecar.yaml +++ b/docs/k8s/sidecar.yaml @@ -18,6 +18,8 @@ spec: value: "{{TS_KUBE_SECRET}}" - name: TS_USERSPACE value: "false" + - name: TS_DEBUG_FIREWALL_MODE + value: auto - name: TS_AUTHKEY valueFrom: secretKeyRef: