[docs] Teleport Kubernetes Access (#4961)

Co-authored-by: Gus Luxton <gus@gravitational.com>
Co-authored-by: Andrew Lytvynov <andrew@goteleport.com>
This commit is contained in:
Ben Arent 2020-12-01 21:29:50 -08:00 committed by GitHub
parent 9c26188d30
commit 58e1be5eb5
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
10 changed files with 257 additions and 127 deletions

View file

@ -715,6 +715,7 @@ Create an identity file(s) for a given user
`--ttl` | none | relative duration like 5s, 2m, or 3h | TTL (time to live) for the generated certificate
`--compat` | `""` | `standard` or `oldssh` | OpenSSH compatibility flag
`--proxy` | `""` | Address of the teleport proxy. | When --format is set to "kubernetes", this address will be set as cluster address in the generated kubeconfig file
`--kube-cluster` | `""` | The name of a leaf cluster. | When --format is set to "kubernetes", the kubeconfig generated will allow access to this leaf cluster.
### [Global Flags](#tctl-global-flags)

View file

@ -46,7 +46,7 @@ nav:
- Quick Start Guide: quickstart.md
- User Manual: user-manual.md
- Application Guide: application-access.md
- Kubernetes Guide: kubernetes-ssh.md
- Kubernetes Guide: kubernetes-access.md
- Admin Manual: admin-guide.md
- Production Guide: production.md
- FAQ: faq.md

View file

@ -106,7 +106,9 @@ numbers.
|3024 | Proxy | SSH port used to create "reverse SSH tunnels" from behind-firewall environments into a trusted proxy server.
|3025 | Auth | SSH port used by the Auth Service to serve its API to other nodes in a cluster.
|3080 | Proxy | HTTPS connection to authenticate `tsh` users and web users into the cluster. The same connection is used to serve a Web UI.
|3026 | Kubernetes Proxy | HTTPS Kubernetes proxy (if enabled)
|3026 | Kubernetes | HTTPS Kubernetes proxy `proxy_service.kube_listen_addr`
|3027 | Kubernetes | Kubernetes Service `kubernetes_service.listen_addr`
### Filesystem Layout
@ -236,6 +238,9 @@ proxy_service:
web_listen_addr: 0.0.0.0:3080
tunnel_listen_addr: 0.0.0.0:3024
# Expose a k8s listening port on the proxy if using Kubernetes
kube_listen_addr: 0.0.0.0:3026
# The DNS name of the proxy HTTPS endpoint as accessible by cluster users.
# Defaults to the proxy's hostname if not specified. If running multiple
# proxies behind a load balancer, this name must point to the load balancer
@ -1092,7 +1097,7 @@ To learn more about Trusted Clusters please visit our [Trusted Cluster Guide](tr
Teleport supports authentication and authorization via external identity
providers such as Github. You can watch the video for how to configure
[Github as an SSO provider](https://gravitational.com/resources/guides/github-sso-provider-kubernetes-ssh/),
[Github as an SSO provider](https://goteleport.com/resources/guides/github-sso-provider-kubernetes-ssh/),
or you can follow the documentation below.
First, the Teleport auth service must be configured to use Github for
@ -1277,10 +1282,10 @@ scp_if_ssh = True
Teleport can be configured as a compliance gateway for Kubernetes clusters.
This allows users to authenticate against a Teleport proxy using [`tsh
login`](cli-docs.md#tsh) command to retrieve credentials for both SSH and
login`](cli-docs.md#tsh) and [`tsh kube login`](cli-docs.md#tsh-kube-login) command to retrieve credentials for both SSH and
Kubernetes API.
Follow our [Kubernetes guide](kubernetes-ssh.md) which contains some more specific
Follow our [Kubernetes guide](kubernetes-access.md) which contains some more specific
examples and instructions.
## High Availability

View file

@ -61,7 +61,7 @@ This should result in three PEM encoded files being generated in the `/certs` di
Move the `/certs` folder into your `/api-examples` folder.
!!! Note
By default, `tctl auth sign` produces certificates with a relatively short lifetime. See our [Kubernetes Section](kubernetes-ssh.md#using-teleport-kubernetes-with-automation) for more information on automating the signing process for short lived certificates.
By default, `tctl auth sign` produces certificates with a relatively short lifetime. See our [Kubernetes Section](kubernetes-access.md#using-teleport-kubernetes-with-automation) for more information on automating the signing process for short lived certificates.
While we encourage you to use short lived certificates, we understand you may not have all the infrastructure to issues and obtain them at the onset. You can use the --ttl flag to extend the lifetime of a certificate in these cases but understand this reduces your security posture

View file

@ -331,6 +331,22 @@ Node Name Address Labels
grav-02 10.156.0.7:3022 os:osx
```
## tsh kube ls
List Kubernetes Clusters
**Usage** `usage: tsh kube ls`
### Examples
``` bsh
$ tsh kube ls
Kube Cluster Name Selected
------------------------------------- --------
gke_bens-demos_us-central1-c_gks-demo *
microk8s
```
## tsh clusters
**Usage**: `tsh clusters [<flags>]`
@ -430,6 +446,28 @@ $ tsh login --proxy=proxy.example.com --format=kubernetes -o kubeconfig
$ tsh login --proxy=proxy.example.com --request-reason="I need to run a debug script on production"
```
## tsh kube login
Log into a Kubernetes cluster. Discover connected clusters by using [`tsh kube ls`](cli-docs.md#tsh-kube-ls)
**Usage**: `tsh kube login <kube-cluster>`
```bash
# tsh kube login to k8s cluster (gke_bens-demos_us-central1-c_gks-demo)
$ tsh kube login gke_bens-demos_us-central1-c_gks-demo
Logged into kubernetes cluster "gke_bens-demos_us-central1-c_gks-demo"
# on login, kubeconfig is pointed at the first cluster (alphabetically)
$ kubectl config current-context
aws-gke_bens-demos_us-central1-c_gks-demo
# but all clusters are populated as contexts
$ kubectl config get-contexts
CURRENT NAME CLUSTER AUTHINFO NAMESPACE
* aws-gke_bens-demos_us-central1-c_gks-demo aws aws-gke_bens-demos_us-central1-c_gks-demo
aws-microk8s aws aws-microk8s
```
## tsh logout
Deletes the client's cluster certificate
@ -448,11 +486,15 @@ Display the list of proxy servers and retrieved certificates
$ tsh status
> Profile URL: https://proxy.example.com:3080
Logged in as: johndoe
Roles: admin*
Logins: root, admin, guest
Valid until: 2017-04-25 15:02:30 -0700 PDT [valid for 1h0m0s]
Extensions: permit-agent-forwarding, permit-port-forwarding, permit-pty
Logged in as: benarent
Cluster: aws
Roles: admin*
Logins: benarent, root, ec2-user, ubunutu
Kubernetes: enabled
Kubernetes cluster: "gke_bens-demos_us-central1-c_gks-demo"
Kubernetes groups: system:masters
Valid until: 2020-11-21 01:50:23 -0800 PST [valid for 11h52m0s]
Extensions: permit-agent-forwarding, permit-port-forwarding, permit-pty
```
# tctl
@ -665,7 +707,11 @@ $ tctl tokens add --type=trusted_cluster --ttl=5m
$ tctl tokens add --type=trusted_cluster --labels=env=prod
# Generate an invite token for a node
# This is equivalent to `tctl nodes add`
$ tctl tokens add --type node
$ tctl tokens add --type=node
# Generate an invite token for a kubernetes_service
$ tctl tokens add --type=kube
# Generate an invite token for an app_service
$ tctl tokens add --type=app
```
## tctl tokens rm
@ -746,6 +792,8 @@ Create an identity file(s) for a given user
`--ttl` | none | relative duration like 5s, 2m, or 3h | TTL (time to live) for the generated certificate
`--compat` | `""` | `standard` or `oldssh` | OpenSSH compatibility flag
`--proxy` | `""` | Address of the teleport proxy. | When --format is set to "kubernetes", this address will be set as cluster address in the generated kubeconfig file
`--leaf-cluster` | `""` | The name of a leaf cluster. |
`--kube-cluster-name` | `""` | Kubernetes Cluster Name |
### [Global Flags](#tctl-global-flags)

View file

@ -329,23 +329,7 @@ proxy_service:
- key_file: /etc/letsencrypt/live/*.teleport.example.com/privkey.pem
cert_file: /etc/letsencrypt/live/*.teleport.example.com/fullchain.pem
# This section configures the Kubernetes proxy service
kubernetes:
# Turns 'kubernetes' proxy on. Default is 'no'
enabled: yes
# Kubernetes proxy listen address.
listen_addr: 0.0.0.0:3026
# The DNS name of the Kubernetes proxy server that is accessible by cluster clients.
# If running multiple proxies behind a load balancer, this name must point to the
# load balancer.
public_addr: ['kube.example.com:3026']
# This setting is not required if the Teleport proxy service is
# deployed inside a Kubernetes cluster. Otherwise, Teleport proxy
# will use the credentials from this file:
kubeconfig_file: /path/to/kube/config
kube_listen_addr: 0.0.0.0:3026
# This section configures the 'application service'
app_service:
@ -370,4 +354,41 @@ app_service:
- name: "os"
command: ["/usr/bin/uname"]
period: "5s"
## This section configures the 'kubernetes service'
kubernetes_service:
enabled: yes
# Optional Public & Listen Addr: Set these if you are connecting to
# Teleport running inside a Kubernetes cluster instead of using a
# reverse tunnel.
#
# Optional Public Addr
public_addr: [k8s.example.com:3026]
# Optional Listen Addr
listen_addr: 0.0.0.0:3026
# Optional kubeconfig_file and kube_cluster_name. Exactly one of these must be set.
#
# When running teleport outside of the kubernetes cluster, use kubeconfig_file to provide
# teleport with cluster credentials.
#
# When running teleport inside of the kubernetes cluster pod, use kube_cluster_name to
# provide a user-visible name. Teleport uses the pod service account credentials to authenticate
# to its local kubernetes API.
kubeconfig_file: /secrets/kubeconfig
kube_cluster_name:
# Optional labels: These can be used in combination with RBAC rules
# to limit access to applications.
# When using kubeconfig_file above, these labels apply to all kubernetes
# clusters specified in the kubeconfig.
labels:
env: "prod"
# Optional Dynamic Labels
- name: "os"
command: ["/usr/bin/uname"]
period: "5s"
# Get cluster name on GKE.
- name: cluster-name
command: ['curl', 'http://metadata.google.internal/computeMetadata/v1/instance/attributes/cluster-name', '-H', 'Metadata-Flavor: Google']
period: 1m0s
```

View file

@ -133,9 +133,16 @@ spec:
# of the list example above can be expressed as:
'environment': '^test|staging$'
kubernetes_labels:
# a user can only acess prod enviroments
'env': 'prod'
# user can access any region in us-west, e.g us-west-1, us-west-2
'region': 'us-west-*'
'cluster_name': '^us.*\.example\.com$'
# defines roles that this user can can request.
# needed for teleport's request workflow
# https://gravitational.com/teleport/docs/enterprise/workflow/
# https://goteleport.com/teleport/docs/enterprise/workflow/
request:
roles:
- dba

View file

@ -3,7 +3,6 @@ title: Teleport FAQ
description: Frequently Asked Questions about using Teleport for SSH and Kubernetes access
---
# FAQ
## Community FAQ

View file

@ -3,36 +3,61 @@ title: Kubernetes Access Guide
description: How to set up and configure Teleport for Kubernetes access with SSO and RBAC
---
# Teleport Kubernetes Access Guide
# Teleport Kubernetes Access
Teleport has the ability to act as a compliance gateway for managing privileged
access to Kubernetes clusters. This enables the following capabilities:
* A Teleport Proxy can act as a single authentication endpoint for both SSH and
Kubernetes. Users can authenticate against a Teleport proxy using Teleport's
[`tsh login`](cli-docs.md#tsh-login) command
and retrieve credentials for both SSH and Kubernetes API.
* Teleport acts as a single authentication endpoint for both SSH and multiple
Kubernetes clusters. Users can authenticate against a Teleport Access Plane using
Teleport's [`tsh login`](cli-docs.md#tsh-login) command and retrieve credentials
for both SSH and Kubernetes API.
* Users RBAC roles are always synchronized between SSH and Kubernetes, making
it easier to implement policies like _developers must not access production
data_.
* Teleport's session recording and audit log extend to Kubernetes, as well.
Regular `kubectl exec` commands are logged into the audit log and the interactive
commands are recorded as regular sessions that can be stored and replayed in the
future.
* Complete `kubectl` auditing and session recordings for `kubectl exec`
* Multi Kubernetes Support. Login to Teleport once and quickly switch between multiple K8s clusters using [`tsh kube login`](cli-docs.md#tsh-kube-login)
## Start Here
Before we dive into setup, we've a few options to help guide you. We've created some example
Teleport Kubernetes Access configurations for different scenarios, all the way from solo developers
accessing minikube through to large enterprises accessing hundreds of Kubernetes clusters.
Teleport Kubernetes service requires Auth & Proxy to be setup. This can be inside or
outside of k8s.
Example Kubernetes Cluster Configurations:
* Connecting to a local MicroK8s Cluster:
* Use Option 2, once setup you'll need to connect to Ingress via TODO
* Connecting to one EKS/GKS hosted Kubernetes:
* Use Option 2, once setup you'll need to connect to Ingress via TODO
* Connecting to multiple Kubernetes clusters in one cloud/region:
* Use Option 1, once setup you'll need to connect via the root Teleport service.
* Connecting to multiple Kubernetes clusters in multiple regions/cloud:
* Use Option 1, once setup you'll need to connect via the root Teleport service.
* Use Option 1 combined with [Trusted Clusters](kubernetes-access.md#multiple-kubernetes-clusters-via-trusted-cluster)
* Migrating pre-5.0 Teleport clusters to new `kubernetes_service`:
* Please review our [Teleport Kubernetes Guide](kubernetes-5.0-migration.md)
## Teleport Kubernetes Service
## Teleport Proxy Service
By default, the Kubernetes integration is turned off in Teleport. The configuration
setting to enable the integration in the proxy service section in the `/etc/teleport.yaml`
config file, as shown below:
```yaml
# snippet from /etc/teleport.yaml on the Teleport proxy service:
proxy_service:
# create the 'kubernetes' section and set 'enabled' to 'yes':
kubernetes:
enabled: yes
public_addr: [teleport.example.com:3026]
listen_addr: 0.0.0.0:3026
kubernetes_service:
enabled: yes
public_addr: [k8s.example.com:3027]
listen_addr: 0.0.0.0:3027
kubeconfig_file: /secrets/kubeconfig
```
Let's take a closer look at the available Kubernetes settings:
@ -45,24 +70,85 @@ Let's take a closer look at the available Kubernetes settings:
- `listen_addr` defines which network interface and port the Teleport proxy server
should bind to. It defaults to port 3026 on all NICs.
### Connecting the Teleport proxy to Kubernetes
## Setup
Connecting the Teleport proxy to Kubernetes.
Teleport Auth And Proxy can be ran anywhere (inside or outside of k8s). The Teleport
proxy must have `kube_listen_addr` set.
- Options for connecting k8s clusters:
- `kubernetes_service` in a pod [Using our Helm Chart](https://github.com/gravitational/teleport/blob/master/examples/chart/teleport-kube-agent/README.md)
- `kubernetes_service` elsewhere, with kubeconfig. Use [get-kubeconfig.sh](https://github.com/gravitational/teleport/blob/master/examples/k8s-auth/) for building kubeconfigs
There are two options for setting up Teleport to access Kubernetes:
## Option 1: Deploy Inside Kubernetes as a pod
### Option 1: Standalone Teleport "gateway" for multiple K8s Clusters
A single central Teleport Access Plane acting as "gateway". Multiple Kubernetes clusters
connect to it over reverse tunnels.
The root Teleport Cluster should be setup following our standard config, to make sure
clients can connect you must make sure that an invite token is set for the `kube`
service and proxy_addr has `kube_lisetn_addr` set.
```yaml
# Example Snippet for the Teleport Root Service
#...
auth_service:
enabled: "yes"
listen_addr: 0.0.0.0:3025
tokens:
- kube:866c6c114724a0fa4d4d73216afd99fb1a2d6bfde8e13a19
#...
proxy_service:
public_addr: proxy.example.com:3080
kube_listen_addr: 0.0.0.0:3027
```
To get quickly setup, we provide a Helm chart that'll connect to the above root cluster.
```bash
# Add Teleport Helm Repo
$ helm repo add teleport https://charts.releases.teleport.dev
# Installing the Helm Chart
helm install teleport-kube-agent teleport/teleport-kube-agent \
--namespace teleport \
--create-namespace \
--set proxyAddr=proxy.example.com:3080 \
--set authToken=$JOIN_TOKEN \
--set kubeClusterName=$KUBERNETES_CLUSTER_NAME
```
| Things to set | Description |
|-|-|
| `proxyAddr` | The Address of the Teleport Root Service, using the proxy listening port |
| `authToken` | A static `kube` invite token |
| `kubeClusterName` | Kubernetes Cluster name (there is no easy way to automatically detect the name from the environment) |
### Option 2: Proxy running inside a k8s cluster.
Deploy Teleport Proxy service as a Kubernetes pod inside the Kubernetes cluster
you want the proxy to have access to.
```yaml
# snippet from /etc/teleport.yaml on the Teleport proxy service:
auth_service:
cluster_name: example.com
public_addr: auth.example.com:3025
# ..
proxy_service:
# create the 'kubernetes' section and set 'enabled' to 'yes':
kubernetes:
enabled: yes
public_addr: proxy.example.com:3080
kube_listen_addr: 0.0.0.0:3026
kubernetes_service:
enabled: yes
listen_addr: 0.0.0.0:3027
kube_cluster_name: kube.example.com
```
If you're using Helm, we've a chart that you can use. Run these commands:
If you're using Helm, we provide a chart that you can use. Run these commands:
```bash
$ helm repo add teleport https://charts.releases.teleport.dev
@ -73,81 +159,12 @@ our [Helm Docs](https://github.com/gravitational/teleport/tree/master/examples/c
![teleport-kubernetes-inside](img/teleport-k8s-pod.svg)
## Option 2: Deploy Outside of Kubernetes
Deploy the Teleport proxy service outside of Kubernetes and update the Teleport
Proxy configuration with Kubernetes credentials. Update the Teleport Proxy
configuration with Kubernetes credentials.
In this case, we need to update `/etc/teleport.yaml` for the proxy service as shown
below:
```yaml
# snippet from /etc/teleport.yaml on the Teleport proxy service:
proxy_service:
# create the 'kubernetes' section and set 'enabled' to 'yes':
kubernetes:
enabled: yes
# The address for the proxy process to accept k8s requests.
listen_addr: 0.0.0.0:3026
# The address used by the clients after tsh login. If you run a load balancer
# in front of this proxy, use the address of that balancer here. Otherwise,
# use the address of the host running this proxy.
public_addr: [teleport.example.com:3026]
kubeconfig_file: /path/to/.kube/config
```
![teleport-ssh-kubernetes-integration](img/teleport-kubernetes-outside.svg)
To generate the `kubeconfig_file` for the Teleport proxy service:
1. Configure your `kubectl` to point at the Kubernetes cluster and have admin-level access.
2. Use [this
script](https://github.com/gravitational/teleport/blob/master/examples/k8s-auth/get-kubeconfig.sh)
to generate `kubeconfig`:
```bash
# Download the script.
$ curl -o get-kubeconfig.sh https://raw.githubusercontent.com/gravitational/teleport/master/examples/k8s-auth/get-kubeconfig.sh
# Make it executable.
$ chmod +x get-kubeconfig.sh
# Run the script, it will write the generated kubeconfig to the current
# directory.
$ ./get-kubeconfig.sh
# Check that the generated kubeconfig has the right permissions.
# The output should look similar to this.
$ kubectl --kubeconfig kubeconfig auth can-i --list
Resources Non-Resource URLs Resource Names Verbs
selfsubjectaccessreviews.authorization.k8s.io [] [] [create create]
selfsubjectrulesreviews.authorization.k8s.io [] [] [create create]
[/api/*] [] [get]
... [] [...]
groups [] [] [impersonate]
serviceaccounts [] [] [impersonate]
users [] [] [impersonate]
```
3. Copy the generated `kubeconfig` file to the host running the Teleport proxy
service.
4. Update `kubeconfig_file` path in `teleport.yaml` to where you copied the
`kubeconfig`.
Alternatively, you can use your existing local config from `~/.kube/config`.
However, it will result in Teleport proxy using your personal Kubernetes
credentials. This is risky: your credentials can expire or get revoked (such as
when leaving your company).
## Impersonation
!!! note
If you used [the script from Option
2](https://github.com/gravitational/teleport/blob/master/examples/k8s-auth/get-kubeconfig.sh)
If you used [the helm chart from Option
1](https://github.com/gravitational/teleport/blob/master/examples/k8s-auth/get-kubeconfig.sh)
above, you can skip this step. The script already configured impersonation permissions.
The next step is to configure the Teleport Proxy to be able to impersonate Kubernetes principals within a given group
@ -243,6 +260,29 @@ $ tctl users add jenkins --k8s-users="jenkins"
# example below
```
### Kubernetes Labels
Labels can be applied to Kubernetes clusters to provide a better inventory of clusters
and more fined grained RBAC.
```yaml
# ... Snippet of teleport.yaml
# Optional labels: These can be used in combination with RBAC rules
# to limit access to applications.
# When using kubeconfig_file above, these labels apply to all kubernetes
# clusters specified in the kubeconfig.
labels:
env: "prod"
# Optional Dynamic Labels
- name: "os"
command: ["/usr/bin/uname"]
period: "5s"
# Get cluster name on GKE.
- name: cluster-name
command: ['curl', 'http://metadata.google.internal/computeMetadata/v1/instance/attributes/cluster-name', '-H', 'Metadata-Flavor: Google']
period: 1m0s
```
### Github Auth
When configuring Teleport to authenticate against Github, you have to create a
@ -404,7 +444,16 @@ $ kubectl --kubeconfig /path/to/kubeconfig get pods
We've a complete guide on setting up Teleport with EKS. Please see the [Using Teleport with EKS Guide](aws-oss-guide.md#using-teleport-with-eks).
## Multiple Kubernetes Clusters
## Multiple Kubernetes Clusters via Teleport Access Plane
Teleport 5.0 adds the [long requested feature](https://github.com/gravitational/teleport/issues/3680)
of supporting multiple Kubernetes Clusters via a single Teleport Root. This setup is
described in Option 1 and uses reverse tunnels to connect back to the root cluster.
Teleport provides an option for running a single Teleport instance. This can be done
using Option 2 and a kubeconfig with multiple entries.
## Multiple Kubernetes Clusters via Trusted Cluster
You can take advantage of the [Trusted Clusters](trustedclusters.md) feature of
Teleport to federate trust across multiple Kubernetes clusters.

View file

@ -40,6 +40,8 @@ Teleport services listen on several ports. This table shows the default port num
| 3022 | Node | SSH port to the Node Service. This is Teleport's equivalent of port `22` for SSH. | Allow inbound traffic from proxy host. | Allow outbound traffic to the proxy host.
| 3025 | Auth | SSH port used by the Auth Service to serve its Auth API to other nodes in a cluster. | Allow inbound connections from all cluster nodes. | Allow outbound traffic to cluster nodes.
| 3024 | Proxy | SSH port used to create "reverse SSH tunnels" from behind-firewall environments into a trusted proxy server. | <TODO> | <TODO>
| 3026 | Kubernetes | Port used for `kubectl` to access Teleport | <TODO> | <TODO>
<!--TODO: Add several diagrams of firewall config examples-->
@ -202,8 +204,6 @@ setting up the proxy nodes start Teleport with:
See [Teleport Proxy HA](admin-guide.md#teleport-proxy-ha) for more info.
<!-- TODO Address vulns in quay? -->
<!-- TODO SSL for Webproxy & Auth Section -->
### CA Pinning