Reduces latency creating ssh sessions via the web ui by:
1) No longer uses `TeleportClient.SSH` to establish a session
2) Reuses the user auth client for the web session to perform MFA ceremony
3) Ensures that connection attempts follow the flow outlined in RFD 93
The web api server now leverages the `proxy.Router` and `srv.SessionController`
directly, instead of doing so indirectly via `TeleportClient.SSH`. Using
the `TeleportClient` required an ssh connection to be established from the web
api server to the proxy ssh server, which are in the same process. This added
overhead can be avoided now that the routing logic and session control logic
exists in a reusable component. To create an interactive session on the node
once the connection is established, `client.NodeClient` is used. A new constructor
was added to facilitate creating one and remove duplicated creation code and a
`RunInteractiveShell` receiver method was added to allow callers outside of
`lib/client` to spawn a session.
`TerminalHandler.issueSessionMFACerts` used to check if per-session mfa was enabled
and perform the mfa ceremony via the `client.ProxyClient` which was constructed
with the `TeleportClient` established from connecting to the proxy ssh server.
This would dial the Auth server under the hood directly and call `IsMFARequired`
and do the ceremony if required. Each web session established via the web ui
already established an auth client with the credentials of the logged in user.
Again overhead is removed by leveraging the existing auth client and performing
the mfa ceremony manually.
Finally `TerminalHandler.makeClient` always attempted to perform the mfa ceremony
prior to returning the `TeleportClient`. As outlined in [RFD 93](https://github.com/gravitational/teleport/blob/master/rfd/0093-offline-access.md),
this causes additional latency and requires Auth connectivity to connect to nodes.
The connection flow is now modified to attempt connection to the nodes first, and
fall back to the mfa ceremony and reconnecting only if the node denies access.
Partially addresses #15167
Session control logic existed within `HandleNewConn` of `srv/regular.Server`.
This prevented any of it from being used by other components that
also needed to enforce session control.
All the logic from within `HandleNewConn` was refactored to a new
`srv.SessionController` object which the `regular.Server` now uses
to perform session control. There were a few additional changes
needed to accomadate that session control now exists outside
the server and to make tests easier to write. Namely, altering
`srv.ComputLockTargets` to not take a `Server` as a parameter and
leveraging a clock within `services.SemaphoreLock`.
This is step 2 in addressing #15167. Before the web apiserver can
leverage the newly introduced proxy.Router and bypass making ssh
connections to the proxy ssh server it needs to be able to perfrom
session control.
* Forward traces from the web UI
Adds a `/webapi/traces` endpoint to the proxy web handler
to allow receiving traces from the UI so that they can
be forwarded to the configured exporter in the proxy_service.
To accomodate traces coming in via websockets the tracing
handler is updated to pull tracing context first from the
standard headers and fallback to retrieving it from a query
parameter as web sockets from the UI cannot alter headers.
Additionally update `web.Terminal` to propagate the tracing
context properly and instruments some of the functions
to ensure spans from the UI are properly correlated to spans
on the backend.
* create package to contain proxy peering code
* Refactor proxy routing logic into a reusable object
Routing logic existed within an unexported handler of ssh subsystem
requests, which prevented it from being reused by other components
within the proxy, like the webapi server. This causes significant
latency issues for web sessions because the web apiserver is required
to dial the proxy ssh server to determine how to route to the host.
Since the web apiserver and the proxy ssh server exist in the same
process this is an entirely unnecesarry step that could be avoided
if the routing and ability to established connections were shared
throughout the proxy.
A new `proxy.Router` object is introduced which contains all the
logic that used to exist in `regular.proxySubsys` for determining
how to connect to servers and clusters. All routing within the
`regular.proxySubsys` now leverages the `proxy.Router` to dial
the target.
This is step 1 in addressing #15167. Now that the `proxy.Router`
exists `web.APIServer` will be able to make use of it to avoid
dialing the same process to establish connections.
We were inconsistent throughout the codebase and would sometimes
use the slices package and other times use our own equivalents
in api/.
This removes our versions in favor of the golang.org/x package that
does the same, which has the added benefit of reducing the surface
area of the public API module.
Note: despite existing uses of the slices package, for some reason
it didn't show up in go.mod or go.sum. Fixed that too.
* protobuf update
* Update proto to use dynamodb request event specific to app-access
We will include a similar event for dynamodb via database-access.
We split the events so that app and database access events are not coupled.
This way we do not have to include optional database/app metadata in one event too.
* Update protos
* Update oneof
* Move AppMetaData up with the other metadata and add a 'target' field
* Remove operation plane
* Fix typo
* Configure signing service with transport instead of http client
* Protect from resource exhaustion attacks
* Add IsDynamoDB to types.Application
* Add new event and code for dynamodb requests
* Add async emitter to app access
* Add audit.go to unify app access auditing
* Refactor auditing in app access
* Use the new audit's onSessionChunk/onRequest methods
* Put the session context in the session chunk
* Use a TeeStreamer to send AppSessionDynamoDBRequest directly to audit log as well as session file
* Change streamWriter to streamCloser in sessionChunk to clarify that it should only be used for closing
* Update handler test to test dynamodb events
* Update test to use streamCloser
* Update sever test
* Add doc strings
* Return error from audit interface methods so callers can choose what do to with it
* Move app session start/end into audit interface
* Configure tcpServer to use the server's emitter instead of auth
client, as an Audit interface.
* Have tcpServer call onSessionStart/End instead of emitting events
itself.
* Remove unneeded check type
* Rename Transport -> RoundTripper
* Fix test after renaming field
* Rename drainBody and defer body closing
* Fix subtle named return mistake
* Update lib/service/service.go
Co-authored-by: Tobiasz Heller <14020794+tobiaszheller@users.noreply.github.com>
* Update lib/service/service.go
Co-authored-by: Tobiasz Heller <14020794+tobiaszheller@users.noreply.github.com>
* Rename ok->shouldSkipCleanup to make the purpose of it more clear
* Refactor request body decoding into aws utils
* Use request instead of signed request for audit event
* Determine if req is for a dynamo endpoint instead of checking app uri
* Remove obsolete app func IsDynamoDB
* Update handler test
* Use generic console app uri to test that we differentiate request by endpoint instead of app uri
* Use a dynamodb request which has a body to test that we include the body in the audit event
* Test for expected body JSON
* fix lint
* Fixup merge
Co-authored-by: Tobiasz Heller <14020794+tobiaszheller@users.noreply.github.com>
This PR presents a watcher for automatic `kube_cluster` discovery for GCP GKE clusters. Given an identity with access to the GCP cloud, the auto-discovery service will scan the cloud and register all clusters available in Kubernetes Engine.
Once the discovery service creates a `kube_cluster` on the Auth Server, the Kubernetes Service will start serving it. The credentials used to access the cluster are short-lived and generated through Google OAuth2 associated with the GCP Service Account configured for the Kubernetes Service.
GCP's Service Account must have the following role def attached:
```yaml
description: 'GKE Auto-Discovery'
includedPermissions:
- container.clusters.impersonate
- container.clusters.get
- container.clusters.list
- container.pods.get
- container.selfSubjectAccessReviews.create
- container.selfSubjectRulesReviews.create
name: projects/{projectID}/roles/GKEKubernetesAutoDisc
stage: GA
title: GKEKubernetesAutoDisc
```
Part of #16135, #13376
Related to #12048, #16276, #16281, #16633, #14991
The appaccess TestTCP* tests are highly reliant on time. This has been
reduced (but not eliminated) by using a fakeClock and a channel for
signaling monitor triggered connection closures.
Application access connection monitoring has been introduced so that, when a
lock is created, application access connections will be interrupted until the
lock has been cleared. This includes web sockets and TCP applications.
Due to a bug in Chrome, secure websockets in Teleport work intermittently on
the Chrome browser when using HTTP/2. Prioritizing HTTP/1.1 fixes this issue,
though the reasons for this are not 100% clear. When or if
https://bugs.chromium.org/p/chromium/issues/detail?id=1379017 is resolved, we
should be able to revert this. When https://github.com/golang/go/issues/49918
is implemented, we may be able to revert this if enabling HTTP/2 websockets
fixes the issue on our end.
* Add API types for CircleCI joining
* Add validation for CircleCI configuration
* Add JoinMethodCircleCI across codebase
* Add token validator and token source for circleci
* Update join methods RFD
* Add CircleCI token source to register.go
* Add serverside support for circleci joining
* Add test for TokenSource
* Add success case for token validation test
* Add expired test case
* Add test case for token from another org
* Test RegisterWithToken for CircleCI
* Refactor GitHub RegisterUsingToken tests
* Refactor CircleCI RegisterUsingToken tests
* Add tests for ProvisionTokenSpecV2
* Appease linters
* Go Imports files
* Fix failing test for msising IDtoken
* Update lib/auth/join_circleci.go
Co-authored-by: Zac Bergquist <zac.bergquist@goteleport.com>
* Move test server cleanup closer to initialisation
* Fix weird import reordering
* Include unexpected type in error message
* Simplify boolean algebra :)
Co-authored-by: Zac Bergquist <zac.bergquist@goteleport.com>
clients close the stream and any connection when they are done with it
servers close the stream after a 1 minute timeout
proxy JoinService gRPC server closes idle connections with no RPC calls after 10 seconds
When recovering after a restart, `host_uuid` changes and Auth Server authentication fails. This happens because `host_uuid` is not stored in the Kubernetes Secret but it's stored in the certificate Common Name.
This PR forces the storage of the `host_uuid` into Kubernetes Secrets for later reuse.
Fixes#17474
This PR presents a watcher for automatic `kube_cluster` discovery for Azure AKS clusters. Given a user with access to the Azure cloud, the auto-discovery service will scan the cloud and register all clusters available in AKS .
Once the discovery service creates a `kube_cluster` in Auth Server, the Kubernetes Service will start serving it. The credentials used to access the cluster depend on the different AKS clusters configurations:
# Authentication
## Local Accounts
If the AKS cluster auth is based on local accounts created during the provisioning phase of the cluster, the agent will use the [`aks:ListClusterUserCredentials`](https://learn.microsoft.com/en-us/rest/api/aks/managed-clusters/list-cluster-user-credentials?tabs=HTTP) endpoint.
This endpoint returns a `kubeconfig` fully populated with user credentials that Teleport can use to access the cluster.
## AZ Active Directory
When AZ active directory integration is enabled, Azure allows login with AD users. Azure forces the login to happen with dynamic short-lived user tokens. These tokens are generated by calling `credentials.GetToken` with a fixed Scope: `6dae42f8-4368-4678-94ff-3960e28e3630` and with the cluster's `tenant_id`. The token contains the user details as well as `group_ids` to match with authorization rules.
```go
// getAzureToken generates an authentication token for clusters with AD enabled.
func (a *aKSClient) getAzureToken(ctx context.Context, tentantID string, clientCfg *rest.Config) (time.Time, error) {
const (
azureManagedClusterScope = "6dae42f8-4368-4678-94ff-3960e28e3630"
)
cred, err := a.azIdentity(&azidentity.DefaultAzureCredentialOptions{
TenantID: tentantID,
})
if err != nil {
return time.Time{}, trace.Wrap(ConvertResponseError(err))
}
cliAccessToken, err := cred.GetToken(ctx, policy.TokenRequestOptions{
// azureManagedClusterScope is a fixed scope that identifies azure AKS managed clusters.
Scopes: []string{azureManagedClusterScope},
},
)
if err != nil {
return time.Time{}, trace.Wrap(ConvertResponseError(err))
}
// reset the old exec provider credentials
clientCfg.ExecProvider = nil
clientCfg.BearerToken = cliAccessToken.Token
return cliAccessToken.ExpiresOn, nil
}
```
# Authorization
## Local Accounts
The [`aks:ListClusterUserCredentials`](https://learn.microsoft.com/en-us/rest/api/aks/managed-clusters/list-cluster-user-credentials?tabs=HTTP) endpoint returns credentials with enough permissions for Teleport to enroll the cluster.
## AZ AD
### Azure RBAC
When Azure RBAC mode is enabled, the cluster authorization is based on rules specified in the Azure Identity permissions.
The AZ group associated with the AZ identity the Teleport Process is running has to define the following permissions:
```json
{
"Name": "AKS Teleport Discovery Permissions",
"Description": "Required permissions for Teleport auto-discovery.",
"Actions": [],
"NotActions": [],
"DataActions": [
"Microsoft.ContainerService/managedClusters/pods/read",
"Microsoft.ContainerService/managedClusters/users/impersonate/action",
"Microsoft.ContainerService/managedClusters/groups/impersonate/action",
"Microsoft.ContainerService/managedClusters/serviceaccounts/impersonate/action",
"Microsoft.ContainerService/managedClusters/authorization.k8s.io/selfsubjectaccessreviews/write",
"Microsoft.ContainerService/managedClusters/authorization.k8s.io/selfsubjectrulesreviews/write",
],
"NotDataActions": [],
"assignableScopes": [
"/subscriptions/{subscription_id}"
]
}
```
If correctly specified, the Azure authentication service automatically grants access to any cluster within `subscription_id`
without any other definition. On the other hand, if it's incorrectly configured, an error is triggered but Teleport cannot gain access to the cluster.
### Kubernetes RBAC
If AZ RBAC integration is disabled, the authorization to the cluster is processed by Kubernetes RBAC. This is done by matching the Az Identity principals (`group_ids`) with `Role`, `ClusterRole` objects that live in the AKS cluster. This mode requires that the `ClusterRole` and `ClusterRoleBinding` must exist and must be well configured for each cluster to enroll.
```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: teleport-role
rules:
- apiGroups:
- ""
resources:
- users
- groups
- serviceaccounts
verbs:
- impersonate
- apiGroups:
- ""
resources:
- pods
verbs:
- get
- apiGroups:
- "authorization.k8s.io"
resources:
- selfsubjectaccessreviews
- selfsubjectrulesreviews
verbs:
- create
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: teleport-role-binding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: teleport-role
subjects:
- kind: Group
name: {group_name}
apiGroup: rbac.authorization.k8s.io
```
#### `ClusterRole` and `ClusterRoleBinding` configured
If cluster operators or previous Teleport run has configured access to the cluster, no further action is required since Teleport already has access to the cluster.
#### Cluster `aks:ListClusterAdminCredentials` returns valid credentials
If the Teleport process has access to [`aks:ListClusterAdminCredentials`](https://learn.microsoft.com/en-us/rest/api/aks/managed-clusters/list-cluster-admin-credentials?tabs=HTTP) and the endpoint returns valid cluster admin credentials, Teleport will automatically create the `ClusterRole` and `ClusterRoleBinding` objects in the cluster configured to the `group_id` that is listed in the access token. In order to extract the `group_id` from the token, Teleport parses the JWT claims and extracts the first element.
If the object creation was successful, Teleport can access it, otherwise, it will use the `aks:BeginRunCommand` method to try to configure access to itself.
#### Cluster `aks:BeginRunCommand` returns valid credentials
When we reach this mode, Teleport tries to run a `kubectl` command against the cluster to configure the `ClusterRole` and `ClusterRoleBinding`. `aks:BeginRunCommand` allows any user with access to that endpoint to run arbitrary commands in the cluster (commands cannot be validated). Teleport will use it as the last resource to configure the access to itself.
If the command failed, Teleport cannot grant access to the cluster and an error is returned.
# UX
Currently, to discover AKS resources created and to have them dynamically served by the `kubernetes_service`one can define the following configuration.
```yaml
discovery_service:
enabled: true
azure:
- subscriptions: ["*"]
types: ["aks"]
regions: ["*"]
tags:
'*': '*'
kubernetes_service:
enabled: true
resources:
labels:
'*': '*'
```
# Future work
- Support AWS dynamic authentication
Part of #16135, #13376
Related to #12048, #16276, #16281
This PR implements the Kubernetes Tester for the ConnectionDiagnostic feature.
The goal is to validate if the Kubernetes cluster is available and if the user can access it.
A series of checks are performed against Teleport using a Kubernetes Client populated with the user's certificates for accessing the desired cluster.
Currently, the checks are:
- Kubernetes Cluster is up and running
- User's role specifies at least one `kubernetes_groups` or `kubernetes_users`
- User's roles allow RBAC access to the cluster: `kubernetes_labels` match.
- Values specified under `kubernetes_groups` or `kubernetes_users` exist in the cluster and allow the user to list pods in the desired namespace.
Testing:
Post: `/v1/webapi/sites/<teleport-cluster>/diagnostics/connections`
```json
{
"resource_kind": "kube_cluster",
"resource_name": "<cluster_name>",
"kubernetes_namespace": "default"
}
```
Closes#16824
* Revert "Azure AD authentication for the Postgres backend (#15757)"
This reverts commit 33c6d82dc3.
* Revert "SQL Backend (#11048)"
This reverts commit 06fef2abf1.
* Remove Postgres backend from the docs
* Remove the Postgres backend from the testplan
* Introduce Github Actions join support
* Go mod tidy
* run goimports on source files
* Address PR comments
* More PR review comments
* Changes to tests based on PR feedback
* Improve error message in github rule validation
* Add support for SHA
* Add short message describing which fields shouldb be included
The 24-hour grace period was kept in Teleport 10 for backwards
compatibility, but is no longer necessary for Teleport 11 due
to the session tracker system.
As a result, it takes about an hour to complete an abandoned upload
instead of 24 hours.
By only providing the tunnel address from the `reversetunnel.Resolver`
callers would still need to lookup the proxy listener mode to determine
how to dial the address. This results in sending a request to
`/webapi/find` once by the resolver to get the tunnel address and then
a second request to `/webapi/find` by users of the `Resolver` to determine
the proxy listener mode. Propagating the listener mode along with the
tunnel address by the `Resolver` ensures only one `/webapi/find` call
is needed.
This is especially impactful because the `reversetunnel.TunnelAuthDialer`
which is used by the auth http client would do this everytime the
`http.Client` connection pool was empty. When the `http.Client` needed
to dial the auth server it was incurring the additional roundtrip to the
proxy.
Prior to this change, each individual service (proxy, app, SSH, db, etc)
would spin up its own uploader service. If you are running multiple
Teleport services in the same process, this means you get multiple
uploaders all looking at the same directory, which can result in
duplicate upload events in the audit log.
Additionally, desktop access has (mistakenly) failed to set up this
service, so desktop sessions would only be uploaded if you happened
to also run some other service in the same process that does spin up
the uploader.
Solve these issues by centralizing the uploader service so that it
runs once per process, and each Teleport service doesn't need to think
about whether or not the service should run.
* Add Yubikey PrivateKey implementation for use by Teleport clients.
- Add yubikey login logic, reusing previously stored private keys.
- Fix identity file decoding with PIV keys, which sign ecdsa certificates.
- Add libpcsclite-dev pre-req for building on linux.
- Remove unnecessary keys.Signer interface and move its functionality to keys.PrivateKey.
- Move retry and jitter utils to new api/utils/retryutils package.
This PR presents a watcher for dynamic `kube_cluster` resources. The cluster credentials can be set using a `kubeconfig` whose payload is defined in `KubernertesClusterV3.Spec.Kubeconfig`. These credentials are used to connect to the Kubernetes API server and if invalid, the cluster is not served and returns an error.
Currently, `kube_cluster` resources created via the API with a kubeconfig payload can be dynamically served by the `kubernetes_service` by enabling dynamic resources.
```yaml
kubernetes_service:
enabled: true
resources:
labels:
'*': '*'
```
# Future work
- The discovery service will have to create these resources.
- Support Azure and AWS dynamic authentication
Part of #16135, #13376
Related to #12048, #16276
* Add initial version of installer
* Resolve comments
- Use aws waiters when checking commands
- Use SSMRunRequest rather than passing instances
- General comments
* Resolve comments, (rebase) pass scriptname parameter
This resolves comments regarding running on multiple ec2 instances at
once by adding state to the instances cache to check if the instance
is known about and how far into installation it is
* Revert cache
* Dont cache on non discovery nodes
* Resolve some comments
* Move discovery out to its own serviec
* Add a `discovery_service` section
* Fix messed up conflict merge
* Make starting a standalone discovery agent work
* Resolve comments
* Resolve comments
- use a regular events.Emitter
- resolve a thousand typos :)
* Resolve comments
* resolve comments, fix a bad merge
* Fail when a non ec2 matcher type is configured
* fix lint-go
* Resolve comments
* Resolve comments, add initial test (currently broken)
* Fix log string so only 1 pair of [] are used
* Chunk instances for sending commands
* add 'isInitialized' to watchers
* Add test for chunked discovery, log output
* lints
* explicetly set matcher.Tags to "*":"*" if its unset
Update metalinter, fix a few lint warnings and replace deprecated linters.
`deadcode`, `structcheck` and `varcheck` are abandoned and now replaced by [`unused`][1].
Since 1.19, `go fmt` reformats godocs according to https://go.dev/doc/comment. I've done a bulk-reformatting of the codebase to keep the linter happy. Backporting is mostly harmless (the exception being `lib/services/role_test.go`, that for some reason breaks the _old_ linter using the new format).
[1]: https://golangci-lint.run/usage/linters/
* Bump golangci-lint version
* Replace abandoned linters
* Fix bodyclose on lib/auth/github.com
* Fix bodyclose on lib/kube/proxy/streamproto/proto_test.go
* Fix bodyclose on lib/srv/alpnproxy/proxy_test.go
* Fix bodyclose on lib/web/conn_upgrade_test.go
* Silence staticcheck on lib/kube/proxy/forwarder_test.go
* Silence staticcheck on lib/utils/certs_test.go
* Address BuildNameToCertificate deprecation warnings
* Run `go fmt ./...`
* Run `go fmt ./...` on api/
* Ignore formatting in role_test.go
* Remove redundant initializers in lib/srv/uacc/
* Update e/
This PR introduces Auth Server CRUD API for managing the lifecycle of dynamic `kube_cluster` resources.
Currently, `kube_cluster` resources cannot be manipulated without using the API, nor are they being listened to. It will be part of future developments.
# Future work
- The `kubernetes_service` will have to watch these `kube_cluster` resources.
- The discovery service will have to create these resources.
Resource manipulation via `tctl` will be introduced later and is not part of the scope.
Part of #16135, #13376
Related to #12048
