teleport/rfd/0125-dynamic-auto-discovery-config.md

authors	state
Marco Dinis (marco.dinis@goteleport.com)	draft

RFD 0125 - Dynamic Auto-Discovery Configuration

Related RFDs:

• RFD 57 - Automatic discovery and enrollment of AWS servers (discovery_service RFD)

Required Approvers

• Engineering: @r0mant && @smallinsky && @tigrato
• Product: @klizhentas || @xinding33
• Security: @reedloden || @jentfoo

What

Dynamically configure matchers for the Auto-Discovery service.

Why

Teleport Discover will implement a flow where users are able to install a discovery_service and configure it without ever leaving the WebApp.

To do so, the discovery_service will be deployed with minimal configuration and let the user configure the matchers afterwards.

Although, this RFD is born from this necessity, there's nothing bounding this feature to using the Discover Wizard, and rather make this dynamic configuration available for any discovery_service.

Having an almost stateless configuration eases the configuration and deployment of discovery_services, which is essential for adoption. Users no longer need to log in to a machine that is running this service to change the matchers, and can do any sort of configuration from the clients (ie WebApp, tctl, Terraform Provider and Kubernetes Operator).

Details

There will be a new resource - DiscoveryConfig - which will be used by the discovery_service to add extra matchers for that particular service.

discovery_service must support having only the discovery_group and still run, even if no DiscoveryConfig matches that group.

New DiscoveryConfig resource

A new resource will be created to store the desired matchers.

The spec schema will contain a discovery_group and a list of matchers.

The discovery_group will be used by discovery_service to include matchers from DiscoveryConfig that have the same group.

The list of matchers is the same list that exists in discovery_service (for AWS, Azure and GCP). Currently, that list is composed using the following matchers:

• lib/config.AWSMatcher
• lib/config.AzureMatcher
• lib/config.GCPMatcher

In order to use those types, we must first define them as protobuf types and then import them in lib/config.

This way, we have a single place where we define the matchers properties. They are then used when loading the configuration from file or when defining DiscoveryConfig resources.

Example:

kind: DiscoveryConfig
version: v1
metadata:
  name: production-resources
spec:
    discovery_group: prod-resources-all-clouds
    aws:
     - types: ["ec2"]
       regions: ["us-east-1","us-west-1"]
       tags:
         "*": "*"
       install:
         join_params:
           token_name:  "aws-discovery-iam-token"
         script_name: "default-installer"
       ssm:
         document_name: "TeleportDiscoveryInstaller"
    azure:
    - types: ["aks"]
      regions: ["eastus", "westus"]
      subscriptions: ["11111111-2222-3333-4444-555555555555"]
      resource_groups: ["group1", "group2"]
      tags:
        "*": "*"
    gcp:
    - types: ["gke"]
      locations: ["*"]
      tags:
        "*": "*"
      project_ids: ["myproject"]

Discovery Group

The Discovery Group is used to match between discovery_service and DiscoveryConfig.

So, all discovery_services with Group X will include all the matchers from DiscoveryConfigs whose Group is also X.

Hot-reloading DiscoveryConfig matchers

When the discovery_service starts, it loads all the static matchers plus the dynamic ones (using DiscoveryConfig resources).

When the user changes (either creating, deleting or updating) a DiscoveryConfig, the discovery_service must update its discovery matchers.

To achieve this, discovery_service must create a Watcher over the DiscoveryConfig resource and when a new change occurs update its internal list of matchers.

	watcher, _ := clt.NewWatcher(ctx, types.Watch{
		Kinds: []types.WatchKind{{Kind: types.KindDiscoveryConfig}},
	})

	defer watcher.Close()

	for {
		select {
		case event := <-watcher.Events():
			// ...
			discoveryConfig := event.Resource
			switch event.Type {
			case types.OpDelete:
				discoveryWatchers.RemoveByDiscoveryConfigName(discoveryConfig.GetName())
			case types.OpPut:
				discoveryWatchers.UpsertDiscoveryConfig(discoveryConfig)
			}

			return nil

		case <-watcher.Done():
			if err := watcher.Error(); err != nil {
				return trace.Wrap(err)
			}
			return nil
		case <-ctx.Done():
			return nil
		}
	}

Proto Specification for DiscoveryConfig

DiscoveryConfig will have the following protobuf specification

// DiscoveryConfigV1 represents a DiscoveryConfig to be used by discovery_services.
message DiscoveryConfigV1 {
  // Header is the resource header.
  ResourceHeader Header = 1;

  // Spec is an DiscoveryConfig specification.
  DiscoveryConfigSpecV1 Spec = 2 ;
}

// DiscoveryConfigSpecV1 contains properties required to create matchers to be used by discovery_service.
// Those matchers are used by discovery_service to watch for cloud resources and create them in Teleport.
message DiscoveryConfigSpecV1 {
  // DiscoveryGroup is used by discovery_service to add extra matchers.
  // All the discovery_services that have the same discovery_group, will load the matchers of this resource.
  string DiscoveryGroup = 1;

  // AWS is a list of AWSMatchers.
  repeated AWSMatcher aws = 2;

  // Azure is a list of AzureMatchers.
  repeated AzureMatcher azure = 3;

  // GCP is a list of GCPMatchers.
  repeated GCPMatcher gcp = 4;
}

// AWSMatcher matches AWS EC2 instances and AWS Databases
message AWSMatcher {
  // Types are AWS database types to match, "ec2", "rds", "redshift", "elasticache",
	// or "memorydb".
  repeated string Types = 1;
  // Regions are AWS regions to query for databases.
	repeated string Regions = 2;
	// AssumeRoleARN is the AWS role to assume for database discovery.
	string AssumeRoleARN = 3;
	// ExternalID is the AWS external ID to use when assuming a role for
	// database discovery in an external AWS account.
	string ExternalID = 4;
	// Tags are AWS tags to match.
  wrappers.LabelValues Tags = 5;
	// InstallParams sets the join method when installing on
	// discovered EC2 nodes
	MatcherInstallParams InstallParams = 6;
	// SSM provides options to use when sending a document command to
	// an EC2 node
	AWSSSM SSM = 7;
}

// MatcherInstallParams sets join method to use on discovered nodes
message MatcherInstallParams {
	// JoinParams sets the token and method to use when generating
	// config on cloud instances
	DiscoveryJoinParams JoinParams = 1;
	// ScriptName is the name of the teleport installer script
	// resource for the cloud instance to execute
	string ScriptName = 2;
	// InstallTeleport disables agentless discovery
	string InstallTeleport = 3;
	// SSHDConfig provides the path to write sshd configuration changes
	string SSHDConfig = 4;
	// PublicProxyAddr is the address of the proxy the discovered node should use
	// to connect to the cluster. Used only in Azure.
	string PublicProxyAddr = 5;
}

// AWSSSM provides options to use when executing SSM documents
message AWSSSM {
	// DocumentName is the name of the document to use when executing an
	// SSM command
	string DocumentName = 1;
}

// JoinParams configures the parameters for Simplified Node Joining.
message DiscoveryJoinParams {
  string TokenName = 1;
	string Method = 2;
	AzureJoinParams Azure = 3;
}

// AzureJoinParams configures the parameters specific to the Azure join method.
message AzureJoinParams {
  string ClientID = 1;
}

// AzureMatcher matches Azure resources.
// It defines which resource types, filters and some configuration params.
message AzureMatcher {
	// Subscriptions are Azure subscriptions to query for resources.
	repeated string Subscriptions = 1;
	// ResourceGroups are Azure resource groups to query for resources.
	repeated string ResourceGroups = 2;
	// Types are Azure types to match: "mysql", "postgres", "aks", "vm"
	repeated string Types = 3;
	// Regions are Azure locations to match for databases.
	repeated string Regions = 4;
	// ResourceTags are Azure tags on resources to match.
  wrappers.LabelValues ResourceTags = 5;
	// InstallParams sets the join method when installing on
	// discovered Azure nodes.
	MatcherInstallParams InstallParams = 6;
}

// GCPMatcher matches GCP resources.
message GCPMatcher {
	// Types are GKE resource types to match: "gke".
	repeated string Types = 1;
	// Locations are GKE locations to search resources for.
	repeated string Locations = 2;
	// Tags are GCP labels to match.
  wrappers.LabelValues Tags = 3;
	// ProjectIDs are the GCP project ID where the resources are deployed.
	repeated string ProjectIDs = 4;
}

Backward Compatibility

Watcher on DiscoveryConfig

If the discovery_service tries to lookup DiscoveryConfig resources, but Teleport Auth/Proxy is not yet aware of this resource, then discovery_service will log a warning but continue with the static configuration.

UX

discovery_service new service configuration properties

Users need to ensure the discovery_group (which is optional) is set, in order for it to include extra matchers from DiscoveryConfig.

Example:

discovery_service:
  enabled: "yes"
  discovery_group: "my-production-resources"

gRPC: manage DiscoveryConfig resource

The following methods will be created in the gRPC server to allow DiscoveryConfig resource management:

// DiscoveryConfigService provides methods to manage DiscoveryConfig resources.
// These resources are used by `discovery_service` to set up the matchers.
service DiscoveryConfigService {
  // ListDiscoveryConfig returns a paginated list of DiscoveryConfig resources.
  rpc ListDiscoveryConfig(ListDiscoveryConfigRequest) returns (ListDiscoveryConfigResponse);

  // GetDiscoveryConfig returns the specified DiscoveryConfig resource.
  rpc GetDiscoveryConfig(GetDiscoveryConfigRequest) returns (types.DiscoveryConfigV1);

  // CreateDiscoveryConfig creates a new DiscoveryConfig resource.
  rpc CreateDiscoveryConfig(CreateDiscoveryConfigRequest) returns (types.DiscoveryConfigV1);

  // UpdateDiscoveryConfig updates an existing DiscoveryConfig resource.
  rpc UpdateDiscoveryConfig(UpdateDiscoveryConfigRequest) returns (types.DiscoveryConfigV1);

  // DeleteDiscoveryConfig removes the specified DiscoveryConfig resource.
  rpc DeleteDiscoveryConfig(DeleteDiscoveryConfigRequest) returns (google.protobuf.Empty);

  // DeleteAllDiscoveryConfigs removes all DiscoveryConfigs.
  rpc DeleteAllDiscoveryConfigs(DeleteAllDiscoveryConfigsRequest) returns (google.protobuf.Empty);
}

WebAPI: manage DiscoveryConfig resource

The following endpoints must be create to be used by WebAPI:

Methods:
GET .../discoveryconfig
GET .../discoveryconfig/:id
POST .../discoveryconfig
PUT .../discoveryconfig/:id
DELETE .../discoveryconfig/:id

JSON representation:

{
   "discovery_group": "prod-resources-all-clouds",
   "aws":[
      {
         "types":[
            "ec2"
         ],
         "regions":[
            "us-east-1",
            "us-west-1"
         ],
         "tags":{
            "*":"*"
         },
         "install":{
            "join_params":{
               "token_name":"aws-discovery-iam-token"
            },
            "script_name":"default-installer"
         },
         "ssm":{
            "document_name":"TeleportDiscoveryInstaller"
         }
      }
   ],
   "azure":[
      {
         "types":[
            "aks"
         ],
         "regions":[
            "eastus",
            "westus"
         ],
         "subscriptions":[
            "11111111-2222-3333-4444-555555555555"
         ],
         "resource_groups":[
            "group1",
            "group2"
         ],
         "tags":{
            "*":"*"
         }
      }
   ],
   "gcp":[
      {
         "types":[
            "gke"
         ],
         "locations":[
            "*"
         ],
         "tags":{
            "*":"*"
         },
         "project_ids":[
            "myproject"
         ]
      }
   ]
}

tctl: manage DiscoveryConfig resource

Users must be able to create, list, read, update and delete DiscoveryConfigs using tctl.

Example of $ tctl get discovery_config:

kind: DiscoveryConfig
version: v1
metadata:
  name: production-resources
spec:
    discovery_group: my-ec2
    aws:
     - types: ["ec2"]
       regions: ["us-east-1","us-west-1"]
       tags:
         "*": "*"

IaC - Terraform

A new resource DiscoveryConfig must be created to allow its management from the Terraform provider.

IaC - Kube Operator

A new resource DiscoveryConfig must be created to allow its management from the Kube Operator.

IaC - Helm Charts

Helm charts must support the new property when setting up a discovery_service.

Security

RBAC for DiscoveryConfig resource

The editor preset role will include read and write access to this new resource.

The discovery system role (used by discovery_service) must be able to list and read DiscoveryConfig resources to be able to update the matchers of its service.