* Delete teleterm's ptyHost/v1, added by mistake
* Add package name to protos conforming to PACKAGE_VERSION_SUFFIX
* use go run in buf-connect-go.gen.yaml directly
* Run protogen in place
* Run the buf-go generation off of go run
This also adds protoc-gen-go-grpc to go.mod
Update the build scripts to properly set up the key for signing packages
using `productsign`, and parameterise the bundle ID for packages in the
packaging scripts.
* Integration resource: add types and grpc methods
This commit adds the new types for Integration resource.
It also adds the gRPC methods that will be used later on for:
- Integrations CRUD management
- Integrations resource caching
* decouple integration service from auth
* return resource on CRU operations
* Add OneOf prop to distinguish Integrations subkind
* Version subkind spec
* godocs
* release: Move Mac signing vars from script to Makefile
Move the variables for Mac signing from the `build-common.sh` shell
script to the `Makefile`. These vars will need to be passed to other
build processes to parameterize the signing for different GitHub Actions
build environments.
The switch on `ENVIRONMENT_NAME` allows different secrets to be
available in GitHub Actions for production (promote) vs developer
(build) builds. The default environment name is `promote` so as to be
compatible with the existing Drone setup, which does not define
`ENVIRONMENT_NAME`.
* release: Determine Mac signing key IDs automatically
Remove the hard-coded MacOS signing key IDs from the Makefile and find
them dynamically based on the name of the key. This allows GitHub
Actions to be set up with new keys different to the ones on the Drone
builders. As long as we keep the same name on the keys, we can rotate
the keys without needing to update the IDs in the Makefile.
This requires us to be more judicious about exporting the variables as
exporting them causes them to be evaluated. We do not want to evaluate
them on non-darwin targets, and on darwin, we should only evaluate it if
needed for a recipe. So use a dynamic `eval` in the recipes that need
the environment variables.
* release: Pass key & team ID to notarize tool
Override the hard-coded values in `notarize-apple-binaries` and pass the
values we get based on the GitHub Actions environment. This allows us to
sign and notarize software in a development branch more easily when
working on the signing and notarizing process. This will not happen
automatically, but it is expected that a developer can manually trigger
a workflow to perform building, signing and notarizing from a dev
branch where the workflow has temporarily changed the environment to
`build`.
A similar change to the `Makefile` in the teleport.e repository goes
with this change.
This adds a new bundle ID of `com.goteleport.dev` for the dev build of
Teleport. This follows the same pattern as used for the dev build of the
`tsh` binary and the current production bundle ID for Teleport.
Previously there was no dev signing/notarizing process for the set of
Teleport binaries.
* release: Add script to setup the MacOS keychain for signing
Add a script for setting up the MacOS keychain for signing applications
and packages. It encapsulates the `security` commands to add either or
both application keys and installer keys. The keys can be either
base64-encoded in environment variables, or `.p12` files on disk, making
it useful for local development.
* release: Split MacOS signing vars into separate mk file
Put the MacOS signing variables into a separate `.mk` file and include
it from the main `Makefile`. Add more comments to document the purpose
of the vars and where some of the values come from.
* release: Add some more comments to keychain-setup.sh
Explain that the purpose of the script is to be run on CI, but can also
be run manually.
Add the default values used to the usage message for the keychain and
password.
* Address PR comments on keychain-setup.sh script
* Change shebang to /bin/bash
* Use heredoc instead of multiple printfs for usage message
* Move `local` declaration next to setting of kpath var
* release: Export DEVELOPER_ID_APPLICATION in release-darwin
The sub-make for enterprise needs this to be set or it cannot sign the
enterprise binaries. Export it if we are doing signing/notarizing.
* build.assets Dockerfiles: Remove unnecessary ENV NODE_URL
NODE_URL is being redefined within the RUN instruction anyway. We suspect
it might be causing problems because sometimes the logs from build failures
suggest that the NODE_URL export was either ignored or ${NODE_URL} passed
to curl reads ENV NODE_URL and not the env var set within the shell.
* Pass fsSl flags to curl
Add a couple of parameters for the developer key ID and bundle ID for
signing/notarizing binaries. Keep the hard-coded values as defaults for
now, but we will remove these soon when all the call sites of the tool
have been updated to pass these values.
We want to parameterize these values so we can use different signing
keys in GitHub Actions and to make the tool agnostic to which binaries
it is signing.
The HTTP client created by `auth.NewClient` has an idle timeout of
30s and each instance periodically gets cert authorities in
`TeleportProcess.syncRotationStateCycle`. If there has been no
activity when the ticker expires then the request will result in a
brand new connection to Auth to be established. This can cause a
spikes in open file descriptors on the Proxy when there are a large
number of instances connected via a tunnel.
To prevent periodic dialing when retrieving cert authorities the
endpoint has been migrated to the new Trust gRPC service. For now
only the get endpoint has been migrated but the rest of the CRUD
operations for cert authorities should be migrated from the HTTP
api to the gRPC service as well.
* Emit new `AgentMetadataEvent`
Part of https://github.com/gravitational/cloud/issues/3550.
This commits adds a new `UpstreamInventoryAgentMetadata` that is sent
from a Teleport agent to an auth server.
Once received, it is transformed into an `AgentMetadataEvent` and sent
to PreHog.
Most `UpstreamInventoryAgentMetadata` fields are intentionally kept as
empty in this PR. Follow up PRs will be opened with the mechanisms
required to fill them (as described in #21337).
* server_id -> host_id
* compute OS and host architecture
* Compute OS version and container runtime
* Close stream if agent receives an agent metadata message
* cmd -> exec ; file -> read
* implement fetchOSVersion for linux
* Remove unused import
* Add note about `agentMetadataCh`
* Allow commands with args
* Remove parseFun abstraction
* fetch glibc version
* fetch container orchestrator
* Fix lint
* Fix TODO
* Add note about glibc version
* cmd -> command
* fetch cloud environment
* fetch install methods
* GLibCVersion -> GlibcVersion
* Use `http.NewRequestWithContext`
* Add missing comment
Co-authored-by: Edoardo Spadolini <edoardo.spadolini@goteleport.com>
* GLibCVersion -> GlibcVersion
* Fix lint
* Fix helm unit tests
* Add missing comments
* 5 second timeout on http requests
* Spawn gorountine that fetches metadata on each new stream
* Use `defaults.HTTPClient()`
Co-authored-by: Edoardo Spadolini <edoardo.spadolini@goteleport.com>
* Add missing import
* Handle error
* Revert "Spawn gorountine that fetches metadata on each new stream"
This reverts commit fe4f2790be.
* Send agent metadata to auth server once per stream
* Improve note about agentMetadataDone
* Don't process command output & file content on the agent
Since agents cannot be trusted, regex validation and sanitization should
happen in PreHog anyways. So this commit removes such logic in favor of
moving it to PreHog.
* Fix lint
* Trim space
* Move handling of `AgentMetadataEvent` to `handleControlStream`
* Use cached hello message
* Move metadata files to lib/inventory/metadata
* make sending of agent metadata more self contained
* Minimize diff
* Send all system roles to PreHog
* Remove unused import
* Add parsing of command output / file content back
* Usage reporter refactor
* Usage reporter refactor
* Add missing handling of inventory agent metadata msg
* Fix ICS usage reporter
* Improve comments
* Add cached `metadata.Get*` methods
* Use systemctl status instead of is-active
* Add `Metadata` struct
* return pointer in `FetchMetadata`
* Pass context to `GetMetadata`
* metadataFetchConfig -> fetchConfig
* GetMetadata -> Get
* Add note about `Get` result
* Ensure install methods are non-nil
* Exit `metadata.Get` if context is closed
* Replace sync.Once with atomic.Bool.Swap
* Initialize channel
* Fix lint
* Fix lint
* Make `metadata.Get` return an error instead of bool
* Allow multiple true/false values for env vars
* Use `strings.Cut`
* Use /etc/os-release ID instead of NAME
* Improve `autoEmitMetadata`
* Use `gnu_get_libc_version`
* Ubuntu -> ubuntu
* Use GOARCH
* gofmt
* Move import C up
* Variables may include quotes
Co-authored-by: Edoardo Spadolini <edoardo.spadolini@goteleport.com>
* Default values for ID and VERSION_ID
Co-authored-by: Edoardo Spadolini <edoardo.spadolini@goteleport.com>
* Blank lines are permitted
Co-authored-by: Edoardo Spadolini <edoardo.spadolini@goteleport.com>
* Anonymize host id
---------
Co-authored-by: Edoardo Spadolini <edoardo.spadolini@goteleport.com>
* Introduce Okta gRPC and client interfaces.
To support the upcoming Okta integration, Okta gRPC calls and client
interfaces have been added. This utilizes a non-legacy proto to try to
get away from our reliance on gogo.
This commit does NOT add the OktaImportRules and OktaAssignments to the
cache, which will occur in a later commit.
* Intermediary.
* Modifications needed in order to move this to enterprise.
* Remove okta service from grpcserver.
An interface has been created that will allow the signing of SAML IdP
requests on the auth server. This will eliminate the need for the SAML IdP
to have private key material.
* Distroless dockerfile and smoke tests for same
This patch adds a Dockerfile to `build.assets/charts` that will construct
a docker image for teleport based on the Distroless Debian images
published by Google. The actual workflows used to construct and publish
these images are defined in `teleport.e` for security reasons.
The Smoke Testing framework exists to make some quick assertions about
the resulting images: will Teleport even start in this context, etc. See
the included README for more details.
* Update README.md
* Linter appeasement
* Revert spurious submodule update
* Rename release component var
* Smoke test docs
* Smoke test docs
* Revert spurious subrepo update
* Added `--target-cloud` flag to OS package repo tool
* Updated OS package repo tool to use "version channel" instead of "artifact version"
* Added help flag examples
Changes the name to better align with the naming used by the rest
of the feature and to prevent ambiguity as the term proxy is quite
overloaded. This has not landed in a release yet so there are no
backward compatability concerns as nothing serves or consumes this
api yet.
* Add plugin exchange service
* Add Plugin methods to auth
* Add gRPC-layer methods for Plugin
* Add RBAC presets for Plugin
* Test GetPlugin()/NoSecrets access
* Make error assertions more correct in role test
* Deny setting credentials if user can not read them
* gofmt
* Apply minor suggestions from code review
Co-authored-by: rosstimothy <39066650+rosstimothy@users.noreply.github.com>
Co-authored-by: Tiago Silva <tiago.silva@goteleport.com>
* Move dependency into an existing block in go.mod
* Improve error messages for failed type assertions
* DRY WithSecrets access checks for Plugins
* Run new tests in parallel
* Improve error assertions in auth_with_roles_test
* TestGetPluginWithSecrets: split cases to subtests
* Clean up test servers and clients
* Add proto for plugin service
* Remove Plugin methods from auth service
Moved to a dedicated service
* Remove plugin-related auth methods
Moved to a dedicated service in Enterprise
* Remove CreatePlugin test from auth_with_roles_test
Moved to a dedicated service in Enterprise
* Pass "backend getter" to local plugins service
This pattern is used in Enterprise to set up secondary services
before auth (and backend) are created.
* Rename InitialCredentials to BootstrapCredentials
* Add plugins service to genproto.sh
* Reformat generated proto
* Remove obsolete PluginExchangeService
The equivalent of this is now in Enterprise
* Add kube service to genproto.sh; regenerate
* Add ListPlugins to plugin backend service
* Reimplement GetPlugins on top of ListPlugins
This is a "convenience" implementation for the backend service layer.
* Replace GetPlugins with ListPlugins in gRPC schema
* Fix ListKubernetesResources unit test
* Simplify plugin pagination key to just the name
* Use existing constant for page size
* Make dummy clients return errors instead of panic
* Remove obsolete field
* Ensure go.mod is valid for corresponding e changes
* Fix passing mutex ref
* Move teleport-plugins import to e_imports
* Revert oauth change in go.mod
* Use limit+1 to look-ahead when paginating plugins
* Test plugin pagination with pageSize > numPlugins
* Add descriptive messages to gRPC dummy clients
* Plugin: add RW for editor; remove secrets from gRPC
* Make message more descriptive for dummy gRPC conn
---------
Co-authored-by: rosstimothy <39066650+rosstimothy@users.noreply.github.com>
Co-authored-by: Tiago Silva <tiago.silva@goteleport.com>
* Install deb/yum repos when using node-join script
When a repo is available for the current Linux distro/version, use it
instead of just installing Teleport from the deb/rpm files.
It fallsback to the traditional binary installation when the repo is not
available.
* comment /etc/os-release
* remove sudo; add comment to runners
* improve is_repo_available function
My previous PR https://github.com/gravitational/teleport/pull/21660 broke the git installation path, as `git` compiles in the `--prefix` provided during configuration. Using`DESTDIR` with `make install` instead of `--prefix` copies the file in the intermediate container with the correct path.
* Unify x86/AMD64 build process
Currently, our ARM64 pipeline builds limited subset of Teleport features as none of the 3rd party dependencies (openssh, libbpf etc) are not built on AMR64. This change build all dependencies on AMR64 in the same way as we do on x86.
FIPS changes are not included as we do not support FIPS on ARM64.
* Apply suggestions from code review
Co-authored-by: Roman Tkachenko <roman@goteleport.com>
---------
Co-authored-by: Roman Tkachenko <roman@goteleport.com>
* BPF build fix
https://github.com/gravitational/teleport/pull/21745 switched CentOS 7 image to the upstream, but I missed a few other places were we're using our fork.
This change fixes all places.
* Add missing FIPS changes
* Update e
Update to libbpf 1.0.1 and github.com/aquasecurity/libbpfgo v0.4.5-libbpf-1.0.1. As we're building our releases on CentOS 7 anyway we can also switch to mainstream libbpf instead of using our fork.
The `tsh appps` family of commands is aliased to `tsh apps`, so both
invocations work correctly. The command itself is defined as `tsh apps`,
so this is what appears in the help message.
Update references to `tsh app` to recommend `tsh apps` instead so that
there isn't confusion when browsing `tsh help` and looking for a missing
`app` subcommand.
Fixes#21367
Moving our CentOS build assets, aka Clang-10 is the first step to enabling our full Teleport to build on ARM64. This change should also save us some $$ as getting the assets from S3 sounds expensive.
* Update JS grpc-tools to 1.12.4
1.11.2 didn't have support for arm64 so we had to do all this extra stuff
in the Dockerfile.
1.11.3 added support for Darwin arm64 and 1.12.4 finally adds support for
Linux arm64. This means we can completely remove extra cruft and just
install grpc-tools 1.12.4 on all architectures.
* Add comment to ptyHostService.proto
* Adjust go_package of lib/prehog Go protobufs
This makes them follow the pattern set out by api/proto and proto.
* Adjust go_package of lib/teleterm Go protobufs
* Use single buf.gen.yaml to generate JS protos
This also entailed changing the location of lib/teleterm protos and changing
the value of their package specifier to match the conventions in other parts
of the codebase. This is a breaking change but that is fine for Connect
as the protos are used locally only and each build ships with matching
protobufs.
* Make web/packages/teleterm use protobufs from gen-proto-js
We used to copy protobufs over to web/packages/teleterm/src/services/tshd
since webapps used to be in a separate repo.
This is no longer the case, so we can just make teleterm use protobufs
from gen-proto-js.
* Move prehog & teleterm protos into proto/teleport/lib
* Generate JS protos to gen/proto/js
* Move lib/teleterm Go protobufs to gen/proto/go
* Move lib/prehog Go protobufs to gen/proto/go
* Rename lib/teleterm proto package
* Re-enable linter rules for teleterm & prehog
* Update prehogv1 path in usagereporter_test.go
* Use except instead of ignore_only to allow Google API-style responses
* Add UNARY_RPC to api/proto & proto
* Ignore gen/ when running addlicense
* buf-js.gen.yaml: Remove comment about lack of go_package for JS
* Move prehog protos to proto/prehog/v1alpha
* Adjust prehog's go_package to match proto package
* Add Plugin resource schema, methods
* Improve shebang of genproto.sh
Execute using bash, no matter where it actually lives
* Use Metadata.Expiry()
Co-authored-by: Edoardo Spadolini <edoardo.spadolini@goteleport.com>
* Remove field reservations from PluginStatusCode
* Add plugin (un)marshaling
* Snake case fields of Plugin (and children)
* Ensure timestamp fields on Plugin are always UTC
https://github.com/gogo/protobuf/issues/519
* Rename credentials according to proto conventions
* Fold check for nil settings into the type switch
* Remove extraneous field checks
These are set in setStaticFields()
* Add missing godocs
---------
Co-authored-by: Edoardo Spadolini <edoardo.spadolini@goteleport.com>
