* Update Dependabot Scheduled Config
Update Dependabot with the following:
* Add `jentfoo` to reviewers list
* Remove `crypto` ignore on `api` (no longer using forked version)
* Add configurations for missing gomod paths
* Update .github/dependabot.yml
Co-authored-by: Alan Parra <alan.parra@goteleport.com>
* Remove `examples` from Dependabot
---------
Co-authored-by: Alan Parra <alan.parra@goteleport.com>
- Add an item to define a scope for resolving the issue to avoid scope
creep and make it easier to close issues.
- Make a stronger plea for "Related Issues" items, since these are often
not included.
Test plan misses testing access when using proxy peering. Nothing should
differ from normal reverse tunnel access but it makes some assumptions
that differ from the reverse tunnel.
This PR adds K8S dependencies to the dependendabot ignore list.
We can revert this PR after
https://github.com/gravitational/teleport/pull/25136 merges to master.
`sigs.k8s.io/controller-runtime` is holding the K8S deps update because
it does not support K8S API 0.27.1. `controller-runtime` will release a
new version once `k8s.io/api@v0.27.2` is released.
* Remove our replacement for Logrus
Recently I attempted to update our Logrus fork. However this comment pointed out that our changes have been merged upstream: https://github.com/gravitational/logrus/pull/12#issuecomment-1515303744
For that reason this removes the dependency on the fork.
* Remove ignored dependabot dependencies that are no longer replaced
* Restore Kubernetes Integration tests
This PR re-enables the Kubernetes integrations tests using a KinD
(Kubernetes in Docker) cluster.
New steps have been introduced to GitHub's Integrations (Non-Root)
Action that configure the KinD cluster using
[`helm/kind-action`](https://github.com/helm/kind-action) and do some
network configurations allowing the container where tests run to connect
to the KinD control plane.
This PR also fixes some of the tests and fixes a bug that affected
joining operations when the target service was a legacy kubernetes
proxy. Some improvements will be introduced in future patches to improve
the logic and reduce the time required for the tests to run.
Fixes#25539
* fix data race in spdystream dep
* address feedback
* remove docker installation
* fix test
- Refresh out-of-date URLs for docs pages and `docs/config.json`
- Remove the step to add pages to the `/docs/older-versions` page, since
we generate this automatically from `gravitational/docs/config.json`,
and there is already a step to check that file
- Add a step to ensure that git submodule directories match those in
`.gitmodules`. This prevents unexpected deployment issues.
- Add more clarity to the changelog step
- Add a step to check on the status of documentation for relevant
features in the release
* Use the GHA base container for Lint (Docs)
This way, we can take advantage of the software the comes pre-installed
on the GHA `ubuntu-latest` container image. Otherwise, we need to find a
way to portably install Chromium on the `gravitational/docs` container
in order to run the Mermaid CLI. Currently, the docs engine exits with
an error during the "Lint (Docs)" job when attempting to build mermaid
diagrams due to not being able to locate Chromium.
For this change to work, the "Lint (Docs)" job checks out
`gravitational/docs`, removes the default git submodule configuration,
then adds a git submodule for the current `gravitational/teleport`
branch. From there, it can install dependencies via `yarn` and run our
CI scripts.
* s/GITHUB_HEAD_REF/GITHUB_SHA/
* Base the submodule branch source on the event type
Some edits I made to the CloudHSM docs while going through the v13 test
plan. The biggest change is an update to use the Client SDK 5, instead
of version 3. This has many benefits, you are not required to run a
client daemon, and it works with the kernel in FIPS mode (v3 doesn't).
I also added much more detail to the guide and added code samples where
I could, you should be able to go through this mostly without reading
the AWS docs, I link there for downloads or extra/optional information.
The AWS docs are very hard to follow.
This was copied from the original test plan template, but the
name was never changed. As a result, the GitHub UI shows an
error: "There is a problem with this template"
* Support spellchecking in docs content
In gravitational/docs#261, we will add a script that checks the spelling
of each version of the docs. This change edits one version of the docs
content to support this, including:
- A cspell configuration file
- A new step in the GitHub Actions in the "Lint (Docs)" workflow that
runs the spellcheck script we will add in `gravitational/docs`
- Fix mispellings so this passes the lint job. The mispellings are in a
file that we generated automatically, but there are few enough of
them, and we haven't merged the auto-generation script yet, that I
think it makes sense to fix them in the generated file for now.
* Respond to PR feedback
- Remove misspellings from the ignore list
- Sort the ignore list (and format it via prettier)
* Use the new yarn spellcheck command
* Spelling fixes
* spell fixes and add words to cspell.json
---------
Co-authored-by: Steven Martin <steven@goteleport.com>
* Disable `build-macos` and `build-windows` on PR
This commit removes the `build-macos` and `build-windows` from the PR flow, instead delegating to the bypass job.
These jobs still run at the merge queue point.
This of course means that failures in these two jobs may not be known until the merge queue.
There is an unequestionable disadvantage in not discovering those issues until that point, but this change is being recommended because:
* Currently MacOS builds are 31% of our Teleport Actions spend (~$3,500 / week)
* Windows builds are also significant at 13% (~$1,400 / week)
* There has been relatively few failures of these jobs (without other jobs also failing)
Although merge queue verification is not ideal because it's later in the process, it is considered the most critical in ensuring that `master` remains stable.
* Make sure all bypass jobs run on `ubuntu-latest`
In a couple cases this allows the jobs to be run on a cheaper instance.
* update Makefile to use cargo sparse protocol in all cargo commands
* Adds a cargo version print to build-macos for debugging
* uses the same setup steps for the rust and go toolchains as are being used in the similar enterprise workflow
* Uses the prepare-toolchain-mac composite action in the build-macos.yaml workflow.
* checkout e so that the prepare-toolchain-mac composite action is available
* Fetch the correct e ref for the composite action
* Attempts to checkout with submodules
* fetch-depth: 0
* seems that I can't get to teleport.e from the oss actions
* updates bypass
* testing ci
* testing ci
* testing for ci
* fixing indentation
* trying to get CI to actually run
* fixing indentation
* fixing lib/srv/desktop/rdp/rdpclient/client.go
In an attempt to reduce our Actions usage this PR removes the workflow execution for `push` actions on several jobs.
The following files were left as an exception to make sure flakey tests are discovered:
* integration-tests-non-root.yaml
* integration-tests-root.yaml
* unit-tests-code.yaml
* unit-tests-integrations.yaml
New endpoints were added to the API server for fetching, creating and deleting locks.
The 'editor' role now has the ability to create, edit, and remove locks by default.
Created new SlidePanel component to easily add a panel that slides in from the right of the screen.
In gravitational/docs#253, we substantially reduced the resource
consumption of docs builds. As a result, we can try building the docs as
part of the "Lint (Docs)" GitHub Actions workflow in order to prevent
build issues from breaking docs deployments.
It is currently possible to merge a docs content PR into
gravitational/teleport that can later end up breaking deployments of the
docs site, e.g., because a video ID is malformed, a code snippet label
is unsupported, etc. By building the docs during the lint job, we can
prevent this kind of thing from happening.
One complication is that the docs engine reads a `config.json` file to
match git submodule directories with version of the docs. In the
`gravitational/docs` container, `config.json` expects three submodules
pointing to three versions of the docs.
To get GitHub Actions to build a single docs version, this change
overrides the `config.json` file in the gravitational/docs container so
it only expects a single version of the docs.
* Add a GitHub Workflow for the Trivy security scanner
* Add initial ignore statements for Trivy
This accepts all the current latent findings in the repository, while still
enabling Trivy to flag new findings.
* Vendor slack plugin and supporting libraries
* Fix up plugin integration tests (wip)
* Run GCI on vendored code
* Use newtype instead of type alias
golangci-lint currently panics on this,
"skip-files" et al don't help, as it is a linter panic, not an error
See d717045480
* Remove long-runing plugins tests from difftest
* Move access plugin tests to unit-tests-integrations
* Update CodeQL Schedule to scan release branches
This allows us to switch our release branches to also use the scheduled CodeQL check so we can remove CodeQL from the PR flow for our release branches too.
* Use go dependency caching with auto build
This change speeds up the build by about ~2 minutes (10%)
This removal is primarily cost motivated. We will continue to have scheduled scans executed by ossfuzz, but for now we are disabling this job from our PR / merge queue workflows.
* Adds bypass for lint go for RFD and Docs changes.
* spacing
* Update docs test to only check for docs, examples changes or bypass
* include merge groups
* test docs change
* remove change
* Test go change
* remove go test change
* test doc changes
* remove docs test change
* fix indentation
* fix indentation
This makes two changes:
* Reduces action instance type to 16 cores. This will half the cost without a significant speed difference
* Switch from `make full` to CodeQL Auto Build. This speeds up the build by about 10 minutes. I confirmed there is not a reduction in the scan coverage
This PR implements the core of the kube-agent-updater, which is part of
https://github.com/gravitational/teleport/issues/21516#issue-1576935859
In order to have a fully working updater we still need to:
- implement the interfaces for version retrieval, image validation and maintenance trigger
- add statefulset support (and deal with the potential deadlocks)
- implement the CI and release pipeline (Dockerfile, README, Makefile, github action, drone)
- integrate in the `teleport-kube-agent` helm chart
Those changes will happen in subsequent PRs.
* Skip building webassets in CI
Building webassets is not always needed and many of our CI build just builds them and ends up not using them after.
This PR skips building the webassets for all pipelines where they are not needed. This should save us some time and $$.
* Make some changes to trigger CI
* Create the missing directory
We still plan to address this root issue, and are tracking making CodeQl faster so it can be a required PR check: https://github.com/gravitational/SecOps/issues/269
However, until then we need to switch this job to be a scheduled task rather than on every PR and push. This is partly cost motivated, but also we are already hitting our timeouts. This PR also increases the analysis timeout so that the daily job can be sure to complete.
* Unify x86/AMD64 build process
Currently, our ARM64 pipeline builds limited subset of Teleport features as none of the 3rd party dependencies (openssh, libbpf etc) are not built on AMR64. This change build all dependencies on AMR64 in the same way as we do on x86.
FIPS changes are not included as we do not support FIPS on ARM64.
* Apply suggestions from code review
Co-authored-by: Roman Tkachenko <roman@goteleport.com>
---------
Co-authored-by: Roman Tkachenko <roman@goteleport.com>
The `tsh appps` family of commands is aliased to `tsh apps`, so both
invocations work correctly. The command itself is defined as `tsh apps`,
so this is what appears in the help message.
Update references to `tsh app` to recommend `tsh apps` instead so that
there isn't confusion when browsing `tsh help` and looking for a missing
`app` subcommand.
Fixes#21367
Moving our CentOS build assets, aka Clang-10 is the first step to enabling our full Teleport to build on ARM64. This change should also save us some $$ as getting the assets from S3 sounds expensive.
* Run go mod tidy in CI
* Update e_imports.go
* Use git diff on just go.mod/go.sum, tidy api too
* Fix the e_imports list by accomodating build tags
* Wording
* Simplify commands in the workflow
* Delete go.sum before go mod tidy
* Shell suggestions
* Fix missing saml imports for upcoming PRs
* Initial pass at lint and test GHA for UI.
* Fix lint
* Add --frozen-lockfile
* Skip the e directory if it doesn't exist while linting.
* Update failing snapshots.
* use a more reasonable filter for eslint to support missing e
* ignore type check on e imports.
* ignore failing file that requires e teleterm file.
* fix lint
* quiet down the log output for prettier-write.
* Add check if protos are up to date.
A new check has been added that will detect if protobufs are up to date. The
script will exit abnormally if protobufs need to be regenerated.
* Alan's feedback.
* Restoring the script.
* Update script comment.
* Add in the set -eu.
* Add a comment for the pull_request/merge_group bit in the new github action.
* Remove helper script.
* Reduce the runner size.
In the past, we've mistakenly introduced dependencies on a newer
Go version than what the API module declares as its minimum
required version.
To prevent this from happening in the future, this job will build
the API with the version of Go declared by the API.
These workflows need to be able to check org membership
for the PR author in order to determine whether or not
the author is an internal employee. This information is
only available when authenticated.
* Switch golang.org/x/crypto to gravitational fork
* Update golden files
* Add comment to go.mod
* Update api module to use crypto fork.
* Move x/crypto to replaced section in dependabot.yml
* Add a new db engine
* Add tests for new engine
* Update tsh db subcommands
* Refactor error message and suggestions for unsupported tsh commands
* Add dynamodb to test plan
* Add AWS external ID to db config and update protos
This commit adds a new joinMethod as described in https://github.com/gravitational/teleport/pull/17905
This method allow pods running in the same Kubernetes cluster than the auth servers to join the Teleport cluster. It relies on Kubernetes tokens to establish trust. The goal is to be able to deploy proxies and auths separately and join them in a single cluser.
Pre Kubernetes 1.20, the tokens are static, long-lived, not bound to pods. We support them for compatibility reasons. Starting with Kubernetes 1.20, tokens are bound to pods (and starting with 1.21 they can be mounted through projected volumes). Starting with 1.21 we should only accept bound tokens. The chart will ensure tokens are properly mounted with projected volumes so we can benefit from the 1h to 10min token lifetime.
- Determine Go version for cache key automatically instead of hardcoding.
- Do not build ghcr CI images (etcd and buildboxes) on PRs to avoid unintended breakages.
- Only build/push them on push events which mirrors our current Drone setup. We might add ability to trigger them manually via workflow_dispatch events later.
- Add release branches pattern for buildbox images trigger as well.
- Remove packages: read permission from test jobs since buildbox images are now public.
Signed-off-by: Roman Tkachenko <roman@goteleport.com>
Co-authored-by: Victor Sokolov <gzigzigzeo@gmail.com>
* Include Go version in the cache key to prevent cache reuse when upgrading Go.
* Push buildboxes to Github container registry to avoid public ECR rate limiting.
Signed-off-by: Roman Tkachenko <roman@goteleport.com>
Co-authored-by: Victor Sokolov <gzigzigzeo@gmail.com>
[A recent /x/crypto commit][1] breaks compatibility with OpenSSH <=7.6, so we
are adding a warning to avoid bumping crypto until that is solved.
As a last resort we have https://github.com/gravitational/crypto, but we are not
using it yet.
[1]: 6fad3dfc18
