feat(help): Add styling to help output
### What does this PR try to resolve?
Try to make `--help` output easier to parse by using terminal styling
Screenshots:
*(`nargo` is my shell script wrapping `cargo run --manifest-path cargo/Cargo.toml`)*
### How should we test and review this PR?
At this time, the only styling snapshotting library I know of is a pain to use, so testing this requires manually running the commands which I did. Screenshots are included for easier evaluation of the general idea.
Snapshotting of the plain text output ensures we don't have accidental formatting regressions from this change since the formatting isn't as obvious from looking at the code.
### Additional information
Traditionally, cargo has disabled clap's styled output. My assumed
reason is that cargo mixes custom help output with auto-generated and
you couldn't previously make it all styled.
Clap 4.2 allowed users to pass in strings styled using ANSI escape
codes, allowing us to pass in styled text that matches clap, unblocking this. In clap
4.4.1, clap gained the ability for the user to override the style.
In this PR, I decided to use the new 4.4.1 feature to style clap's
output to match the rest of cargo's output. Alternatively, we could use
a more subdue style that clap uses by default.
I used the `color-print` crate to allow something almost html-like for styling `&static str`. Alternatively, we could directly embed the ANSI escape codes harder to get write, harder to inspect), or we could do the styling at runtime and enable the `string` feature in clap.
I decided to *not* style `Arg::help` messages because
- It might be distracting to have the descriptions lit up like a
christmas tree
- It is a lot more work
The one exception I made was for `--list` since it is for a
psuedo-command (`...`) and I wanted to intentionally draw attention to
it.
#12593 made styling of `cargo -h` cleaner imo.
#12592 and #12594 were improvements I noticed while doing this.
Ues strip_prefix for cleaner code
### What does this PR try to resolve?
In https://github.com/rust-lang/cargo/pull/12629#pullrequestreview-1614154046 Ed pointed out how much cleaner the code can be using `strip_prefix`, so I found a bunch more places where we should be using it.
### How should we test and review this PR?
Internal refactor and test still pass.
Traditionally, cargo has disabled clap's styled help output. My assumed
reason is that cargo mixes custom help output with auto-generated and
you couldn't previously make it all styled to match. Clap 4.2 allowed
users to pass in strings styled using ANSI escape codes, allowing us to
pass in styled text that matches clap, unblocking this. In clap
4.4.1, clap gained the ability for the user to override the style,
allowing us to choose the styling as we wish.
In this PR, I decided to use the new 4.4.1 feature to style clap's
output to match the rest of cargo's output. Alternatively, we could use
a more subdue style that clap uses by default. That subdued style was
mostly chosen to be app theme neurtral (since we didn't have theming
support yet) and there were problems with our style and no one stepped
up to fix them (cargo has a style we can match to instead).
I decided to *not* style `Arg::help` messages because
- It might be distracting to have the descriptions lit up like a
christmas tree
- It is a lot more work
The one exception I made was for `--list` since it is for a
psuedo-command (`...`) and I wanted to intentionally draw attention to
it.
As of serde 1.0.172, `serde_derive` ships a binary blog for Linux x64
for faster build times. This blob is not yet reproducible to ensure
that the safety of it. See serde-rs/serde#2538
This is not a judgement on serde or on dtolnay but just a precaution to
buy us more time as the community works through this since the beta cut
is coming up. rust-1.72 branch is unaffected.
cargo-credential-gnome-secret: dynamically load libsecret
Building `cargo-credential-gnome-secret` currently requires the `libsecret` development libraries to be installed and findable via `pkg-config`. This is often an extra step for users and complicates CI builds.
This loads the required functions from `libsecret` dynamically using `libloading` which uses `dlopen` internally.
Closes#12503
Testing this requires manually installing the credential provider on a system with libsecret set up. I tested it on Arch Linux.
ci: rewrite bump check and respect semver
### What does this PR try to resolve?
This ports `ci/validate-version-bump.sh` and #12200 to the new xtask `bump-check`. It also adds the ability to set the correct baseline revision when checking version bump. That is, it fixes#12347.
In addition, this integrates the community tool `cargo-semver-checks`. SemVer violation can now be detected earlier.
### How should we test and review this PR?
This PR extracts the registry query part from `xtask-unpublished` and removes other pars. I don't feel like we need it in the short term.
For how it works, please the check doc comment in each function.
One concern is that this check is still a required job for bors. I believe `@obi1kenobi` is quite responsive and willing to help if there is something wrong. So, waiting for an upstream fix won't be a problem for cargo. Also as a good citizen in the community, we can always contribute back.
(Take it easy `@obi1kenobi,` don't be stressed out 🙂)
<!-- homu-ignore:end -->
fix(package): Recognize and normalize `cargo.toml`
### What does this PR try to resolve?
This solution is a blend of conservative and easy
- Normalizes `cargo.toml` to `Cargo.toml` on publish
- Ensuring we publish the `prepare_for_publish` version and include `Cargo.toml.orig`
- Avoids dealing with trying to push existing users to `Cargo.toml`
- All other cases of `Cargo.toml` are warnings
- We could either normalize or turn this into an error in the future
- When helping users with case previously, we've only handle the `cargo.toml` case
- We already should have a fallback in case a manifest isn't detected
- I didn't want to put in too much effort to make the code more complicated to handle this
As a side effect, if a Linux user has `cargo.toml` and `Cargo.toml`, we'll only put one of them in the `.crate` file. We can extend this out to also include a warning for portability for case insensitive filesystems but I left that for after rust-lang/cargo#12235.
### How should we test and review this PR?
A PR at a time will show how the behavior changed as the source was edited
This does add a direct dependency on `unicase` to help keep case-insensitive comparisons easy / clear and to avoid riskier areas for bugs like writing an appropriate `Hash` implementation. `unicase` is an existing transitive dependency of cargo.
### Additional information
Fixes#12384
[Discussion on Zulip](https://rust-lang.zulipchat.com/#narrow/stream/246057-t-cargo/topic/.60cargo.2Etoml.60.20on.20case.20insensitive.20filesystems)
To keep things simple, especially in getting a `Hash` implementation
correct, I'm leveraging `unicase` for case-insensitive
comparisons which is an existing dependency and I've been using for
years on other projects.
This also opens the door for us to add cross-platform compatibility
hazard warnings about multiple paths that would write to the same
location on a case insensitive file system. I held off on that because
I assume we would want #12235 first.
This does mean we can't test the "no manifest" case anymore because the
one case (no pun intended) I knew of for hitting it is now gone.
No code changes are required for the current uses of indexmap.
I also updated `toml_edit` to get its update to `indexmap 2`, and
`gix-hashtable` to align them all on `hashbrown 0.14`.
fix(embedded): Don't create an intermediate manifest
### What does this PR try to resolve?
More immediately, this is to unblock rust-lang/rust#112601
More generally, this gets us away from hackily writing out an out-of-line manifest from an embedded manifest. To parse the manifest, we have to write it out so our regular manifest
loading code could handle it. This updates the manifest parsing code to
handle it.
This doesn't mean this will work everywhere in all cases though. For
example, ephemeral workspaces parses a manifest from the SourceId and
these won't have valid SourceIds.
As a consequence, `Cargo.lock` and `CARGO_TARGET_DIR` are changing from being next to
the temp manifest to being next to the script. This still isn't the
desired behavior but stepping stones.
### How should we test and review this PR?
A Commit at a time
### Additional information
In production code, this does not conflict with #12255 (due to #12262) but in test code, it does.
To parse the manifest, we have to write it out so our regular manifest
loading code could handle it. This updates the manifest parsing code to
handle it.
This doesn't mean this will work everywhere in all cases though. For
example, ephemeral workspaces parses a manifest from the SourceId and
these won't have valid SourceIds.
As a consequence, `Cargo.lock` and `CARGO_TARGET_DIR` are changing from being next to
the temp manifest to being next to the script. This still isn't the
desired behavior but stepping stones.
This also exposes the fact that we didn't disable `autobins` like the
documentation says we should.
The hope is this will result in more resilient comment handling, being
more consistent with rustdoc.
I also hoped for less code but `syn` is doing less than I had expected,
requiring us to copy code over from other parts of rust. It seems every
proc macro has to do this but there is no guide to it, so they all do it
differently, only covering the cases they thought to test for.
Note that this still won't support `include_str!()`.
Previously, fetches and clones would routinely fail with a panic
that indicated that pack-negotiation can't take longer than 1 round
with our previous `Naive` approach.
With this version of `gitoxide` there is now faithful support for both
the `consecutive` and the `skipping` algorithm and multiple rounds of
negotiations, which should make all clones and fetches possible.
support for shallow clones and fetches with `gitoxide`
This PR makes it possible to enable shallow clones and fetches for git dependencies and crate indices independently with the `-Zgitoxide=fetch,shallow_deps` and `-Zgitoxide=fetch,shallow_index` respectively.
### Tasks
* [x] setup the shallow option when fetching, differentiated by 'registry' and 'git-dependency'
* [x] validate registries are cloned shallowly *and* fetched shallowly
* [x] validate git-dependencies are cloned shallowly *and* fetched shallowly
* [x] a test to show what happens if a shallow index is opened with `git2` (*it can open it and fetch like normal, no issues*)
* [x] assure that `git2` can safely operate on a shallow clone - we unshallow it beforehand, both for registries and git dependencies
* [x] assure git-deps with revisions are handled correctly (they should just not be shallow, and they should unshallow themselves if they are)
* [x] make sure shallow index clones aren't seen by older cargo's
* [x] make sure shallow git dependency clones aren't seen by older cargo's
* [x] shallow.lock test and more test-suite runs with shallow clones enabled for everything
* [x] release new version of `gix` with full shallow support and use it here
* [x] check why `shallow` files remain after unshallowing. Should they not rather be deleted if empty? - Yes, `git` does so as well, implemented [with this commit](2cd5054b0a)
* ~~see if it can be avoided to ever unshallow an existing `-shallow` clone by using the right location from the start. If not, test that we can go `shallow->unshallow->shallow` without a hitch.~~ Cannot happen anymore as it can predict the final location perfectly.
* [x] `Cargo.lock` files don't prevent shallow clones
* [x] assure all other tests work with shallow cloning enabled (or fix the ones that don't with regression protection)
* [x] can the 'split-brain' issue be solved for good?
### Review Notes
* there is a chance of 'split brain' in git-dependencies as the logic for determining whether the clone/fetch is shallow is repeated in two places. This isn't the case for registries though.
### Notes
* I am highlighting that this is the `gitoxide` version of shallow clones as the `git2` version [might soon be available](https://github.com/libgit2/libgit2/pull/6396) as well. Having that would be good as it would ensure interoperability remains intact.
* Maybe for when `git2` has been phased out, i.e. everything else is working, I think (unscientifically) there might be benefits in using worktrees for checkouts. Admittedly I don't know the history of why they weren't used in the first place. Also: `gitoxide` doesn't yet support local clones and might not have to if worktrees were used instead.
The implementation hinges on passing information about the kind of clone
and fetch to the `fetch()` method, which then configures the fetch accordingly.
Note that it doesn't differentiate between initial clones and fetches as
the shallow-ness of the repository is maintained nonetheless.
This is a follow up to #12039.
This makes it easier for tools to report less irrelevant information.
I did both `publish = false` and `version = "0.0.0"` to help draw
attention to the fact that these crates are internal (inspired by a
matklad post).
I left `cargo-test-macro` and `cargo-test-support` in for my own
personal bias of one day wanting to see those crates published...
The only one removed that had previously been published was `mdman` but
seeing as that was a `0.0.0`, I'm assuming that was a mistake or just
reserving the name.
Before:
```console
$ cargo unpublished
name published current
==== ========= =======
cargo-platform 0.1.2 0.1.3
cargo-test-macro - 0.1.0
cargo-test-support - 0.1.0
cargo-util 0.2.3 0.2.4
crates-io 0.36.0 0.36.1
mdman 0.0.0 0.1.0
resolver-tests - 0.1.0
cargo 0.70.1 0.72.0
semver-check - 0.1.0
cargo-credential 0.1.0 0.2.0
cargo-credential-1password 0.1.0 0.2.0
cargo-credential-gnome-secret 0.1.0 0.2.0
cargo-credential-macos-keychain 0.1.0 0.2.0
cargo-credential-wincred 0.1.0 0.2.0
benchsuite - 0.1.0
```
After:
```console
name published current
==== ========= =======
cargo-platform 0.1.2 0.1.3
cargo-test-macro - 0.1.0
cargo-test-support - 0.1.0
cargo-util 0.2.3 0.2.4
crates-io 0.36.0 0.36.1
cargo 0.70.1 0.72.0
cargo-credential 0.1.0 0.2.0
cargo-credential-1password 0.1.0 0.2.0
cargo-credential-gnome-secret 0.1.0 0.2.0
cargo-credential-macos-keychain 0.1.0 0.2.0
cargo-credential-wincred 0.1.0 0.2.0
```
chore(xtask): Add `cargo xtask unpublished`
### What does this PR try to resolve?
This tries to make it easy to see what existing versions have not been published. A future version of this could post to a PR what the current delta in version numbers for touched crates so reviewer have more context when deciding if they should ask for a crate version to be bumped
```console
$ cargo unpublished
Finished dev [unoptimized + debuginfo] target(s) in 0.12s
Running `/home/epage/src/personal/cargo/target/debug/xtask unpublished`
Updating crates.io index
name published current
==== ========= =======
cargo-test-macro - 0.1.0
cargo-test-support - 0.1.0
cargo-util 0.2.3 0.2.4
mdman 0.0.0 0.1.0
resolver-tests - 0.1.0
cargo 0.70.0 0.71.0
cargo-credential 0.1.0 0.2.0
cargo-credential-1password 0.1.0 0.2.0
cargo-credential-gnome-secret 0.1.0 0.2.0
cargo-credential-macos-keychain 0.1.0 0.2.0
cargo-credential-wincred 0.1.0 0.2.0
benchsuite - 0.1.0
```
Room for improvement
- Aligning the start of each column
- Filtering the list by a commit range
- Adding this to an action to post to a review
- Maybe sorting the output
- Marking some our crates as `package.publish = false`, like benchsuite and resolver-tests
### How should we test and review this PR?
This is broken down commit by commit for easier seeing of the building blocks for our first xtask
This will allow running an xtask without requiring building the world.
In most cases, a user will already have been building cargo but not in
CI.
The packages keep an `xtask-` prefix to help raise awareness of them but
exposed as `cargo <suffix>` to avoid having a direction proxy to wrap
`cargo run -p xtask-<suffix>` as `cargo xtask <suffix>`.
Update windows-sys
This updates the windows-sys dependency from 0.45 to 0.48. This shouldn't add or remove any duplicate dependencies (since there are other dependencies still using 0.45 and 0.42). The intent is to move it along the direction towards unifying in the future (though it seems like a moving target that will be difficult to ever hit).
This also bumps the home crate version. I think it should be OK to make the migration from winapi to windows-sys a patch version, though there seems to be some issues with the way windows-sys works that could introduce some build-time problems in some situations (such as those encountered in https://github.com/rust-lang/rust/pull/108665 and https://github.com/rust-lang/rust/pull/106610). However, I don't expect too much of an issue.
Bump libc to 0.2.142
libc 0.2.141 cannot build successfully on AIX. (CI on AIX is not available yet)
Upgrade libc to 0.2.142 to make cargo build on AIX.
Update openssl-src to 111.25.3+1.1.1t
### What does this PR try to resolve?
Support for LoongArch has been added to the `openssl-src` start from `v111.25.3`. Therefore, we have updated the version to include this support.
Thanks
There's now a lock file upstream in rust-lang/rust so the one here isn't
actually used, and otherwise this crate is used as a dependency so the lock file
isn't respected anyway!
Fixes#4326
`cargo package` (and so `cargo publish`) parses a crate’s `Cargo.toml`,
makes some modifications, and re-serializes it.
Because the `TomlManifest` struct uses `HashMap`
with its default `RandomState` hasher,
the maps’ iteration order changed on every run.
As a result, when using `cargo vendor`,
updating a dependency would generate a diff larger than necessary,
with non-significant order-changes obscuring significant changes.
This replaces some uses of `HashMap` with `BTreeMap`,
whose iteration order is deterministic (based on `Ord`).
Use `same-file` to avoid unnecessary hard links
This is targeted at removing the need for a workaround in rust-lang/rust#39518,
allowing the main rust build system to move back to hard links which should be
much more efficient.
This is targeted at removing the need for a workaround in rust-lang/rust#39518,
allowing the main rust build system to move back to hard links which should be
much more efficient.
Add gitignore-like pattern matching logic to `list_files()` and throw
warnings for paths getting different inclusion/exclusion results from
the old and the new methods.
Migration Tracking: <https://github.com/rust-lang/cargo/issues/4268>
The API of `termcolor` fits what the system gives us much more nicely and should
be well battle-tested from ripgrep. Additionally we don't really need huge
terminfo parsers, that wasn't every really the intention of the color support
here.
Removing some allocations arounds the stored hashes by having nested hash maps
instead of tuple keys. Also remove an intermediate array when parsing
dependencies through a custom implementation of `Deserialize`. While this
doesn't make this code path blazingly fast it definitely knocks it down in the
profiles below other higher-value targets.
This commit is a relatively serious optimization pass of the resolution phase in
Cargo, targeted at removing as many allocations as possible from this phase.
Executed as an iterative loop this phase of Cargo can often be costly for large
graphs but it's run on every single build!
The main optimization here is to avoid cloning the context and/or pushing a
backtracking frame if there are no candidates left in the current list of
candidates. That optimizes a fast-path for crates with lock files (almost all of
them) and gets us to the point where cloning the context basically disappears
from all profiling.
Add a GNU make jobserver implementation to Cargo
This commit adds a GNU make jobserver implementation to Cargo, both as a client
of existing jobservers and also a creator of new jobservers. The jobserver is
actually just an IPC semaphore which manifests itself as a pipe with N bytes
of tokens on Unix and a literal IPC semaphore on Windows. The rough protocol
is then if you want to run a job you read acquire the semaphore (read a byte on
Unix or wait on the semaphore on Windows) and then you release it when you're
done.
All the hairy details of the jobserver implementation are housed in the
`jobserver` crate on crates.io instead of Cargo. This should hopefully make it
much easier for the compiler to also share a jobserver implementation
eventually.
The main tricky bit here is that on Unix and Windows acquiring a jobserver token
will block the calling thread. We need to either way for a running job to exit
or to acquire a new token when we want to spawn a new job. To handle this the
current implementation spawns a helper thread that does the blocking and sends a
message back to Cargo when it receives a token. It's a little trickier with
shutting down this thread gracefully as well but more details can be found in
the `jobserver` crate.
Unfortunately crates are unlikely to see an immediate benefit of this once
implemented. Most crates are run with a manual `make -jN` and this overrides the
jobserver in the environment, creating a new jobserver in the sub-make. If the
`-jN` argument is removed, however, then `make` will share Cargo's jobserver and
properly limit parallelism.
Closes#1744
This commit adds a GNU make jobserver implementation to Cargo, both as a client
of existing jobservers and also a creator of new jobservers. The jobserver is
actually just an IPC semaphore which manifests itself as a pipe with N bytes
of tokens on Unix and a literal IPC semaphore on Windows. The rough protocol
is then if you want to run a job you read acquire the semaphore (read a byte on
Unix or wait on the semaphore on Windows) and then you release it when you're
done.
All the hairy details of the jobserver implementation are housed in the
`jobserver` crate on crates.io instead of Cargo. This should hopefully make it
much easier for the compiler to also share a jobserver implementation
eventually.
The main tricky bit here is that on Unix and Windows acquiring a jobserver token
will block the calling thread. We need to either way for a running job to exit
or to acquire a new token when we want to spawn a new job. To handle this the
current implementation spawns a helper thread that does the blocking and sends a
message back to Cargo when it receives a token. It's a little trickier with
shutting down this thread gracefully as well but more details can be found in
the `jobserver` crate.
Unfortunately crates are unlikely to see an immediate benefit of this once
implemented. Most crates are run with a manual `make -jN` and this overrides the
jobserver in the environment, creating a new jobserver in the sub-make. If the
`-jN` argument is removed, however, then `make` will share Cargo's jobserver and
properly limit parallelism.
Closes#1744
Convert CargoResult, CargoError into an implementation provided by error-chain. The previous is_human machinery is mostly removed; now errors are displayed unless of the Internal kind, verbose mode will print all errors.
