In 1.71, `.cargo-ok` changed to contain a JSON `{ v: 1 }` to indicate
the version of it. A failure of parsing will result in a heavy-hammer
approach that unpacks the `.crate` file again. This is in response to a
security issue that the unpacking didn't respect umask on Unix systems.
Without this, an attacker can leverage globally writable files buried
in the `.crate` file. After a user downloaded and unpacked the file,
the attacker can then write malicous code to the downloaded sources.
`#[allow(internal_features)]` in RUSTC_BOOTSTRAP test
This will be required in the future (where "the future" is rust-lang/rust#108955 which fails CI because of cargo here).
This does emit the unknown lints lint right now but that doesn't matter as it's just warn-by-default - internal_features is deny-by-default though, so it causes errors.
fix(update): Tweak CLI behavior
### What does this PR try to resolve?
When looking at `cargo update` for #12425, I noticed that the two flags related to `--package` were not next to it or each other. I figured grouping them like that would make things easier to browse.
When looking into that, I noticed that the two flags conflict and figured we'd provide a better error message if we did that through clap.
### How should we test and review this PR?
Looking per commit will help show the behavior changes.
### Additional information
I wanted to scope this to being simple, non-controversial, low effort, incremental improvements with this change so I did not look into the history of `--aggressive` not requiring `--package` like `--precise` does and figure out if there is any consistency we can be working towards.
Use thiserror for credential provider errors
### What does this PR try to resolve?
Errors from credential providers currently must a single string. This leads to a lot of `.map_err(|e|cargo_credential::Error::Other(e.to_string())`, which loses the `source()` of these errors.
This changes the `cargo_credential::Error` to use `thiserror` and adds a custom serialization for `std::error::Error` that preserves the source error chain across serialization / deserialization.
A unit test is added to verify serialization / deserialization.
Per my proposal here: https://github.com/rust-lang/cargo/issues/6669#issuecomment-1658593167
I tried to keep the edit minimalistic to match the surrounding style.
If the maintainers are amenable to it, I think it could also be useful to do one or more of:
- Offer concrete guidance on what to do to run actually-all tests (`--all-targets` then separately `--doc`).
- Link to the issue at: https://github.com/rust-lang/cargo/issues/6669
- Mention that `cargo test` without `--all-targets` runs doctests by default, which seems not immediately obvious.
I'd be happy to attempt to add any of the above that the maintainers feel would be a good fit here.
Git only assumes a submodule URL is a relative path if it starts with `./`
or `../` [^1]. To fetch the correct repo, we need to construct an aboslute
submodule URL.
At this moment it comes with some limitations:
* GitHub doesn't accept non-normalized URLs wth relative paths.
(`ssh://git@github.com/rust-lang/cargo.git/relative/..` is invalid)
* `url` crate cannot parse SCP-like URLs.
(`git@github.com:rust-lang/cargo.git` is not a valid WHATWG URL)
To overcome these, this patch always tries `Url::parse` first to normalize
the path. If it couldn't, append the relative path as the last resort and
pray the remote git service supports non-normalized URLs.
See also rust-lang/cargo#12404 and rust-lang/cargo#12295.
[^1]: <https://git-scm.com/docs/git-submodule>
To keep things simple, especially in getting a `Hash` implementation
correct, I'm leveraging `unicase` for case-insensitive
comparisons which is an existing dependency and I've been using for
years on other projects.
This also opens the door for us to add cross-platform compatibility
hazard warnings about multiple paths that would write to the same
location on a case insensitive file system. I held off on that because
I assume we would want #12235 first.
This does mean we can't test the "no manifest" case anymore because the
one case (no pun intended) I knew of for hitting it is now gone.
refactor(test): Move cargo-config into a dir
This is split out of #11912 and is prep for adding more UI tests.
Generally our UI tests are in a directory named after the full cargo command (`cargo config`). These tend to use `snapbox`.
Here we are tests for the `cargo config` command not written by `snapbox` in a `cargo_config.rs` file. This conflicts with adding snapbox UI tests later in a `cargo_config/` folder. Upon looking at this file, it appears to be UI tests, so I think it would make sense to move them into the `cargo_config/` folder. Definitely wouldn't make sense to move them into `config.rs` since that is general config testing.
This is split out of #11912 and is prep for adding more UI tests.
Generally our UI tests are in a directory named after the full cargo
command (`cargo config`). These tend to use `snapbox`.
Here we are tests for the `cargo config` command not written by
`snapbox` in a `cargo_config.rs` file. This conflicts with adding
snapbox UI tests later in a `cargo_config/` folder. Upon looking at this
file, it appears to be UI tests, so I think it would make sense to move
them into the `cargo_config/` folder. Definitely wouldn't make sense to
move them into `config.rs` since that is general config testing.
Currently, the UI tests are
- `cargo add`
- `cargo new`
- `cargo remove`
- `init`
One of these is not like the others. This change renames `init` to
`cargo_init` to suggest it is the UI tests for the `cargo init` command,
rather than `init` functionality.
feat(crates-io): expose HTTP headers and Error type
### What does this PR try to resolve?
This is part of #11521.
[RFC 3231] mentions the authentication process could have an additional **challenge-response mechanism** to prevent potential replay attacks. The challenge usually comes from HTTP `www-authenticate` header as a opaque string. When a client gets a 401/403 response with such a challenge, it may attach the `challenge` to the payload and request again to anwser the challenge.
```
➡️ cargo requests
⬅️ server responds with `www-authenticate` containing some opaque challenge string
➡️ cargo automatically requests again without any user perception
⬅️ server responds ok
```
However, `crates-io` crate doesn't expose HTTP headers. There is no access to `www-authenticate` header.
This PR make it expose HTTP headers and the custom `Error` type, so `cargo` can access and do further on the authentication process.
[RFC 3231]: https://rust-lang.github.io/rfcs/3231-cargo-asymmetric-tokens.html#the-authentication-process
`parent_remote_url` used to be `&str` before #12244. However, we changed
the type to `Url` and it started failing to parse scp-like URLs since
they are not compliant with WHATWG URL spec.
In this commit, we change it back to `&str` and construct the URL
manually. This should be safe since Cargo already checks if it is a
relative URL for that if branch.
fix(embedded): Always generate valid package names
### What does this PR try to resolve?
The sanitization logic uses a placeholder for the first character that isn't valid in the first character position. #12329 took the approach of always using `_` which has the problem of mixing separators if the user used `-` or we had other placeholders to insert. Instead, this takes the approach of stripping the leading invalid characters and using a placeholder name if nothing is left.
Fixes#12330
### How should we test and review this PR?
Per-commit. The first adds tests so the change in behavior can be observed over each additional commit.
### Additional information
I was also hoping to make the binary name not use placeholders by setting `bin.name` to `file_stem` but then I got
```
Compiling s-h-w-c- v0.0.0 (/home/epage/src/personal/cargo/target/tmp/cit/t133/foo)
error: invalid character `'.'` in crate name: `s_h.w§c!`
error: invalid character `'§'` in crate name: `s_h.w§c!`
error: invalid character `'!'` in crate name: `s_h.w§c!`
error: could not compile `s-h-w-c-` (bin "s-h.w§c!") due to 3 previous errors
```
I decided to not get into what are or aren't valid characters according to rustc.
- `cargo pkgid` is unsupported because we can't (yet) generate valid
pkgids for embedded manifests. Adding support for this would be a
step towards workspace support
- `cargo package` and `cargo publish` are being deferred. These would
be more important for `[lib]` support. `cargo install` for `[[bin]]`s
is a small case and As single-file packages are fairly restrictive, a
`[[bin]]` is a lower priority.
The goal is that we shouldn't interefere with end-user output when
"cargo script"s are used programmatically. The only way to detect this
is when piping. CI will also look like this.
My thought is that if someone does want to do `#!/usr/bin/env -S cargo -v`, it
should have a consistent meaning between local development
(`cargo run --manifest-path`) and "script mode" (`cargo`), so I
effectively added a new verbosity level in these cases. To get normal
output in all cases, add a `-v` like the tests do. Do `-vv` if you want
the normal `-v` mode. If you want it always quiet, do `--quiet`.
I want to see the default verbosity for interactive "script mode" a bit
quieter to the point that all normal output cargo makes is cleared before
running the built binary. I am holding off on that now as that could
tie into bigger conversations / refactors
(see https://rust-lang.zulipchat.com/#narrow/stream/246057-t-cargo/topic/Re-thinking.20cargo's.20output).
fix(script): Process config relative to script, not CWD
### What does this PR try to resolve?
This is part of the work for #12207.
When you put in your path `foo.rs`:
```rust
#!/usr/bin/env cargo
fn main() {}
```
You expect it to build once and then repeatedly run the same version. However, `.cargo/config.toml` doesn't work like that (normally). It is an environment file, like `.env`, and is based on your current working directory. So if you run `foo.rs` from within a random project, it might rebuild due to RUSTFLAGS in `.cargo/config.toml`.
I had some concern about whether this current behavior is right or not and [noted this in the Pre-RFC](https://github.com/epage/cargo-script-mvs/blob/main/0000-cargo-script.md#unresolved-questions). This came up again while we were [discussing editions on zulip](https://rust-lang.zulipchat.com/#narrow/stream/246057-t-cargo/topic/cargo.20script.20and.20edition). In looking further into this, it turns out we already have precedence for this with `cargo install --path <path>`.
### How should we test and review this PR?
The second commit has the fix, the docs, and a change to a test (from the first commit) to show that the fix actually changed behavior.
This is to avoid possible name collisions. For example, a user
creates a file called `.cargo/cache`, and then in the future
cargo wants to create a directory called `.cargo/cache/`, that
would collide with what the user specified. Restricting to `.toml`
extensions would avoid that since we won’t make a directory named
with a `.toml` extension.
fix: Allow embedded manifests in all commands
### What does this PR try to resolve?
This is a part of #12207.
One of the goals is for embedded manifests to be a first class citizen. If you have a script, you should be able to run tests on it, for example.
This expands the error check from just `Cargo.toml` to also single-file packages so you can use it in `--manifest-path`.
This, however, does mean that these *can* be used in places that likely won't work yet, like `cargo publish`.
### How should we test and review this PR?
By commit. We introduce tests for basic commands and then implement and refine the support for this.
### Additional information
Other information you want to mention in this PR, such as prior arts,
future extensions, an unresolved problem, or a TODO list.
feat(cli): Support `cargo Cargo.toml`
### What does this PR try to resolve?
This is making the assumption that we want full unity between places accepting both single-file packages and `Cargo.toml` for #12207. This has not been brought up before in any of the discussions (Internals, eRFC), so I can understand if there are concerns about this and we decide to hold off.
We might want to resolve symlinks before this so people can have a prettier name for these.
### How should we test and review this PR?
The test for this was added in a commit before the actual change, letting people see how the behavior changed.
I originally centralized the error reporting until I realized it likely
is intentionally not centralized so we report errors in terms of the
arguments the user provided.
This puts the lockfile back into a target directory in the users home,
like before #12268.
Another idea that came up was to move the workspace root to be in the
target directory (which would effectively be like pre-#12268) but I
think that is a bit hacky / misleading.
This does mean that the lockfile is buried away from the user and they
can't pass it along with their script. In most cases I've dealt with,
this would be fine. When the lockfile is needed, they will also most
likely have a workspace, so it shoud have a local lockfile in that case.
The inbetween case is something that needs further evaluation for
whether we should handle it and how.
Enable `doctest-in-workspace` by default
This stabilizes and enables the `-Z doctest-in-workspace` flag by default.
Also adds another testcase to make sure that the `include!()` and `file!()` macros interact well together.
fixes#9427
fixes https://github.com/rust-lang/rust/issues/46372
This stabilizes and enables the `-Z doctest-in-workspace` flag by default.
Also adds another testcase to make sure that the `include!()` and `file!()` macros interact well together.
fix(embedded): Don't auto-discover build.rs files
With #12268, we moved the manifest root to be the scripts parent
directory, making it so auto-discovery might pick some things up.
We previously ensured `auto*` don't pick things up but missed `build.rs`
This is now addressed.
fix(embeded): Don't pollute the scripts dir with `target/`
### What does this PR try to resolve?
This PR is part of #12207.
This specific behavior was broken in #12268 when we stopped using an intermediate
`Cargo.toml` file.
Unlike pre-#12268,
- We are hashing the path, rather than the content, with the assumption
that people change content more frequently than the path
- We are using a simpler hash than `blake3` in the hopes that we can get
away with it
Unlike the Pre-RFC demo
- We are not forcing a single target dir for all scripts in the hopes
that we get #5931
### How should we test and review this PR?
A new test was added specifically to show the target dir behavior, rather than overloading an existing test or making all tests sensitive to changes in this behavior.
### Additional information
In the future, we might want to resolve symlinks before we get to this point
With #12268, we moved the manifest root to be the scripts parent
directory, making it so auto-discovery might pick some things up.
We previously ensured `auto*` don't pick things up but missed `build.rs`
This is now addressed.
The `-Znext-lockfile-bump` is added, so we can prepare for all
lockfile format changes and then stabilize then all at once.
`-Znext-lockfile-bump` is not intended for using outside our test
suite and development. Hence it's hidden.
This was broken in #12268 when we stopped using an intermediate
`Cargo.toml` file.
Unlike pre-#12268,
- We are hashing the path, rather than the content, with the assumption
that people change content more frequently than the path
- We are using a simpler hash than `blake3` in the hopes that we can get
away with it
Unlike the Pre-RFC demo
- We are not forcing a single target dir for all scripts in the hopes
that we get #5931
To parse the manifest, we have to write it out so our regular manifest
loading code could handle it. This updates the manifest parsing code to
handle it.
This doesn't mean this will work everywhere in all cases though. For
example, ephemeral workspaces parses a manifest from the SourceId and
these won't have valid SourceIds.
As a consequence, `Cargo.lock` and `CARGO_TARGET_DIR` are changing from being next to
the temp manifest to being next to the script. This still isn't the
desired behavior but stepping stones.
This also exposes the fact that we didn't disable `autobins` like the
documentation says we should.
Background: the hash existed for sharing a target directory. That code isn't
implemented yet and a per-user build cache might remove the need for it,
so let's remove it for now and more carefully weigh adding it back in.
Immediate: This reduces the chance of hitting file length issues on Windows.
Generally: This is a bit hacky and for an official solution, we should
probably try to find a better way. This could become more important as
single-file packages are allowed in workspaces.
chore: Migrate print-ban from test to clippy
This should be more resilient to false positives like in #12245 where a string contains `println`.
See also #12224
Emit error when users try to use a toolchain via the `add` or `install` command
Running `cargo install +nightly` or `cargo add +nightly` does not actually use the nightly toolchain, but the user won't know until the compilation fails. With this PR, an error is emitted if the `install` and `add` command is given a crate name
that starts with a `+` as we assume the user's intention was to use a certain toolchain instead of installing/adding a crate.
Example:
<img width="758" alt="image" src="https://github.com/rust-lang/cargo/assets/45989466/16e59436-32ee-49ee-9933-8b68b176c09d">
Fixes#10362
This commit adds support for passing the keyword "default"
to either the CLI "--jobs" argument on the "[build.jobs]"
section of ".cargo/config".
This is dony by:
1. Changing the "jobs" config type to an enum that holds
a String or an Integer(i.e. i32).
2. Matching the enum & casting it to an integer
Signed-off-by: Charalampos Mitrodimas <charmitro@gmail.com>
It was unnecessary to pass `spilt-debuginfo` if there is no debuginfo.
Tests are touched here only for matching rustflags invocation stderr
in the original test suite.
Previously, `Debuginfo::None` meant "don't pass -C debuginfo" and `Explicit(None)` meant
"-C debuginfo=0", which occasionally led to caching bugs where cargo would sometimes pass
`-C debuginfo=0` and sometimes not. There are no such bugs currently that we know of, but
representing them the same within cargo avoids the possibility of the bug popping up again in the
future.
I tested the `with_stderr_does_not_contain_tests` with this diff to ensure they did not pass:
```diff
diff --git a/src/cargo/core/compiler/mod.rs b/src/cargo/core/compiler/mod.rs
index 55ec17182..c186dd00a 100644
--- a/src/cargo/core/compiler/mod.rs
+++ b/src/cargo/core/compiler/mod.rs
@@ -1073,9 +1073,7 @@ fn build_base_args(
let debuginfo = debuginfo.into_inner();
// Shorten the number of arguments if possible.
- if debuginfo != TomlDebugInfo::None {
cmd.arg("-C").arg(format!("debuginfo={}", debuginfo));
- }
cmd.args(unit.pkg.manifest().lint_rustflags());
if !rustflags.is_empty() {
```
fix(add): Reduce the chance we re-format the user's `[features]` table
### What does this PR try to resolve?
#11743 pointed out that we re-format the users `[features]` table when running `cargo add` which was a bug introduced in #11099.
This reduces the chance people will run into this problem
- Reducing the scope of the `fmt` call
- Preserving formatting in a simple case
Actually removing the `fmt` case can make some common formatting cases more complex to do "right", so I'm punting on that for now.
### How should we test and review this PR?
Look at the individual commits as I show how each change improves the behavior of `cargo add`.
This is a carry-over from cargo-edit where we had to worry about the UX
of all of the behavior while now we are just relying on built-in cargo
behavior and don't need to test it specifically for `cargo add`.
On my machine, this test takes 11s.
refactor(tests): Reduce cargo-add setup load
This just gets rid of irrelevant packages in the registry. Looking into which versions aren't needed would require a deeper pass, so I held off on that for now.
Before, the tests were in the 300-500ms range and now they take 100-300ms.
This did call to my attention that `unrelated` is misspelled as `unrelateed` but holding off on fixing that to reduce conflicts.
Warn when an edition 2021 crate is in a virtual workspace with default resolver
Edition 2021 updates the default resolver to version "2", but developers using virtual workspaces commonly don't get this update because the virtual workspace defaults to version "1". Warn when this situation occurs so those developers can explicitly configure their workspace and will be more likely to know that they will need to update it in the future.
Fixes#10112
This just gets rid of irrelevant packages in the registry. Looking into
which versions aren't needed would require a deeper pass, so I held off
on that for now.
Before, the tests were in the 300-500ms range and now they take
100-300ms.
This did call to my attention that `unrelated` is misspelled as
`unrelateed` but holding off on fixing that to reduce conflicts.
Edition 2021 updates the default resolver to version "2", but developers
using virtual workspaces commonly don't get this update because the
virtual workspace defaults to version "1". Warn when this situation
occurs so those developers can explicitly configure their workspace and
will be more likely to know that they will need to update it in the
future.
refactor(tests): Reduce cargo-remove setup load
This reduces the number of packages published in tests. This is an artifact of when I changed `cargo-edit` from relying on crates.io to test-generated published packages. I took the fastest path to making that conversion and took the shortcut of creating everything for every test. I had assumed the cost was low but `@Muscraft` noticed that this takes up a lot of space which we run out of on CI occasionally and I expect a lot of small files are slowing down windows.
This only updates `cargo-remove`. I'll be doing a follow up for `cargo-add`.
fix(lints): Switch to -Zlints so stable projects can experiment
### What does this PR try to resolve?
In #12115, we explored how we can let stable projects
experiment with `[lints]` to provide feedback. What we settled on is
switching from the `cargo-features` manifest key to the `-Z` flag as
`cargo-features` always requires nightly while `-Z` only requires it
when being passed in. This means a project can have a `[lints]` table
and have CI / contributors run `cargo +nightly check -Zlints` when they
care about warnings.
### How should we test and review this PR?
Demonstrate how you test this change and guide reviewers through your PR.
With a smooth review process, a pull request usually gets reviewed quicker.
If you don't know how to write and run your tests, please read the guide:
https://doc.crates.io/contrib/tests
### Additional information
I considered reworking the code to show the user the errors they would encounter once the feature is stable but held off. I wasn't quite sure what language to use and most likely a user would have something doing error reporting, like CI, so it should be fine.
feat: `lints` feature
### What does this PR try to resolve?
Implement rust-lang/rfcs#3389 which shifts a subset of `.cargo/config.toml` functionality to `Cargo.toml` by adding a `[lints]` table.
This **should** cover all of the user-facing aspects of the RFC
- This doesn't reduce what flags we fingerprint
- This will fail if any lint name as `::` in it. What to do in this case was in the RFC discussion but I couldn't find the thread to see what all was involved in that discussion
- This does not fail if a `[lints]` table is present or malformed unless nightly with the `lints` feature enabled
- The idea is this will act like a `[lints]` table is present in an existing version of rust, ignore it
- The intent is to not force an MSRV bump to use it.
- When disabled, it will be a warning
- When disabled, it will be stripped so we don't publish it
Tracking issue for this is #12115.
### How should we test and review this PR?
1. Look at this commit by commit to see it gradually build up
2. Look through the final set of test cases to make sure everything in the RFC is covered
I tried to write this in a way that will make it easy to strip out the special handling of this unstable feature, both in code and commit history
### Additional information
I'd love to bypass the need for `cargo-features = ["lints"]` so users today can test it on their existing projects but hesitated for now. We can re-evaluate that later.
I broke out the `warn_for_feature` as an experiment towards us systemitizing this stabilization approach which we also used with #9732. This works well when we can ignore the new information which isn't too often but sometimes happens.
This does involve a subtle change to `profile.rustflags` precedence but
its nightly and most likely people won't notice it? The benefit is its
in a location more like the rest of the rustflags.
In rust-lang/cargo#12115, we explored how we can let stable projects
experiment with `[lints]` to provide feedback. What we settled on is
switching from the `cargo-features` manifest key to the `-Z` flag as
`cargo-features` always requires nightly while `-Z` only requires it
when being passed in. This means a project can have a `[lints]` table
and have CI / contributors run `cargo +nightly check -Zlints` when they
care about warnings.
This does involve a subtle change to `profile.rustflags` precedence but
its nightly and most likely people won't notice it? The benefit is its
in a location more like the rest of the rustflags.
The weakening of debuginfo for build script shouldn't turn debuginfo
to `DebugInfo::None`. That will result in not passing `-C debuginfo=0`
to rustc, leading to build artifact cache miss.
Fix redacting tokens in http debug.
Unfortunately it seems like #8222 didn't properly redact tokens when connecting to an http2 server. There were multiple problems:
* For some reason, curl changes the authorization header to be lowercase when using http2.
* Curl also logs the h2h3 lines separately with a different syntax.
This fixes it by checking for these additional cases.
This also adds a test, but it doesn't actually detect this problem because we don't have an http2 server handy. You can test this yourself by running `CARGO_LOG=trace CARGO_HTTP_DEBUG=true cargo publish --token a-unique-token --allow-dirty --no-verify`, and verifying the output does not contain the given token text.
do not try an exponential number of package names
re #11934, and as discussed in the cargo team meeting, this changes the strategy to "the original, all underscore, and all dashes".
I was excessively proud of the `hyphen_combination_num` based implementation when I came up with it. But it's always been a hack. I'm glad to be the one to remove it.
Optimize usage under rustup.
Closes#10986
This optimizes cargo when running under rustup to circumvent the rustup proxies. The rustup proxies introduce overhead that can make a noticeable difference.
The solution here is to identify if cargo would normally run `rustc` from PATH, and the current `rustc` in PATH points to something that looks like a rustup proxy (by comparing it to the `rustup` binary which is a hard-link to the proxy). If it detects this situation, then it looks for a binary in `$RUSTUP_HOME/toolchains/$TOOLCHAIN/bin/$TOOL`. If it finds the direct toolchain executable, then it uses that instead.
## Considerations
There have been some past attempts in the past to address this, but it has been a tricky problem to solve. This change has some risk because cargo is attempting to guess what the user and rustup wants, and it may guess wrong. Here are some considerations and risks for this:
* Setting `RUSTC` (as in https://github.com/rust-lang/rustup/pull/2958) isn't an option. This makes the `RUSTC` setting "sticky" through invocations of different toolchains, such as a cargo subcommand or build script which does something like `cargo +nightly build`.
* Changing `PATH` isn't an option, due to issues like https://github.com/rust-lang/rustup/issues/3036 where cargo subcommands would be unable to execute proxies (so things like `+toolchain` shorthands don't work).
* Setting other environment variables in rustup (as in https://github.com/rust-lang/rustup/pull/3207 which adds `RUSTUP_TOOLCHAIN_DIR` the path to the toolchain dir) comes with various complications, as there is risk that the environment variables could get out of sync with one another (like with `RUSTUP_TOOLCHAIN`), causing tools to break or become confused.
There was some consideration in that PR for adding protections by using an encoded environment variable that could be cross-checked, but I have concerns about the complexity of the solution.
We may want to go with this solution in the long run, but I would like to try a short term solution in this PR first to see how it turns out.
* This won't work for a `rustup-toolchain.toml` override with a [`path`](https://rust-lang.github.io/rustup/overrides.html#path) setting. Cargo will use the slow path in that case. In theory it could try to detect this situation, which may be an exercise for the future.
* Some build-scripts, proc-macros, or custom cargo subcommands may be doing unusual things that interfere with the assumptions made in this PR. For example, a custom subcommand could call a `cargo` executable that is not managed by rustup. Proc-macros may be executing cargo or rustc, assuming it will reach some particular toolchain. It can be difficult to predict what unusual ways cargo and rustc are being used. This PR (and its tests) tries to make extra sure that it is resilient even in unusual circumstances.
* The "dev" fallback in rustup can introduce some complications for some solutions to this problem. If a rustup toolchain does not have cargo, such as with a developer "toolchain link", then rustup will automatically call either the nightly, beta, or stable cargo if they are available. This PR should work correctly, since rustup sets the correct `RUSTUP_TOOLCHAIN` environment variable for the *actual* toolchain, not the one where cargo was executed from.
* Special care should be considered for dynamic linking. `LD_LIBRARY_PATH` (linux), `DYLD_LIBRARY_PATH` (macos), and `PATH` (windows) need to be carefully set so that `rustc` can find its shared libraries. Directly executing `rustc` has some risk that it will load the wrong shared libraries. There are some mitigations for this. macOS and Linux use rpath, and Windows looks in the same directory as `rustc.exe`. Also, rustup configures the dyld environment variables from the outer cargo. Finally, cargo also configures these (particularly for the deprecated compiler plugins).
* This shouldn't impact installations that don't use rustup.
* I've done a variety of testing on the big three platforms, but certainly nowhere exhaustive.
* One of many examples is making sure Clippy's development environment works correctly, which has special requirements for dynamic linking.
* There is risk about future rustup versions changing some assumptions made here. Some assumptions:
* It assumes that if `RUSTUP_TOOLCHAIN` is set, then the proxy *always* runs exactly that toolchain and no other. If this changes, cargo could execute the wrong version. Currently `RUSTUP_TOOLCHAIN` is the highest priority [toolchain override](https://rust-lang.github.io/rustup/overrides.html) and is fundamental to how toolchain selection becomes "sticky", so I think it is unlikely to change.
* It assumes rustup sets `RUSTUP_TOOLCHAIN` to a value that is exactly equal to the name of the toolchain in the `toolchains` directory. This works for user shorthands like `RUSTUP_TOOLCHAIN=nightly`, which gets converted to the full toolchain name. However, it does not work for `path` overrides (see above).
* It assumes the `toolchains` directory layout is always `$RUSTUP_HOME/toolchains/$TOOLCHAIN`. If this changes, then I think the only consequence is that cargo will go back to the slow path.
* It assumes downloading toolchains is not needed (since cargo running from the toolchain means it should already be downloaded).
* It assumes there is no other environment setup needed (such as the dyld paths mentioned above).
My hope is that if assumptions are no longer valid that the worst case is that cargo falls back to the slow path of running the proxy from PATH.
## Performance
This change won't affect the performance on Windows because rustup currently alters PATH to point to the toolchain directory. However, https://github.com/rust-lang/rustup/pull/3178 is attempting to remove that, so this PR will be required to avoid a performance penalty on Windows. That change is currently opt-in, and will likely take a long while to roll out since it won't be released until after the next release, and may be difficult to get sufficient testing.
I have done some rough performance testing on macOS, Windows, and Linux on a variety of different kinds of projects with different commands. The following attempts to summarize what I saw.
The timings are going to be heavily dependent on the system and the project. These are the values I get on my systems, but will likely be very different for everyone else.
The Windows tests were performed with a custom build of rustup with https://github.com/rust-lang/rustup/pull/3178 applied and enabled (stock rustup shows no change in performance as explained above).
The data is summarized in this spreadsheet: https://docs.google.com/spreadsheets/d/1zSvU1fQ0uSELxv3VqWmegGBhbLR-8_KUkyIzCIk21X0/edit?usp=sharing
`hello-world` has a particularly large impact of about 1.68 to 2.7x faster. However, a large portion of this overhead is related to running `rustc` at the start to discover its version and querying it for information. This is cached after the first run, so except for first-time builds, the effect isn't as noticeable. The "check with info" row is an benchmark that removes `target/debug/deps` but keeps the `.rustc_info.json` file.
Incremental builds are a bit more difficult to construct since it requires customizing the commands for each project. I only did an incremental test for cargo itself, running `touch src/cargo/lib.rs` and then `cargo check --lib`.
These measurements excluded the initial overhead of launching the rustup proxy to launch the initial cargo process. This was done just for simplicity, but it makes the test a little less characteristic of a typical usage, which will have some constant overhead for running the proxy.
These tests were done using [`hyperfine`](https://crates.io/crates/hyperfine) version 1.16.1. The macOS system was an M2 Max (12-thread). The Windows and Linux experiments were run on a AMD Ryzen Threadripper 2950X (32-thread). Rust 1.68.2 was used for testing. I can share the commands if people want to see them.
support for shallow clones and fetches with `gitoxide`
This PR makes it possible to enable shallow clones and fetches for git dependencies and crate indices independently with the `-Zgitoxide=fetch,shallow_deps` and `-Zgitoxide=fetch,shallow_index` respectively.
### Tasks
* [x] setup the shallow option when fetching, differentiated by 'registry' and 'git-dependency'
* [x] validate registries are cloned shallowly *and* fetched shallowly
* [x] validate git-dependencies are cloned shallowly *and* fetched shallowly
* [x] a test to show what happens if a shallow index is opened with `git2` (*it can open it and fetch like normal, no issues*)
* [x] assure that `git2` can safely operate on a shallow clone - we unshallow it beforehand, both for registries and git dependencies
* [x] assure git-deps with revisions are handled correctly (they should just not be shallow, and they should unshallow themselves if they are)
* [x] make sure shallow index clones aren't seen by older cargo's
* [x] make sure shallow git dependency clones aren't seen by older cargo's
* [x] shallow.lock test and more test-suite runs with shallow clones enabled for everything
* [x] release new version of `gix` with full shallow support and use it here
* [x] check why `shallow` files remain after unshallowing. Should they not rather be deleted if empty? - Yes, `git` does so as well, implemented [with this commit](2cd5054b0a)
* ~~see if it can be avoided to ever unshallow an existing `-shallow` clone by using the right location from the start. If not, test that we can go `shallow->unshallow->shallow` without a hitch.~~ Cannot happen anymore as it can predict the final location perfectly.
* [x] `Cargo.lock` files don't prevent shallow clones
* [x] assure all other tests work with shallow cloning enabled (or fix the ones that don't with regression protection)
* [x] can the 'split-brain' issue be solved for good?
### Review Notes
* there is a chance of 'split brain' in git-dependencies as the logic for determining whether the clone/fetch is shallow is repeated in two places. This isn't the case for registries though.
### Notes
* I am highlighting that this is the `gitoxide` version of shallow clones as the `git2` version [might soon be available](https://github.com/libgit2/libgit2/pull/6396) as well. Having that would be good as it would ensure interoperability remains intact.
* Maybe for when `git2` has been phased out, i.e. everything else is working, I think (unscientifically) there might be benefits in using worktrees for checkouts. Admittedly I don't know the history of why they weren't used in the first place. Also: `gitoxide` doesn't yet support local clones and might not have to if worktrees were used instead.
Doing so seems cleaner as there should be no logical difference between
shallow or not-shallow when fetching. We want a specific object, and should
get it with the refspec. `git` will assure we see all objects we need,
handling shallow-ness for us.
Note that one test needed adjustments due to the different mechanism used
when fetching local repositories, requiring more changes to properly 'break'
the submodule repo when `gitoxide` is used.
cargo-tree: Handle -e no-proc-macro when building the graph
### What does this PR try to resolve?
Makes `-e no-proc-macro` more useful when combined with `-i` or `-d`. Fixes#12030.
### How should we test and review this PR?
The new and existing tests should cover this, I hope!
### Additional information
Pruning proc-macro crates during graph construction is closer to how the edge-based filters work (`[no-]build` etc.), so even though `no-proc-macro` isn't technically filtering on edges, it's following a well-established code path.
This is an improvement over the previous version which would use unshallowing that effectively
makes a shallow repo *not* shallow.
Furthermore, we will now only fetch a single commit, each time we fetch, which should be faster
for the server as well as for the client.
We also make it possible to fetch individual commits that would be specified via Cargo.lock.
A couple of test expectations are adjusted accordingly.
Is this desirable behaviour? Unfortunately, there is no alternative
as adding shallow to an existing index most definitely breaks backwards
compatibility.
The implementation hinges on passing information about the kind of clone
and fetch to the `fetch()` method, which then configures the fetch accordingly.
Note that it doesn't differentiate between initial clones and fetches as
the shallow-ness of the repository is maintained nonetheless.
crates.io reads rust-version from the tarball directly, but we can include it in
the publish request for the sake of consistency for third-party registries.
The headers can significantly contribute to noise in the output,
drowning out the rest of the output. Most investigation will likely be
focused on the case where cargo completely fails to download, so this
only shows the full detail in the final error message.
