Fix O0 build scripts by default without `[profile.release]`
This fixes an issue where #8500 didn't quite work as expected, since it
only worked if a crate had a `[profile.release]` section.
Update lock file encodings on changes
This commit alters Cargo's lockfile encoding update strategy from its
previous incarnation. Previously Cargo had two versions, one for new
lock files and one for old lock files. Each of these versions were
different and would affect how Cargo manages lock file updates. The
intention was that we'd roll out defaults to new lock files first and
then later to preexisting lock files. This requires two separate
changes, though, and it's not necessarily clear when to start updating
old lock files. Additionally when old lock files were opted in it would
break builds using `--locked` if they simply updated Cargo because Cargo
would would want to bring the lock file versions forward.
The purpose of this change is to solve these issues. The new strategy
for updating a lock file's encoding is to simply preserve what's already
existing on the filesystem until we actually decide to write out a new
lock file. When Cargo updates a lock file on-disk then it will, at that
time, update the lock file encoding to whatever the current default is.
This means that there's only one version number to keep track of (the
default for encoding). Cargo will always preserve the preexisting
encoding unless another change is required to the lock file.
Fix sporadic lto test failures.
These tests can fail because the order of the messages is not deterministic. The two `foo` jobs start in parallel, so the order can be swapped. Results in an error like:
```
---- lto::test_profile stdout ----
running `/home/runner/work/cargo/cargo/target/debug/cargo test -v`
thread 'lto::test_profile' panicked at '
Expected: execs
but: differences:
6 - |[RUNNING] `rustc --crate-name foo [..]--crate-type lib --emit=dep-info,metadata,link -Cembed-bitcode=no[..]|
+ | Running `rustc --crate-name foo --edition=2018 src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --emit=dep-info,link -C lto=thin -C debuginfo=2 --test -C metadata=af771ad588185fac -C extra-filename=-af771ad588185fac --out-dir /home/runner/work/cargo/cargo/target/cit/t1080/foo/target/debug/deps -L dependency=/home/runner/work/cargo/cargo/target/cit/t1080/foo/target/debug/deps --extern bar=/home/runner/work/cargo/cargo/target/cit/t1080/foo/target/debug/deps/libbar-398704963dd4e38b.rlib`|
7 - |[RUNNING] `rustc --crate-name foo [..]--emit=dep-info,link -C lto=thin [..]--test[..]|
+ | Running `rustc --crate-name foo --edition=2018 src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type lib --emit=dep-info,metadata,link -Cembed-bitcode=no -C debuginfo=2 -C metadata=1e8f5d385f4ffeb6 -C extra-filename=-1e8f5d385f4ffeb6 --out-dir /home/runner/work/cargo/cargo/target/cit/t1080/foo/target/debug/deps -L dependency=/home/runner/work/cargo/cargo/target/cit/t1080/foo/target/debug/deps --extern bar=/home/runner/work/cargo/cargo/target/cit/t1080/foo/target/debug/deps/libbar-398704963dd4e38b.rmeta`|
```
The failure rate is pretty small (depends heavily on the hardware).
This commit alters Cargo's lockfile encoding update strategy from its
previous incarnation. Previously Cargo had two versions, one for new
lock files and one for old lock files. Each of these versions were
different and would affect how Cargo manages lock file updates. The
intention was that we'd roll out defaults to new lock files first and
then later to preexisting lock files. This requires two separate
changes, though, and it's not necessarily clear when to start updating
old lock files. Additionally when old lock files were opted in it would
break builds using `--locked` if they simply updated Cargo because Cargo
would would want to bring the lock file versions forward.
The purpose of this change is to solve these issues. The new strategy
for updating a lock file's encoding is to simply preserve what's already
existing on the filesystem until we actually decide to write out a new
lock file. When Cargo updates a lock file on-disk then it will, at that
time, update the lock file encoding to whatever the current default is.
This means that there's only one version number to keep track of (the
default for encoding). Cargo will always preserve the preexisting
encoding unless another change is required to the lock file.
Display builtin aliases with `cargo --list`
As stated in #8486 it would help to the discovery of the
builtin aliases the facto of printing them with the
`cargo --list` command.
- Extracted the builtin aliases currently implemented to a
separated `const`.
- Make all of the functions that interact with these aliases
point to that function.
- Refactored the `list_commands` fn in order to include with the
builtin and external commands, the builtin aliases that come with
cargo by default.
- Added a test that checks that the aliases that currently
are builtin with cargo are indeed being printed with the rest
of the commands when `cargo --list` is called.
The output on my machine looks like this:
```
$ cargo --list
Installed Commands:
b alias: build
bench Execute all benchmarks of a local package
build Compile a local package and all of its dependencies
c alias: check
check Check a local package and all of its dependencies for errors
clean Remove artifacts that cargo has generated in the past
doc Build a package's documentation
fetch Fetch dependencies of a package from the network
fix Automatically fix lint warnings reported by rustc
generate-lockfile Generate the lockfile for a package
git-checkout This subcommand has been removed
init Create a new cargo package in an existing directory
install Install a Rust binary. Default location is $HOME/.cargo/bin
locate-project Print a JSON representation of a Cargo.toml file's location
login Save an api token from the registry locally. If token is not specified, it will be read from stdin.
metadata Output the resolved dependencies of a package, the concrete used versions including overrides, in machine-readable format
new Create a new cargo package at <path>
owner Manage the owners of a crate on the registry
package Assemble the local package into a distributable tarball
pkgid Print a fully qualified package specification
publish Upload a package to the registry
r alias: run
read-manifest Print a JSON representation of a Cargo.toml manifest.
run Run a binary or example of the local package
rustc Compile a package, and pass extra options to the compiler
rustdoc Build a package's documentation, using specified custom flags.
search Search packages in crates.io
t alias: test
test Execute all unit and integration tests and build examples of a local package
tree Display a tree visualization of a dependency graph
uninstall Remove a Rust binary
update Update dependencies as recorded in the local lock file
vendor Vendor all dependencies for a project locally
verify-project Check correctness of crate manifest
version Show version information
yank Remove a pushed crate from the index
clippy
clippy
clippy
clippy
flamegraph
fmt
fmt
fmt
fmt
miri
miri
miri
miri
outdated
tree
```
As discussed with @ehuss the `BTreeSet` enforces `Ord` therefore, the aliases get mixed with the commands since they're passed through the same function.
It can be refactored to appear separately, but, the code will be more spread and now it's all in just one file (which I believe is easier to maintain and review).
Closes#8486
Check manifest for requiring nonexistent features
Fixes#4854: Examples requiring a nonexistent feature should be an error
Thanks @lukaslueg with his https://github.com/rust-lang/cargo/pull/4874 for the inspiration!
Added a test that checks that the aliases that currently
are builtin with cargo are indeed being printed with the rest
of the commands when `cargo --list` is called.
Closes#8486
This commit implements a simple warning to indicate when `DefaultBranch`
is unified with `Branch("master")`, meaning `Cargo.toml` inconsistently
lists `branch = "master"` and not. The intention here is to ensure that
all projects uniformly use one or the other to ensure we can smoothly
transition to the new lock file format.
This commit lays the groundwork for an eventual V3 of the lock file
format. The changes in this format are:
* A `version` indicator will be at the top of the file so we don't have
to guess what format the lock is in, we know for sure. Additionally
Cargo now reading a super-from-the-future lock file format will give a
better error.
* Git dependencies with `Branch("master")` will be encoded with
`?branch=master` instead of with nothing.
The motivation for this change is to eventually switch Cargo's
interpretation of default git branches.
This commit implements a warning in Cargo for when a dependency
directive is using `DefaultBranch` but the default branch of the remote
repository is not actually `master`. We will eventually break this
dependency directive and the warning indicates the fix, which is to
write down `branch = "master"`.
This commit is intended to be an effective but not literal revert
of #8364. Internally Cargo will still distinguish between
`DefaultBranch` and `Branch("master")` when reading `Cargo.toml` files,
but for almost all purposes the two are equivalent. This will namely fix
the issue we have with lock file encodings where both are encoded with
no `branch` (and without a branch it's parsed from a lock file as
`DefaultBranch`).
This will preserve the change that `cargo vendor` will not print out
`branch = "master"` annotations but that desugars to match the lock file
on the other end, so it should continue to work.
Tests have been added in this commit for the regressions found on #8468.
Stabilize -Z crate-versions
This stabilizes the `-Z crate-versions` flag which forwards the crate version to rustdoc so it can display the version in the sidebar.
The flag in rustdoc was stabilized in 1.44.
Closes#7907
Mask out system core.autocrlf settings before resetting git repos
This fixes an issue the gecko developers noticed when vendoring
on windows. \[0\] If a user has `core.autocrlf=true` set
(a reasonable default on windows), vendoring from a git source
would cause all the newlines to be rewritten to include carriage
returns, creating churn and platform-specific results.
To fix this, we simply set the global cargo checkout's "local"
core.autocrlf value before performing a `reset`. This masks out
the system configuration without interfering with the user's
own system/project settings.
\[0\]: https://bugzilla.mozilla.org/show_bug.cgi?id=1647582
Check workspace member existence as dir.
Cargo command fails if workspace members are set to something like `crates/*` and if there's any non project file in `crates` directory. This PR makes member discovery logic to exclude non-directory paths.
In my case, `.DS_Store` (which is made automatically by Finder on macOS) file triggered this issue.
Apply workspace.exclude to workspace.default-members.
Not sure how controversial the feature request was, it seemed easy to do, so I did it (I'm aware it's possible this won't be accepted).
Fixes#8460
This fixes an issue the gecko developers noticed when vendoring
on windows. [0] If a user has `core.autocrlf=true` set
(a reasonable default on windows), vendoring from a git source
would cause all the newlines to be rewritten to include carriage
returns, creating churn and platform-specific results.
To fix this, we simply set the global cargo checkout's "local"
core.autocrlf value before performing a `reset`. This masks out
the system configuration without interfering with the user's
own system/project settings.
[0]: https://bugzilla.mozilla.org/show_bug.cgi?id=1647582
Build host dependencies with opt-level 0 by default
This commit updates Cargo's build of host dependencies to build them
with optimization level 0 by default instead of matching the profile of
the final binary.
Since Cargo's inception build dependencies have, by default, been built
in a profile that largely matches the profile of the final target
artifact. Build dependencies, however, rarely actually need to be
optimized and are often executing very small tasks, which means that
optimizing them often wastes a lot of build time. A great example of
this is procedural macros where `syn` and friends are pretty heavyweight
to optimize, and the amount of Rust code they're parsing is typically
quite small, so the time spent optimizing rarely comes as a benefit.
The goal of this PR is to improve build times on average in the
community by not spending time optimizing build dependencies (build
scripts, procedural macros, and their transitive dependencies). The PR
will not be a universal win for everyone, however. There's some
situations where your build time may actually increase:
* In some cases build scripts and procedural macros can take quite a
long time to run!
* Cargo may not build dependencies more than once if they're shared with
the main build. This only applies to builds without `--target` where
the same crate is used in the final binary as in a build script.
In these cases, however, the `build-override` profile has existed for
some time know and allows giving a knob to tweak this behavior. For
example to get back the previous build behavior of Cargo you would
specify, in `Cargo.toml`:
[profile.release.build-override]
opt-level = 3
or you can configure this via the environment:
export CARGO_PROFILE_RELEASE_BUILD_OVERRIDE_OPT_LEVEL=3
There are two notable features we would like to add in the future which
would make the impact of a change like this smaller, but they're not
implemented at this time (nor do we have concrete plans to implement
them). First we would like crates to have a way of specifying they
should be optimized by default, despite default profile options. Often
crates, like lalrpop historically, have abysmal performance in debug
mode and almost always (even in debug builds) want to be built in
release mode. The second feature is that ideally crate authors would be
able to tell Cargo to minimize the number of crates built, unifying
profiles where possible to avoid double-compiling crates.
At this time though the Cargo team feels that the benefit of changing
the defaults is well worth this change. Neither today nor directly after
this change will be a perfect world, but it's hoped that this change
makes things less bad!
This commit updates Cargo's build of host dependencies to build them
with optimization level 0 by default instead of matching the profile of
the final binary.
Since Cargo's inception build dependencies have, by default, been built
in a profile that largely matches the profile of the final target
artifact. Build dependencies, however, rarely actually need to be
optimized and are often executing very small tasks, which means that
optimizing them often wastes a lot of build time. A great example of
this is procedural macros where `syn` and friends are pretty heavyweight
to optimize, and the amount of Rust code they're parsing is typically
quite small, so the time spent optimizing rarely comes as a benefit.
The goal of this PR is to improve build times on average in the
community by not spending time optimizing build dependencies (build
scripts, procedural macros, and their transitive dependencies). The PR
will not be a universal win for everyone, however. There's some
situations where your build time may actually increase:
* In some cases build scripts and procedural macros can take quite a
long time to run!
* Cargo may not build dependencies more than once if they're shared with
the main build. This only applies to builds without `--target` where
the same crate is used in the final binary as in a build script.
In these cases, however, the `build-override` profile has existed for
some time know and allows giving a knob to tweak this behavior. For
example to get back the previous build behavior of Cargo you would
specify, in `Cargo.toml`:
[profile.release.build-override]
opt-level = 3
or you can configure this via the environment:
export CARGO_PROFILE_RELEASE_BUILD_OVERRIDE_OPT_LEVEL=3
There are two notable features we would like to add in the future which
would make the impact of a change like this smaller, but they're not
implemented at this time (nor do we have concrete plans to implement
them). First we would like crates to have a way of specifying they
should be optimized by default, despite default profile options. Often
crates, like lalrpop historically, have abysmal performance in debug
mode and almost always (even in debug builds) want to be built in
release mode. The second feature is that ideally crate authors would be
able to tell Cargo to minimize the number of crates built, unifying
profiles where possible to avoid double-compiling crates.
At this time though the Cargo team feels that the benefit of changing
the defaults is well worth this change. Neither today nor directly after
this change will be a perfect world, but it's hoped that this change
makes things less bad!
This commit fixes an issue in Cargo where when an entire project
directory is renamed (preserving the target directory) then path
dependencies with build scripts would have their build scripts rereun
when building again. The problem with this was that when a build script
doesn't print `rerun-if-changed` Cargo's conservative fingerprint listed
an absolute path in it, which was intended to be a relative path.
The fix here is to use a relative path in the fingerprint to ensure that
it's not the reason a rebuild happens when directories are renamed.
Add a `-Zbuild-std-features` flag
This flag is intended to pair with `-Zbuild-std` as necessary to
configure the features that libstd is built with. This is highly
unlikely to ever be stabilized in any form (unlike `-Zbuild-std` which
we'd like to stabilize at some point), but can be useful for
experimenting with the standard library. For example today it can be
used to test changes to binary size by disabling backtraces.
My intention is that we won't need a `--no-default-features` equivalent
for libstd, where after rust-lang/rust#74377 is merged we can
unconditionally specify default features are disabled but the default
set of features lists `default`. That way if users want to override the
list *and* include the default feature, they can just be sure to include
`default`.
This flag is intended to pair with `-Zbuild-std` as necessary to
configure the features that libstd is built with. This is highly
unlikely to ever be stabilized in any form (unlike `-Zbuild-std` which
we'd like to stabilize at some point), but can be useful for
experimenting with the standard library. For example today it can be
used to test changes to binary size by disabling backtraces.
My intention is that we won't need a `--no-default-features` equivalent
for libstd, where after rust-lang/rust#74377 is merged we can
unconditionally specify default features are disabled but the default
set of features lists `default`. That way if users want to override the
list *and* include the default feature, they can just be sure to include
`default`.
Ensure `unstable.build-std` works like `-Zbuild-std`
This fixes an issue where the deserializer for `-Zbuild-std` was a bit
fancier than the `unstable.build-std` directive.
cc #8393
Make `cargo metadata` output deterministic
Uses BTreeMap instead of HashMap for the `cargo metadata` command, ensuring the output is sorted.
The change did not cause a measurable performance impact for running `cargo metadata` on `cargo` itself.
Fixes#8477
Allow configuring unstable flags via config file
# Summary
This fixes#8127 by mapping the `unstable` key in `.cargo/config` to Z flags.
It should have no impact on stable/beta cargo, and on nightlies it gives folks the ability to configure Z flags for an entire project or workspace. This is meant to make it easier to try things like the [new features resolver](https://github.com/rust-lang/cargo/issues/8088) behavior, mtime-on-use, build-std, or timing reports across a whole project.
I've also included a small (but entirely independent) ergonomics change -- treating dashes and underscores identically in Z flags. That's along for the ride in this PR as the last commit, and is included here because it makes for more idiomatic toml file keys (`print_im_a_teapot = yes` vs `print-im-a-teapot = yes`). Please let me know if y'all would prefer that be in a separate PR, or not happen at all.
# Test Plan
Apologies if I've missed anything -- this is my first cargo contrib and I've tried to hew to the contributing guide. If I've slipped up, please let me know how and I'll fix it.
NB. My linux machine doesn't have multilib set up, so I disabled cross tests.
* `cargo test` passes for each commit in the stack
* I formatted each commit in the stack with `rustfmt`
* New tests are included alongside the relevant change for each change
* I've validated each test by locally undoing the code change they support and confirming failure.
* The CLI wins, for both enable and disabling Z flags, as you'd expect.
Keys in `unstable` which do not correspond with a Z flag will trigger an error indicating the invalid flag came from a config file read:
```
Invalid [unstable] entry in Cargo config
Caused by:
unknown `-Z` flag specified: an-invalid-flag
```
If you'd like to see a test case which isn't represented here, I'm happy to add it. Just let me know.
# Documentation
I've included commits in this stack updating the only docs page that seemed relevant to me, skimming the book -- `unstable.md`.
These tests demonstrate the current failure mode around
overlapping env keys and inner structs. To some extent this
is a limitation of mapping on to environment variables with
an underscore as both the namespace separator and the
substitute for dashes in flag names in that the mapping is
not strictly one-to-one.
Add support for rustc's `-Z terminal-width`.
This PR continues the work started in #7315, adding support for rustc's `-Z terminal-width` flag, which is used to trim diagnostic output to fit within the current terminal and was added in rust-lang/rust#63402 (with JSON emitter support in rust-lang/rust#73763).
At the time of writing, rust-lang/rust#73763 isn't in nightly, so the test added in this PR will fail, but it should pass tomorrow (I've confirmed that it works with a local rustc build).
cc @estebank
This commit adds support for rustc's `-Z terminal-width` flag, which is
used to trim diagnostic output to fit within the current terminal.
Co-authored-by: David Wood <david@davidtw.co>
Signed-off-by: David Wood <david@davidtw.co>
The current behavior, with the default-false workaround in place,
is not able to identify extra members of the `unstable` table, so
it can't error on unexpected members.
I'll add this test back in with a Deserializer refactor to clean up
the extra logic added to CliUnstable.
Tests are currently failing, looks like there's something in the
Deserializer impl that's forcing field-missing errors even when
the serde `default` annotation is applied.
Obviously this only works with nightlies and all
that, but if you're e.g. testing resolver v2, want
to always have Ztiming on in your workspace, or
something like that, this makes it easier to do.
Add support for deserializing enums in config files
Implements `deserialize_enum` functionality to allow config options which are Rust enums.
@ehuss The code currently has some `todo!`s because I'm not sure how the custom `Deserializer` is supposed to do error handling.
Fixes#8450
Write GNU tar files, supporting long names.
Fixes#8452
If I understand the previous bugs correctly, long trees packaged using Cargo with this patch will be misinterpreted by Cargo from before Jan 2016. (Without this patch, they can't be written at all.)
To me that seems like long enough ago that it's safe to land this now.
- [x] Add a test.
Exclude the target directory from backups using CACHEDIR.TAG
This patch follows the lead of #4386 (which excludes target directories
from Time Machine backups) and is motived by the same reasons listen
in #3884. CACHEDIR.TAG is an OS-independent mechanism supported by Borg,
restic, GNU Tar and other backup/archiving solutions.
See https://bford.info/cachedir/ for more information about the
specification. This has been discussed in Rust Internals earlier this
year[1] and it seems like it's an uncontroversial improvement so I went
ahead with the patch.
One thing I'm wondering is whether this should maybe cover the whole main target directory (right now it applies to `target/debug`, `target/release` etc. but not to target root).
[1] https://internals.rust-lang.org/t/pre-rfc-put-cachedir-tag-into-target/12262/11
This is following the discussion on GitHub. The doc tests are no longer
necessary because Layout::new() creates CACHEDIR.TAG directly in target
root, no doc-specific code is necessary anymore.
Parse `# env-dep` directives in dep-info files
This commit updates Cargo's parsing of rustc's dep-info files to account
for changes made upstream in rust-lang/rust#71858. This means that if
`env!` or `option_env!` is used in crate files Cargo will correctly
rebuild the crate if the env var changes.
Closes#8417
This commit updates Cargo's parsing of rustc's dep-info files to account
for changes made upstream in rust-lang/rust#71858. This means that if
`env!` or `option_env!` is used in crate files Cargo will correctly
rebuild the crate if the env var changes.
Closes#8417
Improve git error messages a bit
This commit is targeted at further improving the error messages
generated from git errors. For authentication errors the actual URL
fetched is now printed out as well if it's different from the original
URL. This should help handle `insteadOf` logic where SSH urls are used
instead of HTTPS urls and users can know to track that down.
Otherwise the logic about recommending `net.git-fetch-with-cli` was
tweaked a bit and moved to the same location as the rest of our error
reporting.
Note that a change piggy-backed here as well is that `Caused by:` errors
are now automatically all tabbed over a bit instead of only having the
first line tabbed over. This required a good number of tests to be
updated, but it's just an updated in renderings.
This commit is targeted at further improving the error messages
generated from git errors. For authentication errors the actual URL
fetched is now printed out as well if it's different from the original
URL. This should help handle `insteadOf` logic where SSH urls are used
instead of HTTPS urls and users can know to track that down.
Otherwise the logic about recommending `net.git-fetch-with-cli` was
tweaked a bit and moved to the same location as the rest of our error
reporting.
Note that a change piggy-backed here as well is that `Caused by:` errors
are now automatically all tabbed over a bit instead of only having the
first line tabbed over. This required a good number of tests to be
updated, but it's just an updated in renderings.
Improve support for non-`master` main branches
This commit improves Cargo's support for git repositories whose "main
branch" is not called `master`. Cargo currently pretty liberally assumes
that if nothing else about a git repository is specified then `master`
is the branch name to use. Instead now Cargo has a fourth option as the
desired reference of a repository named `DefaultBranch`. Cargo doesn't
know anything about the actual name of the default branch, it just
updates how git references are fetched internally.
This commit is motivated by news that GitHub is likely to switch away
from the default branch being named `master` in the near future. It
would be a bit of a bummer if from now on everyone had to type
`branch = '...'`, so this tries to improve that!
Closes#3517
Add support for `workspace.metadata` table
Implements feature request #8309
Additionally includes the information in the output of "cargo metadata" through a new top-level field `metadata`, similar to the per-package `metadata` field
Fix order-dependent feature resolution.
There is a situation where if you have `pkg/feature` syntax, and `pkg` is an optional dependency, but also a dev-dependency, and the dev-dependency appears before the (optional) normal dependency in the summary, then the optional dependency would not get activated. This is because the feature code used `find` to get the first entry.
Fixes#8394
This commit improves Cargo's support for git repositories whose "main
branch" is not called `master`. Cargo currently pretty liberally assumes
that if nothing else about a git repository is specified then `master`
is the branch name to use. Instead now Cargo has a fourth option as the
desired reference of a repository named `DefaultBranch`. Cargo doesn't
know anything about the actual name of the default branch, it just
updates how git references are fetched internally.
This commit is motivated by news that GitHub is likely to switch away
from the default branch being named `master` in the near future. It
would be a bit of a bummer if from now on everyone had to type
`branch = '...'`, so this tries to improve that!
This commit refactors various logic of the git source internals to
ensure that if we have a locked revision that we plumb the desired
branch/tag all the way through to the `fetch`. Previously we'd switch to
`Rev` very early on, but the fetching logic for `Rev` is very eager and
fetches too much, so instead we only resolve the locked revision later
on.
Internally this does some various refactoring to try to make various
bits and pieces of logic a bit easyer to grok, although it's still
perhaps not the cleanest implementation.
Support linker with -Zdoctest-xcompile.
This adds support for `-Clinker` with `-Zdoctest-xcompile`.
I'm not entirely sure how `-Zdoctest-xcompile` was supposed to work without setting the linker. I tested this with std on arm-unknown-linux-gnueabihf with qemu. It seems to work (although it was quite slow).
Closes#7529.
Fix doctests not running with --target=HOST.
There was a regression in #8167 where `cargo test --target=$HOST` stopped running doctests. This caused doctests to silently stop running in rust-lang/rust (https://github.com/rust-lang/rust/issues/73286). This PR restores the original behavior where `--target=$HOST` behaves as-if it is a normal host test.
There was a discussion about this at https://github.com/rust-lang/cargo/pull/8167#pullrequestreview-402196323, but I think I let it slip through the cracks.
Some LTO fixes.
This reworks the LTO computation a little to address a few issues:
- `cargo build` in a project with both a lib and bin would not engage the optimization introduced in #8192 where the lib *should* be compiled with `-C linker-plugin-lto` (bitcode only). This happened because the old code was starting root units as `Lto::None`. The solution here is to conditionally choose the starting Lto for roots.
- A project with a dylib dependency would fail to build. It was building the dylib with `-C linker-plugin-lto` which is not valid.
- A project with a bin/lib would build the lib differently based on whether or not it was selected. This changes it so that the lib is built the same. See `lto::between_builds`, where the second build the lib is now fresh.
- Tests/benchmarks of a `lib` target will now support LTO.
- Treats example libs a little more consistently as regular libs.
I scattered some comments throughout, hopefully it's not too difficult to follow.
Closes#8337
Default values for `readme` if not specified
If the a value for `readme` is not specified in Cargo.toml, we will now check for the existence of files named `README.md`, `README.txt` or `README`. If one does exist, the name of that file will be defaulted in to the manifest for the project.
This behavior can be suppressed if `readme` is set to `false`.
Closes#8133
Add environment variables to identify the binary and crate name
Closes#8251
This adds `CARGO_BIN_NAME` and `CARGO_CRATE_NAME` to rustc/rustdoc process env.
`CARGO_BIN_NAME` is added for binary compilation units, `CARGO_CRATE_NAME` is added for binary and library units.
The `build::crate_env_vars` test was updated to test for this. The test is currently only checking behavior for the binary compile unit.
Documentation was updated to reflect the added environment variables.
The manifest has a few different ways of specifying whether a crate is a
procedural macro, and there's a `TomlTarget::proc_macro()` method to
unify these various lines. Unfortunately though we had a bug where one
location forgot to call the method and read the raw field! This led to
surprising behavior where the different ways to specify a proc macro
would have subtly different changes in behavior. The fix here in this PR
is to ensure that we access the property always via the method.
Closes#8315
Warn if using hash in git URL, Fixes#8241
This fixes an issue where if the user wants to set the git rev but doesn't know how and as results tries to set the ref in the url hash as also shown when downloading the dependency.
Now cargo returns a warning notifying the user about the correct way to set the ref.
Fixes#8241
reset lockfile information between resolutions
#8249 pointed out that some kind of lockfile data was leaking between calls to the resolver. @ehuss made a reproducing test case. This PR resets the `LockedMap` data structure when calling `register_previous_locks`.
lets see if CI likes it.
fix#8249
Add support for rustdoc root URL mappings.
This adds an experimental configuration setting to allow Cargo to pass the `--extern-html-root-url` flag to rustdoc. This flag allows rustdoc to link to other locations when a dependency is not locally documented. See the documentation in `unstable.md` for more details.
There are some known issues with this implementation:
* Rustdoc doesn't seem to know much about renamed dependencies. The links it generates are to the package name, not the renamed name. The code is written to pass in package names, but if there are multiple dependencies to the same package, it won't work properly.
* Similarly, if there are multiple versions of the same package within the dep graph, rustdoc will only link to one of them. To fix this, Cargo would need to pass metadata info into rustdoc (such as the package version).
* If a dependency is built with different features than what is on docs.rs, some links may break.
* This explodes the command-line length significantly. Before stabilizing, we may want to consider addressing that. I'm not sure if it would make sense to change rustdoc's interface, or to use response files?
* This does not pass mappings for transitive dependencies. This normally isn't an issue, but can arise for re-exports (see the `alt_registry` test for an example). I'm not sure if this is a bug in rustdoc or not (there is a large number of issues regarding reexports and rustdoc). Cargo could include these, but this would make the command-line length even longer. Not sure what to do here.
* The config value does not support environment variables. This would be very difficult to support, because Cargo doesn't retain the registry name in `SourceId`. I looked into fixing that, but it is very difficult, and hard to make it reliable.
I have tried to consider future changes in this design, to ensure it doesn't make them more difficult:
* Single-tab browsing. This would be a mode where the std docs are merged with the local crate's docs so that the std docs are shown in the same place (and included in the index). This could be expressed with something like `doc.extern-map.std = "include"` or something like that. (Or maybe just use build-std?)
* Direct-dependencies only. Often transitive dependencies aren't that interesting, and take up a lot of space in the output, and clog the search index. Some users want the ability to (locally) document their package + direct dependencies only. I think this could be implemented with some kind of command-line flag, perhaps with a config setting in the `[doc]` table. `--extern-html-root-url` flag will automatically handle second-level dependencies.
* Manual-exclusions. Sometimes there are specific dependencies that are very expensive to document locally, but you still want everything else. I think this could be implemented with a command-line flag (`--exclude winapi`?), and the rustdoc-map feature would automatically link those excluded crates' items to docs.rs. This could also be added to the `[doc]` table.
We can also consider at any time to change the defaults (such as making `crates-io = "https://docs.rs"` the default). It could also potentially auto-detect `std = "local"`, although rustdoc could do the same internally.
Closes#6279
This stops using `to_string` as a proxy for this now-provided precise API.
This reverts commit b71927224f and bupms the
dependency version in Cargo.toml.
CARGO_BIN_NAME is added for binary compilation units, CARGO_CRATE_NAME is added for binary and library units.
The `crate_env_vars` test was updated to test for this. The test is currently only checking behavior for the binary compile unit.
Provide better error messages for a bad `patch`.
This attempts to provide more user-friendly error messages for some situations with a bad `patch`. This is a follow-up to #8243.
I think this more or less covers all the issues from #4678. I imagine there are other corner cases, but those will need to wait for another day. The main one I can think of is when the patch location is missing required features. Today you get a "blah was not used in the crate graph." warning, with some suggestions added in #6470, but it doesn't actually check if there is a feature mismatch.
Closes#4678
Try installing exact versions before updating
When an exact version is being installed, if we already have that
version from the index, we don't need to update the index before
installing it. Don't do this if it's not an exact version, because the
update may find us a newer version.
This is particularly useful for scripts which unconditionally run
`cargo install some-crate --version=1.2.3`. Before install-update, I
wrote a crate to do this
(https://crates.io/crates/cargo-ensure-installed) which I'm trying to
replace with just `cargo install`, but the extra latency of updating the
index for a no-op is noticeable.
Add option to strip binaries
This PR adds a Cargo option for stripping symbols from generated binaries.
This is based on the `-Z strip` flag for `rustc`, which has been [recently implemented](https://github.com/rust-lang/rust/issues/71757).
Notes for reviewers: I'm not entirely sure how we test this, I've created a crate locally and it seems to be working.
Supersedes my previous work in #8191.
Fixes#3483.
Handle LTO with an rlib/cdylib crate type
In the case that LTO is happening but we're also generating a
cdylib/rlib simultatneously that means that the final artifact will use
the rlib but the cdylib still needs to be produced. To get this to work
the cdylib's dependency tree needs to be compiled with embedded bitcode.
The cdylib itself will be linked with the linker because we can't LTO an
rlib+cdylib compilation, but the final executable will need to load
bitcode from all its deps.
This is meant to address rust-lang/rust#72268
In the case that LTO is happening but we're also generating a
cdylib/rlib simultatneously that means that the final artifact will use
the rlib but the cdylib still needs to be produced. To get this to work
the cdylib's dependency tree needs to be compiled with embedded bitcode.
The cdylib itself will be linked with the linker because we can't LTO an
rlib+cdylib compilation, but the final executable will need to load
bitcode from all its deps.
This is meant to address rust-lang/rust#72268
Rephrased error message for disallowed sections in virtual workspace
I changed the error message from `virtual manifests do not specify [target]` to `This virtual manifest specifies a [target] section, which is not allowed` because the old one confused me.
I encountered it while hacking on Rustc in CLion. I made a change to the `Cargo.toml` I had copied from StackOverflow and after the next restart code analysis and go-to-definition stopped working. After some fiddling, I tracked that down to my change in `Cargo.toml`. It took me a while because, like I said, the error message from CLion didn't make sense to me.
```
19:13 Cargo project update failed:
Execution failed (exit code 101).
/home/julian/.cargo/bin/cargo metadata --verbose --format-version 1 --all-features
stdout : error: failed to parse manifest at `/home/julian/Dokumente/Rust/Compiler/rust/Cargo.toml`
Caused by:
virtual manifests do not specify [target]
stderr :
```
I hope this change avoids future confusion :)
With `opt-level = "3"` this previously said:
must be an integer, `z`, or `s`, but found: 3 for ...
The error message doesn't make that super clear.
This should now be a bit more clear.
Fixes#8234
Turns out a case I thought didn't happen can indeed happen. Units may
depend on other units which are LTO-able because integration tests can
depend on binaries. Handle that case internally and remove a few panics.
Closes#8223
Rework rustc output file tracking.
This is some cleanup around how Cargo computes the outputs from rustc, and changes how `cargo clean -p` works. This also includes some minor observable differences detailed below.
**clean -p changes**
Previously `cargo clean -p` would build the unit graph and attempt to guess what all the filename hashes are. This has several drawbacks. It incorrectly guesses the hashes in many cases (such as different features). It also tends to cause bugs because it passes every permutation of every setting into Cargo's internals, which may not be prepared to accept unsupported combinations like "test a build-script".
This new implementation uses a strategy of querying rustc for the filename prefix/suffix, and then deletes the files using globs.
`cargo clean -p` should now be more complete in deleting a package's artifacts, for example:
- Now deletes incremental files.
- Deletes dep-info files (both rustc and cargo).
- Handles changes in profiles, features, (anything in the hash).
- Covers a few more files (for example, dSYM in some cases, etc.). Should delete almost all files for most targets.
**Internal changes**
There are a bunch of internal changes to make Cargo do a better job of tracking the outputs from rustc, and to make the code easier to understand:
- The list of output files is now solely computed in `TargetInfo`. The files which are uplifted are solely computed in `CompilationFiles::uplift_to`. Previously both responsibilities were awkwardly spread in both locations.
- Whether or not a file should have a hyphen or underscore is now determined in one place (`FileType::should_replace_hyphens`).
- Added `CrateType` to replace `LibKind`, to avoid usage of strings, and to use the same structure for all of the target kinds.
- Added `FileFlavor::Rmeta` to be more explicit as to which output files are ".rmeta". (Previously the `Linkable{rmeta}` flag was specific for pipelining, and rmeta was `false` for things like `cargo check`, which was a bit hard to deal with.)
- Removed hyphen/underscore renaming in `rustc`. This was mostly unused, because it did not consider hashes in the filename, so it only applied to binaries without hashes, which is essentially just wasm32 and msvc. This hyphen/underscore translation still happens during "uplift".
- Changed it so that `Metadata` is always computed for every unit. The logic of whether or not something should use it is moved to `should_use_metadata`. I didn't realize that multiple units were sharing the same fingerprint directory (when they don't have a hash), which caused some bugs (like bad output caching).
**Behavioral changes**
Cargo now does more complete tracking of the files generated by rustc (and the linker). This means that some files will now be uplifted that previously weren't. It also means they will show up in the artifact JSON notifications. The newly tracked files are:
- `.exp` export files for Windows MSVC dynamic libraries. I don't know if these are actually useful (nobody has asked for them AFAIK). Presumably the linker creates them for some reason. Note that lld *doesn't* generate these files, this is only link.exe.
- Proc-macros on Windows track import/export files.
- All targets (like tests, etc.) that generate separate debug files (pdb/dSYM) are tracked.
- Added `.map` files for wasm32-unknown-emscripten.
Some other subtle changes:
- A binary example with a hyphen on Windows MSVC will now show up as `examples/foo_bar.exe` and `examples/foo-bar.exe`. Previously Cargo would just rename it to contain the hyphen. This is a consequence of simplifying the code, and I was reluctant to add a special case for this very narrow situation.
- Example libs now follow the same rules for hyphen/underscore translation as normal libs (they will now use underscores).
- Fingerprint changes:
- Fingerprint files no longer have a hash in them. Their parent directory already contained the hash.
- The fingerprint filename now uses hyphens instead of converting to underscores.
- The fingerprint directory is now separated even if the unit doesn't use Metadata, to fix a caching bug.
- macOS: dSYM is uplifted for all dynamic libraries (dylib/cdylib/proc-macro) and for build-script-build (in case someone wants to debug a build script?).
**Notes**
- I suspect that the implementation of `clean -p` may change again in the future. If and when Cargo implements some kind of on-disk database that tracks artifacts (for the purpose of garbage collection), then `cargo clean -p` can be rewritten to use that mechanism if appropriate.
- The `build_script_build` incremental directory isn't deleted because Cargo doesn't know which one belongs to which package. I'm uncertain if that's reasonably fixable. The only option I've thought of is to place each package's incremental output in a separate directory.
- Should Cargo use `globset` to handle non-utf-8 filenames? I suspect that Cargo's support for non-utf-8 names is pretty poor, so I'm uncertain how important that is.
Closes#8149Closes#6937Closes#5788Closes#5375Closes#3530
When a unit does not have Metadata, the computation of fingerprints
depends on reusing the same fingerprint file to detect if the
output needs to be rebuilt. The previous change that put each unit's
fingerprint into a separate directory was wrong, and broke change
detection in this case (for example, executables on MSVC).
Instead, use a different approach to deal with compiler output caching
by using the same naming convention as the fingerprint file itself.
Don't force rustc to do codegen for LTO builds
This commit updates Cargo's implementation of LTO builds to do less work
and hopefully be faster when doing a cold build. Additionaly this should
save space on disk! The general idea is that the compiler does not need
object files if it's only going to perform LTO with some artifacts. In
this case all rustc needs to do is load bitcode from dependencies. This
means that if you're doing an LTO build generating object code for
intermediate dependencies is just wasted time!
Here Cargo is updated with more intrusive knowledge about LTO. Cargo
will now analyze the dependency graph to figure out which crates are
being compiled with LTO, and then it will figure out which dependencies
only need to have bitcode in them. Pure-bitcode artifacts are emitted
with the `-Clinker-plugin-lto` flag. Some artifacts are still used in
multiple scenarios (such as those shared between build scripts and final
artifacts), so those are not compiled with `-Clinker-plugin-lto` since
the linker is not guaranteed to know how to perform LTO.
This functionality was recently implemented in rust-lang/rust#71528
where rustc is now capable of reading bitcode from `-Clinker-plugin-lto`
rlibs. Previously rustc would only read its own format of bitcode, but
this has now been extended! This support is now on nightly, hence this
PR.
This commit updates Cargo's implementation of LTO builds to do less work
and hopefully be faster when doing a cold build. Additionaly this should
save space on disk! The general idea is that the compiler does not need
object files if it's only going to perform LTO with some artifacts. In
this case all rustc needs to do is load bitcode from dependencies. This
means that if you're doing an LTO build generating object code for
intermediate dependencies is just wasted time!
Here Cargo is updated with more intrusive knowledge about LTO. Cargo
will now analyze the dependency graph to figure out which crates are
being compiled with LTO, and then it will figure out which dependencies
only need to have bitcode in them. Pure-bitcode artifacts are emitted
with the `-Clinker-plugin-lto` flag. Some artifacts are still used in
multiple scenarios (such as those shared between build scripts and final
artifacts), so those are not compiled with `-Clinker-plugin-lto` since
the linker is not guaranteed to know how to perform LTO.
This functionality was recently implemented in rust-lang/rust#71528
where rustc is now capable of reading bitcode from `-Clinker-plugin-lto`
rlibs. Previously rustc would only read its own format of bitcode, but
this has now been extended! This support is now on nightly, hence this
PR.
Hint git-fetch-with-cli on git errors
Our team has struggled with making Cargo git dependencies work in CI, until learning about this setting.
Cargo doesn't have a dedicated system for error hints like `rustc`, so I've stuffed the hint into the error message.
This commit refactors the internals of Cargo to no longer have a
singular `--target` flag (and singular `requested_target` kind throught)
but to instead have a list. The semantics of multiple `--target` flags
is to build the selected targets for each of the input `--target` flag
inputs.
For now this is gated behind `-Zmultitarget` as an unstable features,
since I'm not entirely sure this is the interface we want. In general
it'd be great if we had a way to simply specify `Unit` structures of
what to build on the CLI, but we're in general very far away from that,
so I figured that this is probably sufficient at least for testing for
now.
cc #8156
build-std: Don't treat std like a "local" package.
This changes it so that build-std will not treat the std crates like a "local" package. This has the following changes:
- std does not build with incremental. This generally shouldn't be needed, and incremental has various bugs with std crates.
- Cargo's dep-info fingerprint tracking will not track the std crate sources, these are tracked via other means.
- Cargo's `.d` dep-info file does not include std crate sources.
- Lints are capped to "allow" for std crates, and warnings are not shown by default.
Closes https://github.com/rust-lang/wg-cargo-std-aware/issues/44
Closes https://github.com/rust-lang/wg-cargo-std-aware/issues/55
Use associated constants directly on primitive types instead of modules
This PR is in no way critical. It's more of a code cleanup. It comes as a result of me making https://github.com/rust-lang/rust/pull/70857 and search-and-replacing all usage of the soft-deprecated ways of reaching primitive type constants.
It makes the code slightly shorter, that's basically it. And showcases the recommended way of reaching these consts on new code :)
Fix flaky linking_interrupted test.
The `linking_interrupted` test is a little flaky, and can randomly fail.
On windows-gnu, the last call to `cargo test` was failing because it was hitting a file lock. Presumably this is because the `kill` had not finished? Added a wait to ensure that cargo has exited.
Also, sometimes on Windows, the rustc process gets killed. Not sure why that doesn't happen all the time, maybe related to the race mentioned above.
Finally, on macOS, I was noticing some errors where I think the rustc process had not yet exited, and deleted the `*.o` files while the next cargo build was running.
Add windows-gnu CI and fix tests
One remaining failure:
```
---- features::feature_off_dylib stdout ----
running `d:\a\1\s\target\debug\cargo.exe build --features f1`
running `d:\a\1\s\target\debug\cargo.exe run -p bar`
thread 'features::feature_off_dylib' panicked at '
Expected: execs
but: exited with exit code: 101
--- stdout
--- stderr
Compiling foo v0.0.1 (D:\a\1\s\target\cit\t663\foo)
Compiling bar v0.0.1 (D:\a\1\s\target\cit\t663\foo\bar)
Finished dev [unoptimized + debuginfo] target(s) in 0.69s
Running `target\debug\bar.exe`
thread 'main' panicked at 'assertion failed: `(left == right)`
left: `"f1"`,
right: `"no f1"`', bar\src\main.rs:5:17
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
error: process didn't exit successfully: `target\debug\bar.exe` (exit code: 101)
', crates\cargo-test-support\src\lib.rs:833:13
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
failures:
features::feature_off_dylib
```
I disassembled the dylibs and `cargo run -p bar` correctly rebuilt it inside `target/debug/deps/` but did not copy it to `target/debug`. To further confirm, calling `cp target/debug/deps/foo.dll target/debug/` manually solved the issue.
Any idea?
----
I left `FIXME` in places where import lib should be added with https://github.com/rust-lang/cargo/pull/6875.
`TOOLCHAIN: nightly-x86_64-pc-windows-gnu` can be replaced with beta on Thursday.
Several updates to token/index handling.
This attempts to tighten up the usage of token/index handling, to prevent accidental leakage of the crates.io token.
* Make `registry.index` config a hard error. This was deprecated 4 years ago in #2857, and removing it helps simplify things.
* Don't allow both `--index` and `--registry` to be specified at the same time. Otherwise `--index` was being silently ignored.
* `registry.token` is not allowed to be used with the `--index` flag. The intent here is to avoid possibly leaking a crates.io token to another host.
* Added a warning if source replacement is used and the token is loaded from `registry.token`.
Closes#6545
Add `resolver` opt-in for new feature resolver.
This adds a new `resolver` field to `Cargo.toml` to provide an opt-in mechanism for backwards-incompatible resolver changes. This enables the new feature resolver and `-Zpackage-features`. See `unstable.md` for documentation.
cc #8088
