Show transfer rate when fetching/updating registry index
Possibly fixes#8483.
To avoid blinking too frequently, update rate is throttled by one second.
I am not sure how to write tests for it 😂
<img width="896" alt="image" src="https://user-images.githubusercontent.com/14314532/115879831-ac62fb00-a47c-11eb-9b12-735ce8192ebe.png">
# Updated (2020-04-28)
Current looking
```
Updating crates.io index
Fetch [==> ] 14.50%, 258.45KiB/s
Updating crates.io index
Fetch [======> ] 40.50%, (1234/282342) resolving deltas
```
Fix collision doc tests randomly failing.
This fixes some tests that were randomly failing on CI. The cause is that #9419 added a remove_dir_all on the `doc` directory. However, if two jobs are trying to write to that directory at the same time, this can cause errors. The failure rate is low (a little over 1%), and I was unable to reproduce locally (only on GitHub's CI and only on the Windows job).
The solution is to run the jobs with -j1 so they run serially.
I only saw errors for `collision_doc_sources`, but to be on the safe side I added j1 to similar tests.
Fix dep-info files emitting paths relative to deps' roots
Sample `shoo.d` file prior to this change is below, note the `build.rs`
at the end, which was not from my package.
From booping the debugger, I found this was coming from
`compiler_builtins`. This is not really their bug though: if a build.rs
asks for rerun-if-changed on some crate relative path, this will happen
in general. So I've fixed it in Cargo and added a test to prevent it
regressing.
```
target/riscv64imac-mu-shoo-elf/release/shoo: /home/jade/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/../../stdarch/crates/core_arch/src/core_arch_docs.md /home/jade/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/../../stdarch/crates/core_arch/src/macros.rs /home/jade/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/../../stdarch/crates/core_arch/src/mod.rs /home/jade/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/../../stdarch/crates/core_arch/src/simd.rs /home/jade/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/../../stdarch/crates/core_arch/src/simd_llvm.rs crates/build_bits/src/lib.rs shoo/src/main.rs shoo/src/task.rs shoo/src/vectors.s build.rs
```
This change fixes it so it's like:
```
target/riscv64imac-mu-shoo-elf/release/shoo: /home/jade/.cargo/registry/src/github.com-1ecc6299db9ec823/compiler_builtins-0.1.39/build.rs /home/jade/.cargo/registry/src/github.com-1ecc6299db9ec823/log-0.4.14/build.rs /home/jade/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/../../stdarch/crates/core_arch/src/core_arch_docs.md /home/jade/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/../../stdarch/crates/core_arch/src/macros.rs /home/jade/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/../../stdarch/crates/core_arch/src/mod.rs /home/jade/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/../../stdarch/crates/core_arch/src/simd.rs /home/jade/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/../../stdarch/crates/core_arch/src/simd_llvm.rs crates/build_bits/src/lib.rs shoo/src/main.rs shoo/src/task.rs shoo/src/vectors.s
```
Fix rebuild issues with rustdoc.
This fixes two issues related to rebuilds with rustdoc:
* Switching features when running `cargo doc` would result in Cargo not rebuilding the documentation. This is because it was keeping the fingerprints in separate directories based on the features used. However, the rustdoc output isn't keyed off the metadata hash, so although the old fingerprint seemed "up to date", in reality the documentation was rewritten and needs to be rebuilt. The solution is to use a simplified hash for the fingerprint directory name.
* Removing items does not remove the files from the doc directory. This changes it to clear the package's doc directory before running rustdoc, to ensure any stale files are removed.
I'm a little concerned about potential performance impact of running `remove_dir_all`, but I think it shouldn't be too bad?
Fixes#7370
Always use full metadata hash for -C metadata.
This changes it so that cargo always uses the full metadata hash for `-C metadata`. This ensures that even if a unit isn't using `-C extra-filename` that the symbol hashing uses all the fields used in the other cases.
This fixes an issue on macOS where a combination of split-debuginfo and incremental caused the same `.o` filenames to be used, which caused corruption in the incremental cache (see issue for details).
Fixes#9353.
Sample `shoo.d` file prior to this change is below, note the `build.rs`
at the end, which was not from my package.
From booping the debugger, I found this was coming from
`compiler_builtins`. This is not really their bug though: if a build.rs
asks for rerun-if-changed on some crate relative path, this will happen
in general. So I've fixed it in Cargo and added a test to prevent it
regressing.
```
target/riscv64imac-mu-shoo-elf/release/shoo: /home/jade/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/../../stdarch/crates/core_arch/src/core_arch_docs.md /home/jade/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/../../stdarch/crates/core_arch/src/macros.rs /home/jade/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/../../stdarch/crates/core_arch/src/mod.rs /home/jade/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/../../stdarch/crates/core_arch/src/simd.rs /home/jade/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/../../stdarch/crates/core_arch/src/simd_llvm.rs crates/build_bits/src/lib.rs shoo/src/main.rs shoo/src/task.rs shoo/src/vectors.s build.rs
```
This change fixes it so it's like:
```
target/riscv64imac-mu-shoo-elf/release/shoo: /home/jade/.cargo/registry/src/github.com-1ecc6299db9ec823/compiler_builtins-0.1.39/build.rs /home/jade/.cargo/registry/src/github.com-1ecc6299db9ec823/log-0.4.14/build.rs /home/jade/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/../../stdarch/crates/core_arch/src/core_arch_docs.md /home/jade/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/../../stdarch/crates/core_arch/src/macros.rs /home/jade/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/../../stdarch/crates/core_arch/src/mod.rs /home/jade/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/../../stdarch/crates/core_arch/src/simd.rs /home/jade/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/../../stdarch/crates/core_arch/src/simd_llvm.rs crates/build_bits/src/lib.rs shoo/src/main.rs shoo/src/task.rs shoo/src/vectors.s
```
Expose build.target .cargo/config setting as packages.target in Cargo.toml
Hey!
I'm trying to do my first cargo contribution by implementing per-crate target settings as per [the irlo thread](https://internals.rust-lang.org/t/proposal-move-some-cargo-config-settings-to-cargo-toml/13336) ; and I think I have a draft that looks good-ish (the root units returned by `generate_targets` have the right kinds set).
Closes#7004
**_Edit: the below problem description is now solved in the latest version of this PR, please ignore_**
But for some reason running on a test project now blocks on `Blocking waiting for file lock on build directory` and I have literally no idea how my changes could trigger this… would anyone have an idea of how the changes could lead to infinitely blocking there? (I already tried cargo clean just in case and it didn't appear to help)
FWIW, the output that looks hopeful to me is, on my testbed workspace:
```
Root units [out of generate_targets] are [...]:
- package ‘smtp-client’ with kind ‘Target(CompileTarget { name: "x86_64-unknown-linux-gnu" })’
- package ‘smtp-server’ with kind ‘Target(CompileTarget { name: "x86_64-unknown-linux-gnu" })’
- package ‘yuubind-config’ with kind ‘Target(CompileTarget { name: "x86_64-unknown-linux-gnu" })’
- package ‘smtp-message-fuzz’ with kind ‘Target(CompileTarget { name: "x86_64-unknown-linux-gnu" })’
- package ‘yuubind-rpc’ with kind ‘Target(CompileTarget { name: "x86_64-unknown-linux-gnu" })’
- package ‘yuubind-config-example’ with kind ‘Target(CompileTarget { name: "x86_64-unknown-linux-gnu" })’
- package ‘smtp-message-fuzz’ with kind ‘Target(CompileTarget { name: "x86_64-unknown-linux-gnu" })’
- package ‘yuubind-config-example’ with kind ‘Target(CompileTarget { name: "wasm32-unknown-unknown" })’
- package ‘smtp-queue’ with kind ‘Target(CompileTarget { name: "x86_64-unknown-linux-gnu" })’
- package ‘smtp-message-fuzz’ with kind ‘Target(CompileTarget { name: "x86_64-unknown-linux-gnu" })’
- package ‘smtp-message’ with kind ‘Target(CompileTarget { name: "x86_64-unknown-linux-gnu" })’
- package ‘smtp-server-fuzz’ with kind ‘Target(CompileTarget { name: "x86_64-unknown-linux-gnu" })’
- package ‘yuubind-config’ with kind ‘Target(CompileTarget { name: "wasm32-unknown-unknown" })’
- package ‘yuubind’ with kind ‘Target(CompileTarget { name: "x86_64-unknown-linux-gnu" })’
- package ‘smtp-queue-fs’ with kind ‘Target(CompileTarget { name: "x86_64-unknown-linux-gnu" })’
```
(where both `yuubind-config` and `yuubind-config-example` are being configured to be `wasm32-unknown-unknown` and the other ones stay as host).
Interestingly enough, if I remove the `target` setting from `yuubind-config` (and leave it on `yuubind-config-example`) then it does no longer block on waiting for file lock on build directory, even though it does not actually compile with `wasm32-unknown-unknown`. And it does appear to correctly build yuubind-config-example as wasm32.
My investigation shows that it appears to happen iff there is a package with `package.target` being set that has dependencies.
This most likely is a bug in my code (eg. I build only the root units and not the whole unit graph maybe?), and am going to keep investigating it as such, but maybe someone would already know how dependency resolution could interact with build lock acquisition and give me hints?
Anyway, thank you all for all you do cargo!
Some changes to rustdoc fingerprint checking.
#8640 introduced a check which deletes the `doc` directory if cargo detects it has stale contents from a different toolchain version. Rustdoc has some shared files (js and css for example) that can get corrupted between versions. Unfortunately that caused some problems with rustbuild which does a few unusual things. Rustbuild will:
* Create the `doc` directory before running `cargo doc` and places a `.stamp` file inside it.
* Creates symlinks of the `doc` directory so that they can be shared across different target directories (in particular, between rustc and rustdoc).
In order to address these issues, this PR does several things:
* Adds `-Z skip-rustdoc-fingerprint` to disable the `doc` clearing behavior.
* Don't delete the `doc` directory if the rustdoc fingerprint is missing. This is intended to help with the scenario where the user creates a `doc` directory ahead of time with pre-existing contents before the first build. The downside is that cargo will not be able to protect against switching from pre-1.53 to post-1.53.
* Don't delete the `doc` directory itself (just its contents). This should help if the user created the `doc` directory as a symlink to somewhere else.
* Don't delete hidden files in the `doc` directory. This isn't something that rustdoc creates.
Only the `-Z` change is needed for rustbuild. The others I figured I'd include just to be on the safe side in case there are other users doing unusual things (and I had already written them thinking they would work for rustbuild). Hopefully the rustbuild `.stamp` mechanism will be enough protection there.
Fixes#9336
This is a hidden flag intended to only be used by rustbuild which will
skip the rustdoc fingerprint check. rustbuild does some funky things
with sharing the doc directory across multiple target directories via
symlinks, and that causes problems where after building in one target
directory, then switching to the second one, it will clear the contents.
In some cases, the directory may actually be a symlink created by the
user, and we don't want to delete it. Also, skip any hidden files added
by the user as well.
Restore crates.io's `SourceId` hash value to before
This commit restores the hash value of the crates.io `SourceId` to what
it was before #9384. In #9384 the enum variants of `SourceKind` were
reordered which accidentally changed the hash value of the `SourceId`
for crates.io. A change here means that users with a new version of
Cargo will have to redownload the index and all crates, which is
something that we strive to avoid forcing.
In changing this, though, it required a manual implementation of `Ord`
to still contain the actual fix from #9384 which is to sort `SourceKind`
differently from how it's defined. I was curious as to why this was
necessary since it wasn't ever necessary in the past and this led to an
odd spelunking which turned up some interesting information. Turns out
Rust 1.47 and after had a breaking change where Cargo would sort
dependencies differently. This means that #9334 *could* have been opened
up much earlier, but it never was. We ironically only saw an issue when
we fixed this regression (although we didn't realize we were fixing a
regression). This means that we are now permanently codifying the
regression in Cargo.
This commit restores the hash value of the crates.io `SourceId` to what
it was before #9384. In #9384 the enum variants of `SourceKind` were
reordered which accidentally changed the hash value of the `SourceId`
for crates.io. A change here means that users with a new version of
Cargo will have to redownload the index and all crates, which is
something that we strive to avoid forcing.
In changing this, though, it required a manual implementation of `Ord`
to still contain the actual fix from #9384 which is to sort `SourceKind`
differently from how it's defined. I was curious as to why this was
necessary since it wasn't ever necessary in the past and this led to an
odd spelunking which turned up some interesting information. Turns out
Rust 1.47 and after had a breaking change where Cargo would sort
dependencies differently. This means that #9334 *could* have been opened
up much earlier, but it never was. We ironically only saw an issue when
we fixed this regression (although we didn't realize we were fixing a
regression). This means that we are now permanently codifying the
regression in Cargo.
Fix loading `branch=master` patches in the v3 lock transition
This commit fixes an issue pointed out during #9352 where in the v2->v3
lock file transition (currently happening on nightly) Cargo will not
correctly use the previous lock file entry for `[patch]` directives that
point to git dependencies using `branch = 'master'` explicitly. The
reason for this is that Cargo previously, with the v2 format, considered
`branch=master` and `DefaultBranch` to be equivalent dependencies. Now
that Cargo treats those as distinct resolve nodes we need to load lock
files that use `DefaultBranch` and transparently use those for
`branch=master` dependencies.
These lock file nodes do not naturally unify so we have to go out of our
way to get the two to line up in modern Cargo. This was previously done
for the lock file at large, but the previous logic didn't take `[patch]`
into account. Unfortunately almost everything to do with `[patch]` and
lock files is pretty complicated, and this is no exception. The fix here
is wordy, verbose, and quite subtle in how it works. I'm pretty sure it
does work though and I think that this should be good enough to at least
transition most users off the v2 lock file format. Once this has baked
in Cargo for some time (on the scale of a year) I would hope that we
could just remove this logic since it's only really here for a
transitionary period.
Closes#9352
Fix build-std updating the index on every build.
https://github.com/rust-lang/rust/pull/83776 has caused a problem where build-std will update the index on every build. That PR added `std_detect` from the `stdarch` submodule as a path dependency of `std`. However, since `stdarch` has a workspace of its own, an exclusion had to be added to `Cargo.toml` so that it does not treat `std_detect` as a workspace member (because nested workspaces are not supported).
The problem is that the std `Cargo.lock` file is built thinking that `std_detect` is *not* a workspace member. This means that its dev-dependencies are not included. However, when cargo resolves the std workspace, it doesn't know that `std_detect` should be excluded, so it considers it a workspace member (because it is a path dependency). This means that it expects the dev-dependencies to be in Cargo.lock. Because they are missing, it ends up poisoning the registry and triggering an update:
> poisoning registry `https://github.com/rust-lang/crates.io-index` because std_detect v0.1.5 (/Users/eric/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/src/rust/library/stdarch/crates/std_detect) looks like it changed auxv
The solution here is to skip dev-dependencies if they are not actively being resolved, even if the package is a workspace member.
This has happened before (#8962), so I have updated the test to check for it.
There are some alternative solutions I considered:
* Add support for nested workspaces. 😄
* Use a symlink to `std_detect` in the `rust-lang/rust` repository so that it appears to cargo as-if it is "outside" of the stdarch workspace, and thus can be treated like a normal workspace member (and remove the "exclude"). That seems a little hacky.
Fixes#9390
