This commit is the Cargo half of support necessary for
rust-lang/rust#70458. Today the compiler emits embedded bytecode in
rlibs by default, but compresses it. This is both extraneous disk space
and wasted build time for almost all builds, so the PR in question there
is changing rustc to have a `-Cembed-bitcode` flag which, when enabled,
places the bitcode in the object file rather than an auxiliary file (no
extra compression), but also enables `-Cembed-bitcode=no` to disable
bitcode emission entirely.
This Cargo support changes Cargo to pass `-Cembed-bitcode=no` for almost
all compilations. Cargo will keep `lto = true` and such working by not
passing this flag (and thus allowing bitcode to get embedded), but by
default `cargo build` and `cargo build --release` will no longer have
any bitcode in rlibs which should result in speedier builds!
Most of the changes here were around the test suite and various
assertions about the `rustc` command lines we spit out. One test was
hard-disabled until we can get `-Cembed-bitcode=no` into nightly, and
then we can make it a nightly-only test. The test will then be stable
again once `-Cembed-bitcode=no` hits stable.
Note that this is intended to land before the upstream `-Cembed-bitcode`
change. The thinking is that we'll land everything in rust-lang/rust all
at once so there's no build time regressions for anyone. If we were to
land the `-Cembed-bitcode` PR first then there would be a build time
regression until we land Cargo changes because rustc would be emitting
uncompressed bitcode by default and Cargo wouldn't be turning it off.
Checking for binary that is built as an implicit dependency of an integration test.
When a project containing binary with required-features, the binary will only be built with the given dependency's feature is enabled. But this feature syntax is not checked when the binary is built as an implicit dependency of an integration test. This pr addresses this issue.
More info is in the issue page: this pr fixes#7853.
Re-implement proc-macro feature decoupling.
This is essentially a rewrite of #8003. Instead of adding proc-macro to the index, it uses a strategy of downloading all packages before doing feature resolution. Then the package can be inspected for the proc-macro field.
This is a fairly major change. A brief overview:
- `PackageSet` now has a `download_accessible` method which tries to download a minimal set of every accessible package. This isn't very smart right now, and errs on downloading too much. In most cases it should be the same (or nearly the same) as before. It downloads extra in the following cases:
- The check for `[target]` dependencies checks both host and target for every dependency. I could tighten that up a little so build dependencies only check for the host, but it would add some complexity and I wanted to get feedback first.
- Optional dependencies disabled by the new feature resolver will get downloaded.
- Removed the loop in computing unit dependencies where downloading used to reside.
- When downloading starts, it should now show a more accurate count of how many crates are to be downloaded. Previously the count would fluctuate while the graph is being built.
Add proc-macro to index, and new feature resolver.
This adds the "pm" field to the index so that Cargo can detect which packages contain a proc-macro without downloading the package.
The second commit builds on that to support proc-macros in the new "de-unification" of the new feature resolver. This prevents dependencies shared between proc-macros and other dependency kinds from having features unified.
cc #7915
Avoid buffering large amounts of rustc output.
If `rustc` prints out a lot of information (such as with `RUSTC_LOG`, or a huge number of diagnostics), cargo would buffer up large amounts of that in memory. For normal builds, this would happen if the terminal does not print fast enough. For "fresh" replay, *everything* was being buffered.
There are two issues:
1. There is no back-pressure on the mpsc queue. If messages come in faster than they can be processed, it grows without bounds.
2. The cache-replay code runs in the "fresh" code path which does not spawn a thread. Thus the main thread was blocked and unable to process `Message`s while the replay is happening.
The solution here is to use a bounded queue, and to always spawn a thread for the "fresh" case.
The main concern here is performance. Previously the "fresh" jobs avoided spawning a thread to improve performance. I did a fair bit of profiling to understand the impact, using projects with anywhere from 100 to 500 units. On my macOS machine, I found spawning a thread to be slightly faster (1-5%). On Linux and Windows, it was generally about 0 to 5% slower. It might be helpful for others to profile it on their own system.
I'm on the fence for the cost/benefit here. It seems generally good to reduce memory usage, but the slight performance hit is disappointing. I tried several other approaches to fix this, all with worse trade offs (I can discuss them if interested).
Fixes#6197
Try to better handle restricted crate names.
This attempts to improve handling of restricted crate names, particularly for `cargo new` and `cargo init`. Hopefully the code is straightforward to follow, but in summary the changes are:
**General changes**
* Add more details to the error messages about why a name is not allowed, and what is allowed.
* Change the valid package name check to be restricted to Unicode XID. This brings it in line with non_ascii_idents support in rustc. For the most part, this is pretty much the same as before. Note: this is used for the package name, and registry names. The differences are:
* Package names cannot start with numbers. Previously this was only rejected in `cargo new`. crates.io also rejects numbers. Numbers are also not valid crate names.
* Package names cannot start with dash `-`. This is a somewhat arbitrary change, but seems like it would stem problems. crates.io also rejects this.
* Package names cannot start with these characters that were previously allowed: https://gist.github.com/ehuss/804a797950001b5226e1264b6f65211f#file-not_start_but_alphanumeric-txt
* Most of these are wacky numbers or other strange things.
* Package names cannot contain these characters that were previously allowed: https://gist.github.com/ehuss/804a797950001b5226e1264b6f65211f#file-not_continue_but_alphanumeric-txt
* These are mostly odd things that for whatever reason the Unicode people decided not to include. It seems unlikely to me that someone would want to use one of these.
* Display a warning on Windows if a Cargo target is a special Windows filename. The build error tends to be hard to understand, so the hope is the warning will make it evident.
* `cargo package/publish`: Warn if a special Windows name is in the package.
**cargo new/init specific changes**
* Update keyword list to 2018 edition.
* Add warning if creating a library that has one of the conflicting names (deps/examples/build/incremental).
* Warn about conflicting std names (core/std/alloc/proc-macro).
* Windows reserved names: Rejected on windows, warned on other platforms.
* Warn about non-ASCII package names.
* Only print the `--name` suggestion for `cargo init`. I found the suggestion confusing, and I feel like it doesn't really make sense for `cargo new` (since it would only affect the directory name).
Ignore broken Cargo.toml in git sources
Commit 3d6de41774 (#3998) made cargo
ignore Cargo.toml files that are invalid TOML in a git source.
This change further ignores Cargo.toml files that are valid TOML but
cannot really be loaded in a git source.
This is potentially an alternative fix for #6822.
Commit 3d6de41774 (#3998) made cargo
ignore Cargo.toml files that are invalid TOML in a git source.
This change further ignores Cargo.toml files that are valid TOML but
cannot really be loaded.
This is potentially an alternative fix for #6822.
Provide extra context on a query failure.
This adds error context when a query fails, primarily to tell you which parent package included the dependency that failed. For example, imagine deep within your dependency graph you have a `git` dependency, and it fails to download. The current error doesn't tell you where in the graph that `git` dependency was included.
I also slightly tweaked the `failed to load source` error message. I felt like the existing wording could be misinterpreted that it was an error loading the dependency *for* the given package. I felt like there were multiple ways to interpret it, so I tried to simplify it to avoid that possibility.
