This commit fixes#8966 by updating the unit generation logic to avoid
generating an erroneous circular dependency between the execution of two
build scripts. This has been present for Cargo in a long time since #5651
(an accidental regression), but the situation appears rare enough that
we didn't get to it until now!
Closes#8966
This commit is the Cargo half of support necessary for
rust-lang/rust#70458. Today the compiler emits embedded bytecode in
rlibs by default, but compresses it. This is both extraneous disk space
and wasted build time for almost all builds, so the PR in question there
is changing rustc to have a `-Cembed-bitcode` flag which, when enabled,
places the bitcode in the object file rather than an auxiliary file (no
extra compression), but also enables `-Cembed-bitcode=no` to disable
bitcode emission entirely.
This Cargo support changes Cargo to pass `-Cembed-bitcode=no` for almost
all compilations. Cargo will keep `lto = true` and such working by not
passing this flag (and thus allowing bitcode to get embedded), but by
default `cargo build` and `cargo build --release` will no longer have
any bitcode in rlibs which should result in speedier builds!
Most of the changes here were around the test suite and various
assertions about the `rustc` command lines we spit out. One test was
hard-disabled until we can get `-Cembed-bitcode=no` into nightly, and
then we can make it a nightly-only test. The test will then be stable
again once `-Cembed-bitcode=no` hits stable.
Note that this is intended to land before the upstream `-Cembed-bitcode`
change. The thinking is that we'll land everything in rust-lang/rust all
at once so there's no build time regressions for anyone. If we were to
land the `-Cembed-bitcode` PR first then there would be a build time
regression until we land Cargo changes because rustc would be emitting
uncompressed bitcode by default and Cargo wouldn't be turning it off.
This commit enables pipelined compilation by default in Cargo now that
the requisite support has been stablized in rust-lang/rust#62766. This
involved minor updates in a number of locations here and there, but
nothing of meat has changed from the original implementation (just
tweaks to how rustc is called).
Implement the Cargo half of pipelined compilation (take 2)
This commit starts to lay the groundwork for #6660 where Cargo will
invoke rustc in a "pipelined" fashion. The goal here is to execute one
command to produce both an `*.rmeta` file as well as an `*.rlib` file
for candidate compilations. In that case if another rlib depends on that
compilation, then it can start as soon as the `*.rmeta` is ready and not
have to wait for the `*.rlib` compilation.
Initially attempted in #6864 with a pretty invasive refactoring this
iteration is much more lightweight and fits much more cleanly into
Cargo's backend. The approach taken here is to update the
`DependencyQueue` structure to carry a piece of data on each dependency
edge. This edge information represents the artifact that one node
requires from another, and then we a node has no outgoing edges it's
ready to build.
A dependency on a metadata file is modeled as just that, a dependency on
just the metadata and not the full build itself. Most of cargo's backend
doesn't really need to know about this edge information so it's
basically just calculated as we insert nodes into the `DependencyQueue`.
Once that's all in place it's just a few pieces here and there to
identify compilations that *can* be pipelined and then they're wired up
to depend on the rmeta file instead of the rlib file.
Closes#6660
This commit updates Cargo to process JSON directives emitted by rustc
when we're pipelining compilation. In this mode Cargo will attempt to
start subsequent compilations of crates as soon as possible, fully
completing the features of pipelined compilations in Cargo!
This commit refactors slightly how we actually spawn rustc/rustdoc
processes and how their output is routed. Over time lots has changed
around this, but it turns out that we unconditionally capture all output
from the compiler/rustdoc today, no exceptions. As a result simplify the
various execution functions in the `Executor` trait as well as branches
for emitting json messages. Instead throw everything in the same bucket
and just always look for lines that start with `{` which indicate a
JSON message.
This also fixes a few issues where output printed in each thread is now
routed through the main coordinator thread to handle updating the
progress bar if necessary.
This commit starts to lay the groundwork for #6660 where Cargo will
invoke rustc in a "pipelined" fashion. The goal here is to execute one
command to produce both an `*.rmeta` file as well as an `*.rlib` file
for candidate compilations. In that case if another rlib depends on that
compilation, then it can start as soon as the `*.rmeta` is ready and not
have to wait for the `*.rlib` compilation.
Initially attempted in #6864 with a pretty invasive refactoring this
iteration is much more lightweight and fits much more cleanly into
Cargo's backend. The approach taken here is to update the
`DependencyQueue` structure to carry a piece of data on each dependency
edge. This edge information represents the artifact that one node
requires from another, and then we a node has no outgoing edges it's
ready to build.
A dependency on a metadata file is modeled as just that, a dependency on
just the metadata and not the full build itself. Most of cargo's backend
doesn't really need to know about this edge information so it's
basically just calculated as we insert nodes into the `DependencyQueue`.
Once that's all in place it's just a few pieces here and there to
identify compilations that *can* be pipelined and then they're wired up
to depend on the rmeta file instead of the rlib file.
This has proven to be a very unreliable piece of information to hash, so
let's not! Instead we track what files are supposed to be relative to,
and we check both mtimes when necessary.
Ever since the inception of Cargo and the advent of incremental
compilation at the crate level via Cargo, Cargo has tracked whether it
needs to recompile something at a unit level in its "dependency queue"
which manages when items are ready for execution. Over time we've fixed
lots and lots of bugs related to incremental compilation, and perhaps
one of the most impactful realizations was that the model Cargo started
with fundamentally doesn't handle interrupting Cargo halfway through and
resuming the build later.
The previous model relied upon implicitly propagating "dirtiness" based
on whether the one of the dependencies of a build was rebuilt or not.
This information is not available, however, if Cargo is interrupted and
resumed (or performs a subset of steps and then later performs more).
We've fixed this in a number of places historically but the purpose of
this commit is to put a nail in this coffin once and for all.
Implicit propagation of whether a unit is fresh or dirty is no longer
present at all. Instead Cargo should always know, irrespective of it's
in-memory state, whether a unit needs to be recompiled or not. This
commit actually turns up a few bugs in the test suite, so later commits
will be targeted at fixing this.
Note that this required a good deal of work on the `fingerprint` module
to fix some longstanding bugs (like #6780) and some serious hoops had to
be jumped through for others (like #6779). While these were fallout from
this change they weren't necessarily the primary motivation, but rather
to help make `fingerprints` a bit more straightforward in what's an
already confusing system!
Closes#6780
This picks up on the work done in PR #5683.
The extra output is only displayed with `-vv`.
The Windows output has the form `set FOO=foo && BAR=bar rustc ...` instead of
the form that suggested in #5683 to make escaping easier and since it's
simpler.
Fix renaming directory project using build scripts with cross-compiling.
The rename-protection introduced in #4818 checks for paths with a prefix of `/…/target/debug`, but this does not work for paths used during cross-compiling.
This change checks for paths with the full `OUT_DIR` prefix, and the `build/PKG/root-output` file now includes that full `OUT_DIR` instead of `/…/target/debug`.
This also includes a few other changes:
- Support fixing KIND=PATH style paths.
- Support fixing paths in metadata (like `cargo:root=…` or `cargo:include=…` which are common).
- Adds a "version" value to the metadata hash to ensure that cargo doesn't get confused with the new `root-output` file format.
Fixes#6177
cargo's extra verbose mode is useful for getting detailed information out of
builds in CI where it can be difficult to examine the build environment
after-the-fact. However, when multiple build scripts are running as part of a
build it's not always clear what output is from which build script. This patch
makes cargo prefix each line of build script output with the crate name and
version this case.
This change ensures cargo will output file paths in the expected format
(C:\foo\... on Windows, /foo/... elsewhere). Previously it would output
file:// URLs instead.
To support this change, additional changes were made to the test suite
string processing such that [ROOT] is now replaced with the appropriate
file path root for the platform.
The CWD template was also updated to use [CWD] like other replacement
templates and to do the replacement on the expected value rather than
the actual value to avoid replacing things we don't expect with CWD.
.. with mutliple calls of:
fastmod --accept-all '\.cargo\("([^"]+)"\)\.arg\("([^"]+)"\)' '.cargo("${1} ${2}")' tests/testsuite/
until no changes are left.
* Collapse the nested cargotest::support module into the cargotest
module (merge the mod.rs's)
* Rename the cargotest module to support
* Nest the top-level hamcrest module into support
Generally that means either switching "foo" and "bar" around (reversing
the arrow), or it means push "foo" to "bar" (and sometimes "bar" to
"baz", etc..) to free up "foo".
For trivia that leaves 80/1222 outliers, therefore 93.4% of test
project use the default. :)
By rewriting the tests, with rerast (https://github.com/google/rerast),
to use the newly introduced "at" method.
First I added the following temporary function to cargotest::support:
pub fn project_foo() -> ProjectBuilder {
project("foo")
}
Then I defined the following rewrite.rs:
use cargotest::support::{ project, project_foo };
fn rule1(a: &'static str) {
replace!(project("foo") => project_foo());
replace!(project(a) => project_foo().at(a));
}
Then I ran rerast:
cargo +nightly rerast --rules_file=rewrite.rs --force --targets tests --file tests/testsuite/main.rs
Finally I searched and replaced the references to project_foo with
argument-less project (a little awkardly on macOS with a git clean).
find tests -type f -exec sed -i -e 's/project_foo/project/g' {} +
git clean -d tests
Some logic which was tweaked around the dependencies of build script targets was
tweaked slightly in a way that causes cargo to stack overflow by accientally
adding a dependency loop. This commit implements one of the strategies discussed
in #5711 to fix this situation.
The problem here is that when calculating the deps of a build script we need the
build scripts of *other* packages, but the exact profile is somewhat difficult
to guess at the moment we're generating our build script unit. To solve this the
dependencies towards other build scripts' executions is added in a different
pass after all other units have been assembled. At this point we should know for
sure that all build script executions are in the dependency graph, and we just
need to add a few more edges.
Closes#5708
Fixes#5650. cc #5435
As part of my recent work on profiles, I introduced some situations where a
library can be compiled multiple times with different settings. Doctests were
greedily grabbing all dependencies for a package, regardless of which target
is was for. This can cause doctests to fail if it links multiple copies of
the same library.
One way to trigger this is `cargo test --release` if you have dependencies, a
build script, and `panic="abort"`. There are other (more obscure) ways to
trigger it with profile overrides.
This commit cops out trying to fix `rename_with_link_search_path` by simply
adding a loop on Windows to retry the operation that looks to need retrying.
We warn if a feature was specified corresponding to a dependency which
is not optional. However, a dependency can be both optional and
required, and we shouldn't warn in that case.
Fix optional deps in multiple sections
This commit fixes an issue where an optional dependency was listed multiple
times in a manifest (multiple sections). This regression was introduced by #5415
and happened because in the resolver we didn't record a `Dependency` as it was
accidentally deduplicated too soon.
The fix here was to ensure that all `Dependency` annotations make their way into
`Resolve` now that we rely on the listed `Dependency` values for correctness.
Closes#5475
This commit fixes an issue where an optional dependency was listed multiple
times in a manifest (multiple sections). This regression was introduced by #5415
and happened because in the resolver we didn't record a `Dependency` as it was
accidentally deduplicated too soon.
The fix here was to ensure that all `Dependency` annotations make their way into
`Resolve` now that we rely on the listed `Dependency` values for correctness.
Closes#5475
I *think* the issue is that `link.exe` is generating debuginfo in the background
which keeps the file open and prevents us from deleting it, but hopefully by
disabling debug information we'll either be able to confirm or deny this
hypothesis.
Added tests for the environment variable RUSTC_LINKER that is
passed to the build script.
Also slightly improved readability of the code responsible for
passing the env var.
