Ignore remap-path-prefix in metadata hash.
Including this flag in the metadata hash causes problems with reproducible builds.
I spent some time considering the different alternatives (such as providing a config option, or an unhashed RUSTFLAGS alternative), and decided this might be the best option.
- It is a very simple, small change.
- It should be safe.
- It is transparent to the user, they don't need to do anything special.
- It doesn't expand Cargo's interface.
Fixes#6914.
Currently when Cargo is invoked on the command like `cargo build
--features foo/bar` then it will actually always compile the current
crate with `feature = "foo"` even if `foo` is a non-optional dependency.
This isn't intended because the crate doesn't actually have a `foo`
feature as so no directive should be emitted or passed to the compiler.
This was discovered in rust-lang/rust where Cargo is being built with
the `rustc-workspace-hack` feature but when the RLS depends on Cargo it
doesn't enable the same feature. This feature, however, doesn't actually
exist for Cargo!
Import the cargo-vendor subcommand into Cargo
This commit imports the external [alexcrichton/cargo-vendor
repository][repo] into Cargo itself. This means it will no longer be
necessary to install the `cargo-vendor` subcommand in order to vendor
dependencies. Additionally it'll always support the latest feature set
of Cargo as it'll be built into Cargo!
All tests were imported as part of this commit, but not all features
were imported. Some flags have been left out that were added later in
the lifetime of `cargo vendor` which seem like they're more questionable
to stabilize. I'm hoping that they can have separate PRs adding their
implementation here, and we can make a decision of their stabilization
at a later date.
The current man page for `cargo vendor -h` will look like:
cargo-vendor
Vendor all dependencies for a project locally
USAGE:
cargo vendor [OPTIONS] [--] [path]
OPTIONS:
-q, --quiet No output printed to stdout
--manifest-path <PATH> Path to Cargo.toml
--no-delete Don't delete older crates in the vendor directory
-s, --sync <TOML>... Additional `Cargo.toml` to sync and vendor
--respect-source-config Respect `[source]` config in `.cargo/config`
-v, --verbose Use verbose output (-vv very verbose/build.rs output)
--color <WHEN> Coloring: auto, always, never
--frozen Require Cargo.lock and cache are up to date
--locked Require Cargo.lock is up to date
-Z <FLAG>... Unstable (nightly-only) flags to Cargo, see 'cargo -Z help' for details
-h, --help Prints help information
ARGS:
<path> Where to vendor crates (`vendor` by default)
This cargo subcommand will vendor all crates.io and git dependencies for a
project into the specified directory at `<path>`. After this command completes
the vendor directory specified by `<path>` will contain all remote sources from
dependencies specified. Additionally manifest beyond the default one can be
specified with the `-s` option.
The `cargo vendor` command will also print out the configuration necessary
to use the vendored sources, which when needed is then encoded into
`.cargo/config`.
Since this change is not importing 100% of the functionality of the
existing `cargo vendor` this change does run a risk of being a breaking
change for any folks using such functionality. Executing `cargo vendor`
will favor the built-in command rather than an external subcommand,
causing unimplemented features to become errors about flag usage.
[repo]: https://github.com/alexcrichton/cargo-vendor
Test the Resolver against the varisat Library
Resolution can be reduced to the SAT problem. So this is an alternative implementation of the resolver that uses a SAT library for the hard work. This is intended to be easy to read, as compared to the real resolver, and run as part of the test sweet to make sure the real resolver works as expected. Part of #6120.
Some notes on performance:
The initial version did not support public & private deps:
~64 loc, `O(nln(n))` vars, `O(nln(n) + n*d)` clauses, 0.5x slower on `prop_passes_validation`
The final version:
~163 loc, `O(dn^2`) vars, `O(dn^3)` clauses, 1.5x slower on `prop_passes_validation`
That comparison makes me feel better about spending months trying to get public & private deps to be fast enough for stabilization.
cargo package: detect new empty directories
Detect the creation of directories in a build script, which is currently a very tempting workaround for the fact that a crate built from a package will be missing any empty directories.
Maybe it would be better to only include directories in the map of hashes if they are completely empty.
The removal of a directory that is initially empty cannot be tested because, as stated above, a crate built from a package will not *have* any empty directories.
Closes#6931.
This commit imports the external [alexcrichton/cargo-vendor
repository][repo] into Cargo itself. This means it will no longer be
necessary to install the `cargo-vendor` subcommand in order to vendor
dependencies. Additionally it'll always support the latest feature set
of Cargo as it'll be built into Cargo!
All tests were imported as part of this commit, but not all features
were imported. Some flags have been left out that were added later in
the lifetime of `cargo vendor` which seem like they're more questionable
to stabilize. I'm hoping that they can have separate PRs adding their
implementation here, and we can make a decision of their stabilization
at a later date.
The current man page for `cargo vendor -h` will look like:
cargo-vendor
Vendor all dependencies for a project locally
USAGE:
cargo vendor [OPTIONS] [--] [path]
OPTIONS:
-q, --quiet No output printed to stdout
--manifest-path <PATH> Path to Cargo.toml
--no-delete Don't delete older crates in the vendor directory
-s, --sync <TOML>... Additional `Cargo.toml` to sync and vendor
--respect-source-config Respect `[source]` config in `.cargo/config`
-v, --verbose Use verbose output (-vv very verbose/build.rs output)
--color <WHEN> Coloring: auto, always, never
--frozen Require Cargo.lock and cache are up to date
--locked Require Cargo.lock is up to date
-Z <FLAG>... Unstable (nightly-only) flags to Cargo, see 'cargo -Z help' for details
-h, --help Prints help information
ARGS:
<path> Where to vendor crates (`vendor` by default)
This cargo subcommand will vendor all crates.io and git dependencies for a
project into the specified directory at `<path>`. After this command completes
the vendor directory specified by `<path>` will contain all remote sources from
dependencies specified. Additionally manifest beyond the default one can be
specified with the `-s` option.
The `cargo vendor` command will also print out the configuration necessary
to use the vendored sources, which when needed is then encoded into
`.cargo/config`.
Since this change is not importing 100% of the functionality of the
existing `cargo vendor` this change does run a risk of being a breaking
change for any folks using such functionality. Executing `cargo vendor`
will favor the built-in command rather than an external subcommand,
causing unimplemented features to become errors about flag usage.
[repo]: https://github.com/alexcrichton/cargo-vendor
Clippy fixes
Some of these are arguably not strictly better, but didn't really seem worse to me either.
I tried changing all `&Unit` to `Unit`, but I felt like the code was worse with more frequent use of `*` and `&`.
Some tools can support rust doctests (e.g. highlighting
or launching). So it should be possible to find out if
doctests are enabled for a target or not. This commit
adds `doctest` field to `cargo metadata` output.
Previously Cargo would attempt to work as much as possible with a
previously filled out CARGO_HOME, even if it was mounted as read-only.
In #6880 this was regressed as a few global locks and files were always
attempted to be opened in writable mode.
This commit fixes these issues by correcting two locations:
* First the global package cache lock has error handling to allow
acquiring the lock in read-only mode inaddition to read/write mode. If
the read/write mode failed due to an error that looks like a readonly
filesystem then we assume everything in the package cache is readonly
and we switch to just acquiring any lock, this time a shared readonly
one. We in theory aren't actually doing any synchronization at that
point since it's all readonly anyway.
* Next when unpacking package we're careful to issue a `stat` call
before opening a file in writable mode. This way our preexisting guard
to return early if a package is unpacked will succeed before we open
anything in writable mode.
Closes#6928
Migrate package include/exclude to gitignore patterns.
This moves to the next phase of #4268.
This also includes a fdew more changes which can be removed if desired:
- Add support for `!` negate gitignore patterns.
- Add a warning if both package.include and package.exclude are specified.
Implement the Cargo half of pipelined compilation (take 2)
This commit starts to lay the groundwork for #6660 where Cargo will
invoke rustc in a "pipelined" fashion. The goal here is to execute one
command to produce both an `*.rmeta` file as well as an `*.rlib` file
for candidate compilations. In that case if another rlib depends on that
compilation, then it can start as soon as the `*.rmeta` is ready and not
have to wait for the `*.rlib` compilation.
Initially attempted in #6864 with a pretty invasive refactoring this
iteration is much more lightweight and fits much more cleanly into
Cargo's backend. The approach taken here is to update the
`DependencyQueue` structure to carry a piece of data on each dependency
edge. This edge information represents the artifact that one node
requires from another, and then we a node has no outgoing edges it's
ready to build.
A dependency on a metadata file is modeled as just that, a dependency on
just the metadata and not the full build itself. Most of cargo's backend
doesn't really need to know about this edge information so it's
basically just calculated as we insert nodes into the `DependencyQueue`.
Once that's all in place it's just a few pieces here and there to
identify compilations that *can* be pipelined and then they're wired up
to depend on the rmeta file instead of the rlib file.
Closes#6660
Always include `Cargo.toml` when packaging.
Since `Cargo.toml` is required, might as well include it automatically rather than force everyone to include it explicitly. If it is not listed in `include`, there was a somewhat confusing error message when packaging.
Closes#6830Closes#4660
This commit fixes an issue when pipelining mode is used in handling
recompilations. Previously a sequence of compilations could look like:
* Crate A starts to build
* Crate A produces metadata
* Crate B, which depends on A, starts
* Crate B finishes
* Crate A finishes
In this case the mtime for B is before that of A, which fooled Cargo
into thinking that B needed to be recompiled. In this case, however, B
doesn't actually need to be recompiled because it only depends on the
metadata of A, not the final artifacts.
This unfortunately resulted in some duplication in a few places, but not
really much moreso than already exists between fingerprinting and compilation.
This commit updates Cargo to process JSON directives emitted by rustc
when we're pipelining compilation. In this mode Cargo will attempt to
start subsequent compilations of crates as soon as possible, fully
completing the features of pipelined compilations in Cargo!
This commit refactors slightly how we actually spawn rustc/rustdoc
processes and how their output is routed. Over time lots has changed
around this, but it turns out that we unconditionally capture all output
from the compiler/rustdoc today, no exceptions. As a result simplify the
various execution functions in the `Executor` trait as well as branches
for emitting json messages. Instead throw everything in the same bucket
and just always look for lines that start with `{` which indicate a
JSON message.
This also fixes a few issues where output printed in each thread is now
routed through the main coordinator thread to handle updating the
progress bar if necessary.
This commit starts to lay the groundwork for #6660 where Cargo will
invoke rustc in a "pipelined" fashion. The goal here is to execute one
command to produce both an `*.rmeta` file as well as an `*.rlib` file
for candidate compilations. In that case if another rlib depends on that
compilation, then it can start as soon as the `*.rmeta` is ready and not
have to wait for the `*.rlib` compilation.
Initially attempted in #6864 with a pretty invasive refactoring this
iteration is much more lightweight and fits much more cleanly into
Cargo's backend. The approach taken here is to update the
`DependencyQueue` structure to carry a piece of data on each dependency
edge. This edge information represents the artifact that one node
requires from another, and then we a node has no outgoing edges it's
ready to build.
A dependency on a metadata file is modeled as just that, a dependency on
just the metadata and not the full build itself. Most of cargo's backend
doesn't really need to know about this edge information so it's
basically just calculated as we insert nodes into the `DependencyQueue`.
Once that's all in place it's just a few pieces here and there to
identify compilations that *can* be pipelined and then they're wired up
to depend on the rmeta file instead of the rlib file.
Small things
This has two small changes that are worth saving from my most recent attempt to speedup https://github.com/rust-lang/cargo/issues/6258#issuecomment-479204465:
1. This removes the `ConflictCache::contains` added in #6776. Replacing it with a more general and easier to explain system based on the existing `ConflictCache::find_conflicting`.
2. This adds code to print the used part of the input for failing resolver tests. This is very helpful when
1. The proptest shrinking algorithm is interrupted, at least you have the smallest one found so far.
2. Hand minimizing, remove a dep and it will tell you all the packages that are no longer needed for the test to fail.
This accidentally regressed in the previous caching PR for Cargo.
Invalid lines of JSON in the registry are intended to be skipped over,
but when skipping we forgot to update some indices which meant that all
future versions would fail to parse as well!
This commit updates the locking strategy in Cargo handle the recent
addition of creating a cache of the on-disk index also on disk. The goal
here is reduce the overhead of locking both cognitively when reading but
also performance wise by requiring fewer locks. Previously Cargo had a
bunch of fine-grained locks throughout the index and git repositories,
but after this commit there's just one global "package cache" lock.
This global lock now serves to basically synchronize the entire crate
graph resolution step. This shouldn't really take that long unless it's
downloading, in which case there's not a ton of benefit to running in
parallel anyway. The other intention of this single global lock is to
make it much easier on the sources to not worry so much about lock
ordering or when to acquire locks, but rather they just assert in their
various operations that they're locked.
Cargo now has a few coarse-grained locations where locks are held (for
example during resolution and during package downloading). These locks
are a bit sprinkled about but they have in-code asserts which assert
that they're held, so we'll find bugs quickly if any lock isn't held
(before a race condition is hit that is)
Add some help with updating the registry in offline mode.
This includes the following:
- Move offline tests to a dedicated file.
- If an index has never been downloaded, and `-Z offline` is used, provide a suggestion to run without `-Z offline`.
- ~~If an index needs to be updated, and the network appears to be down, suggest running with `-Z offline`.~~ (removed this)
- The `offline_resolve_optional_fail` test is added to show that resolver errors are still possible (though hopefully rare!).
This makes two changes:
- If an index has never been downloaded, and `-Z offline` is used, provide a
suggestion to run without `-Z offline`.
- If an index needs to be updated, and the network appears to be down, suggest
running with `-Z offline`.
This is part of https://github.com/rust-lang/rust/issues/44663
This implements the 'frontend' portion of RFC 1977. Once PRs
https://github.com/rust-lang/rust/pull/59335 and
https://github.com/rust-lang/crates.io/pull/1685 are merged,
it will be possible to test the full public-private dependency feature:
marking a dependency a public, seeing exported_private_dependencies
warnings from rustc, and seeing pub-dep-reachability errors from Cargo.
Everything in this commit should be fully backwards-compatible - users
who don't enable the 'public-dependency' cargo feature won't notice any
changes.
Note that this commit does *not* implement the remaining two features of
the RFC:
* Choosing smallest versions when 'cargo publish' is run
* Turning exported_private_dependencies warnings into hard errors when
'cargo publish' is run
The former is a major change to Cargo's behavior, and should be done
in a separate PR with some kind of rollout plan.
The latter is described by the RFC as being enabled at 'some point in
the future'. This can be done via a follow-up PR.
publish-lockfile: Various updates
This is a collection of changes to the `publish-lockfile` feature. They are separated by commit (see each for more detailed commit messages), and I can separate them into separate PRs if necessary.
- Move publish-lockfile tests to a dedicated file.
- Regenerate `Cargo.lock` when creating a package.
- Ignore `Cargo.lock` in `cargo install` by default (requires `--locked` to use).
- Add warnings for yanked dependencies.
- Change default of publish-lockfile to true.
This includes `Cargo.lock` in the git dirty check. It explicitly excludes
`Cargo.lock` as an untracked file, since it is not relevant for the dirty check;
it is only checked if it is committed to git.
This also adds `Cargo.lock` to the "did anything modify this" check during
verification. I don't see a reason to exclude it (particularly since ephemeral
workspaces do not save the lock file).
Also add "Archiving: Cargo.lock" to the verbose output.
The macos dynamic linker behavior wrt DYLD_FALLBACK_LIBRARY_PATH is to
use the value it is set with, and if there is no such value (the
environment variable is either not set or set but empty), it uses a
default value of $HOME/lib:/usr/local/lib:/usr/lib.
Currently, cargo takes the value of DYLD_FALLBACK_LIBRARY_PATH, prepends
its paths to it, and then unconditionally adds
$HOME/lib:/usr/local/lib:/usr/lib, which in principle, shouldn't happen
if DYLD_FALLBACK_LIBRARY_PATH was set originally.
