Some tools can support rust doctests (e.g. highlighting
or launching). So it should be possible to find out if
doctests are enabled for a target or not. This commit
adds `doctest` field to `cargo metadata` output.
Previously Cargo would attempt to work as much as possible with a
previously filled out CARGO_HOME, even if it was mounted as read-only.
In #6880 this was regressed as a few global locks and files were always
attempted to be opened in writable mode.
This commit fixes these issues by correcting two locations:
* First the global package cache lock has error handling to allow
acquiring the lock in read-only mode inaddition to read/write mode. If
the read/write mode failed due to an error that looks like a readonly
filesystem then we assume everything in the package cache is readonly
and we switch to just acquiring any lock, this time a shared readonly
one. We in theory aren't actually doing any synchronization at that
point since it's all readonly anyway.
* Next when unpacking package we're careful to issue a `stat` call
before opening a file in writable mode. This way our preexisting guard
to return early if a package is unpacked will succeed before we open
anything in writable mode.
Closes#6928
Migrate package include/exclude to gitignore patterns.
This moves to the next phase of #4268.
This also includes a fdew more changes which can be removed if desired:
- Add support for `!` negate gitignore patterns.
- Add a warning if both package.include and package.exclude are specified.
Implement the Cargo half of pipelined compilation (take 2)
This commit starts to lay the groundwork for #6660 where Cargo will
invoke rustc in a "pipelined" fashion. The goal here is to execute one
command to produce both an `*.rmeta` file as well as an `*.rlib` file
for candidate compilations. In that case if another rlib depends on that
compilation, then it can start as soon as the `*.rmeta` is ready and not
have to wait for the `*.rlib` compilation.
Initially attempted in #6864 with a pretty invasive refactoring this
iteration is much more lightweight and fits much more cleanly into
Cargo's backend. The approach taken here is to update the
`DependencyQueue` structure to carry a piece of data on each dependency
edge. This edge information represents the artifact that one node
requires from another, and then we a node has no outgoing edges it's
ready to build.
A dependency on a metadata file is modeled as just that, a dependency on
just the metadata and not the full build itself. Most of cargo's backend
doesn't really need to know about this edge information so it's
basically just calculated as we insert nodes into the `DependencyQueue`.
Once that's all in place it's just a few pieces here and there to
identify compilations that *can* be pipelined and then they're wired up
to depend on the rmeta file instead of the rlib file.
Closes#6660
Always include `Cargo.toml` when packaging.
Since `Cargo.toml` is required, might as well include it automatically rather than force everyone to include it explicitly. If it is not listed in `include`, there was a somewhat confusing error message when packaging.
Closes#6830Closes#4660
This commit fixes an issue when pipelining mode is used in handling
recompilations. Previously a sequence of compilations could look like:
* Crate A starts to build
* Crate A produces metadata
* Crate B, which depends on A, starts
* Crate B finishes
* Crate A finishes
In this case the mtime for B is before that of A, which fooled Cargo
into thinking that B needed to be recompiled. In this case, however, B
doesn't actually need to be recompiled because it only depends on the
metadata of A, not the final artifacts.
This unfortunately resulted in some duplication in a few places, but not
really much moreso than already exists between fingerprinting and compilation.
This commit updates Cargo to process JSON directives emitted by rustc
when we're pipelining compilation. In this mode Cargo will attempt to
start subsequent compilations of crates as soon as possible, fully
completing the features of pipelined compilations in Cargo!
This commit refactors slightly how we actually spawn rustc/rustdoc
processes and how their output is routed. Over time lots has changed
around this, but it turns out that we unconditionally capture all output
from the compiler/rustdoc today, no exceptions. As a result simplify the
various execution functions in the `Executor` trait as well as branches
for emitting json messages. Instead throw everything in the same bucket
and just always look for lines that start with `{` which indicate a
JSON message.
This also fixes a few issues where output printed in each thread is now
routed through the main coordinator thread to handle updating the
progress bar if necessary.
This commit starts to lay the groundwork for #6660 where Cargo will
invoke rustc in a "pipelined" fashion. The goal here is to execute one
command to produce both an `*.rmeta` file as well as an `*.rlib` file
for candidate compilations. In that case if another rlib depends on that
compilation, then it can start as soon as the `*.rmeta` is ready and not
have to wait for the `*.rlib` compilation.
Initially attempted in #6864 with a pretty invasive refactoring this
iteration is much more lightweight and fits much more cleanly into
Cargo's backend. The approach taken here is to update the
`DependencyQueue` structure to carry a piece of data on each dependency
edge. This edge information represents the artifact that one node
requires from another, and then we a node has no outgoing edges it's
ready to build.
A dependency on a metadata file is modeled as just that, a dependency on
just the metadata and not the full build itself. Most of cargo's backend
doesn't really need to know about this edge information so it's
basically just calculated as we insert nodes into the `DependencyQueue`.
Once that's all in place it's just a few pieces here and there to
identify compilations that *can* be pipelined and then they're wired up
to depend on the rmeta file instead of the rlib file.
Small things
This has two small changes that are worth saving from my most recent attempt to speedup https://github.com/rust-lang/cargo/issues/6258#issuecomment-479204465:
1. This removes the `ConflictCache::contains` added in #6776. Replacing it with a more general and easier to explain system based on the existing `ConflictCache::find_conflicting`.
2. This adds code to print the used part of the input for failing resolver tests. This is very helpful when
1. The proptest shrinking algorithm is interrupted, at least you have the smallest one found so far.
2. Hand minimizing, remove a dep and it will tell you all the packages that are no longer needed for the test to fail.
This accidentally regressed in the previous caching PR for Cargo.
Invalid lines of JSON in the registry are intended to be skipped over,
but when skipping we forgot to update some indices which meant that all
future versions would fail to parse as well!
This commit updates the locking strategy in Cargo handle the recent
addition of creating a cache of the on-disk index also on disk. The goal
here is reduce the overhead of locking both cognitively when reading but
also performance wise by requiring fewer locks. Previously Cargo had a
bunch of fine-grained locks throughout the index and git repositories,
but after this commit there's just one global "package cache" lock.
This global lock now serves to basically synchronize the entire crate
graph resolution step. This shouldn't really take that long unless it's
downloading, in which case there's not a ton of benefit to running in
parallel anyway. The other intention of this single global lock is to
make it much easier on the sources to not worry so much about lock
ordering or when to acquire locks, but rather they just assert in their
various operations that they're locked.
Cargo now has a few coarse-grained locations where locks are held (for
example during resolution and during package downloading). These locks
are a bit sprinkled about but they have in-code asserts which assert
that they're held, so we'll find bugs quickly if any lock isn't held
(before a race condition is hit that is)
Add some help with updating the registry in offline mode.
This includes the following:
- Move offline tests to a dedicated file.
- If an index has never been downloaded, and `-Z offline` is used, provide a suggestion to run without `-Z offline`.
- ~~If an index needs to be updated, and the network appears to be down, suggest running with `-Z offline`.~~ (removed this)
- The `offline_resolve_optional_fail` test is added to show that resolver errors are still possible (though hopefully rare!).
This makes two changes:
- If an index has never been downloaded, and `-Z offline` is used, provide a
suggestion to run without `-Z offline`.
- If an index needs to be updated, and the network appears to be down, suggest
running with `-Z offline`.
