Implement the 'frontend' of public-private dependencies
This is part of https://github.com/rust-lang/rust/issues/44663
This implements the 'frontend' portion of [RFC 1977](https://github.com/rust-lang/rfcs/blob/master/text/1977-public-private-dependencies.md). Once PRs https://github.com/rust-lang/rust/pull/59335 and https://github.com/rust-lang/crates.io/pull/1685 are merged,
it will be possible to test the full public-private dependency feature:
marking a dependency a public, seeing exported_private_dependencies
warnings from rustc, and seeing pub-dep-reachability errors from Cargo.
Everything in this commit should be fully backwards-compatible - users
who don't enable the 'public-dependency' cargo feature won't notice any
changes.
Note that this commit does *not* implement the remaining two features of
the RFC:
* Choosing smallest versions when 'cargo publish' is run
* Turning exported_private_dependencies warnings into hard errors when 'cargo publish' is run
The former is a major change to Cargo's behavior, and should be done in a separate PR with some kind of rollout plan.
The latter is described by the RFC as being enabled at 'some point in the future'. This can be done via a follow-up PR.
Caching the dependencies
There are 2 sources of facts for the resolver:
1. The `Registry` tells us for a Dependency what versions are available to fulfil it.
2. The `Summary` tells us for a version (and features) what dependencies need to be fulfilled for it to be activated.
The `Registry` was cached with a `RegistryQueryer` back in #5112. This adds a `DepsCache` to cache the calculation of which dependencies are activated by features.
In the happy path `flag_activated` means that we don't get to reuse `build_deps`, but the more we backtrack the more time we save. In pathological cases like https://github.com/rust-lang/cargo/issues/6258#issuecomment-479204465, I have measured this as a 10% improvement with release.
This also means that `build_deps` can be run in a context free way, which may be useful in a follow up PR to solve https://github.com/rust-lang/cargo/issues/6258#issuecomment-482620770.
This is part of https://github.com/rust-lang/rust/issues/44663
This implements the 'frontend' portion of RFC 1977. Once PRs
https://github.com/rust-lang/rust/pull/59335 and
https://github.com/rust-lang/crates.io/pull/1685 are merged,
it will be possible to test the full public-private dependency feature:
marking a dependency a public, seeing exported_private_dependencies
warnings from rustc, and seeing pub-dep-reachability errors from Cargo.
Everything in this commit should be fully backwards-compatible - users
who don't enable the 'public-dependency' cargo feature won't notice any
changes.
Note that this commit does *not* implement the remaining two features of
the RFC:
* Choosing smallest versions when 'cargo publish' is run
* Turning exported_private_dependencies warnings into hard errors when
'cargo publish' is run
The former is a major change to Cargo's behavior, and should be done
in a separate PR with some kind of rollout plan.
The latter is described by the RFC as being enabled at 'some point in
the future'. This can be done via a follow-up PR.
Backend refactorings and perf improvements
This PR is extracted from https://github.com/rust-lang/cargo/pull/6864 and then also additionally adds some commits related to performance optimizations that I noticed while profiling https://github.com/rust-lang/cargo/pull/6864. Each commit is in theory standalone and should pass all the tests, as well as being descriptive about what it's doing.
Make required dependency as future an error, remove RcList
This makes it an error to set a feature on a dependency if it is the name of a required dependency not a feature. This has been a warning for a long time. The resolver will backtrack to find a version that works, if one is available. So I think it is safe to make the change.
If you need to make a optional dependency into a required dependency without a semver breaking change this can be done with rename dependencies.
The result is that we can remove 3 data structures that get cloned once per tick of the resolver.
This commit moves a linear scan which happens once-per-each-dependency
to an O(1) lookup which happens only once for each package. This removes
another 30ms or so from a null build in Servo.
This commit moves it from a linear query which does a lot of hashing of
`PathBuf` to an `O(1)` query that only hashes `PackageId`. This improves
Servo's null build performance by 50ms or so, causing a number of
functions to disappear from profiles.
This commit starts to intern `Unit` structures for a few reasons:
* This primarily makes equality and hashing much faster. We have tons of
hash lookups with units, and they were showing up quite high
in profiles. It turns out `Unit` hashes a *ton* of data, and most of
it is always redundant. To handle this they're all only hashed once
now and hashing/equality are just pointer checks.
* The size of `Unit` is now drastically reduced to just one pointer, so
movement of units throughout the backend should be much more
efficient.
The code lying around in `dep_targets` is pretty old at this point, but
given the current iteraiton there's no need to re-sort on each call to
`dep_targets`, only initially during construction!
This isn't actually necessary with a bit of refactoring, and in general
it makes management of `JobQueue` simpler since there's one less type to
worry about. The one main usage of `Key` vs `Unit` was that `Key` was
`Send` to be placed in `Message`, but this was replaced by just
assigning each `Unit` an ever-increasing integer, and then `Finished`
contains the integer rather than the `Key` itself which we can map once
we get back to the main thread.
This looks like it was a bug ever present from the original
implementation of a GNU jobserver in #4110, but we currently
unconditionally request a token is allocated for any job we pull off our
job queue. Rather we only need to request tokens for everything but the
first job because we already have an implicit token for that job.
I... don't think this needed any more! Doing some digging looks like
this was originally added in 79768eb0d. That was so early it didn't even
use a PR and was almost 5 years ago. Since then we've had a huge number
of changes to the backend and `Unit` nowadays does all the deduplication
we need, so no need to store a `Vec` here and we can just have a mapping
of `Key` to `Job` and that's it.
Improved docs for `maintenance` options
I thought the lack of clear meanings for the available options could use some improvement.
Basically an upgrade of #4431.
Got the relevant text from rust-lang/crates.io#704 and the corresponding [rfc](https://github.com/rust-lang/rfcs/blob/master/text/1824-crates.io-default-ranking.md)
*Questions*
1. Should I put the descriptions outside of the code block?
2. I copied the wording including "maintainer" and "author", should I replace it with "you" or something more appropriate?
I thought the lack of clear meanings for the available options could use some improvement.
Basically an upgrade of #4431.
Got the relevant text from rust-lang/crates.io#704 and the corresponding [rfc](https://github.com/rust-lang/rfcs/blob/master/text/1824-crates.io-default-ranking.md)
*Questions*
1. Should I put the descriptions outside of the code block?
2. I copied the wording including "maintainer" and "author", should I replace it with "you" or something more appropriate?
