Part 2 of RFC2906 -- allow inheriting from a different `Cargo.toml`
Tracking issue: #8415
RFC: rust-lang/rfcs#2906
[Part 1](https://github.com/rust-lang/cargo/pull/10497)
This PR focuses on inheriting from a root workspace:
- Allow inheriting from a different `Cargo.toml`
- Add in searching for a workspace root in `to_real_manifest` as needed
- Fixed problem where a package would try to pull a dependency from a workspace and specify `{ workspace = true, optional = true }` and it would not respect the `optional`
- Added tests to verify everything is in working order
Remaining implementation work for the RFC
- Correctly inherit fields that are relative paths
- Including adding support for inheriting `license_file`, `readme`, and path-dependencies
- Path dependencies infer version directive
- Lock workspace dependencies and warn when unused
- Optimizations, as needed
- Evaluate any new fields for being inheritable (e.g. `rust-version`)
Problems:
- There is duplication of code that can't be removed without significant refactoring
- Potential to parse the same manifest many times when searching for a root
- This should not happen when a `[package]` specifies its workspace
- This should only happen if the workspace root is greater than one folder above
File Cache is valid if checkout or contents hasn't changed
#10482 allow the registry to have cashe keys at a per file level.
On master if the currently checked out commit changes all cached files are regenerated.
After this commit we also check if GITs hash of the particular file has changed.
To avoid thrashing cached files this PR only checks for the presence of both hashes, but does not write them both. This means that nightly after this branch will still be able to share file caches with older versions, specifically current stable.
When sufficient versions of stable know how to read the new format, a one line change can be made to start writing the new format.
Fix how scrape-examples handles proc macros
### What does this PR try to resolve?
This PR fixes#10500.
Previously, the scrape-examples extension did some shenanigans in `build_unit_dependencies` in order to scrape `proc-macro` crates. But this code is useless since `proc-macro` crates cannot export functions, so we avoid this issue entirely by filtering proc-macro targets.
### How should we test and review this PR?
I added the test `scrape_examples_configure_profile` to ensure that the issue in #10500 is fixed.
r? `@ehuss`
Don't error if no binaries were installed
### What does this PR try to resolve?
Fixes#10289, which contains a thorough description of the problem.
Briefly, if we interpret `cargo install` and `cargo install --bins` as "install
the binaries that are available", it should not be considered an error
if no binaries ended up being installed due to required features.
Instead, this should provide the user with a warning that this may not
have been what they intended.
### Additional information
Given that #9576 seems to have stalled, I figured I'd try to land this first [after all](https://rust-lang.zulipchat.com/#narrow/stream/246057-t-cargo/topic/cargo.20install.20--bins.20when.20no.20binaries.20match).
Fix wrong info in "Environment variables" docs
`target_family` can be more than just Unix or Windows, build scripts need to know this and handle it properly.
### What does this PR try to resolve?
Documentation was wrong/misleading
Use the correct flag in --locked --offline error message
### What does this PR try to resolve?
close https://github.com/rust-lang/cargo/issues/10504
[Use the correct the flag in --locked --offline error message](17ec41515e)
[Add offline_and_locked_and_no_frozen test](7363f43d02)
### How should we test and review this PR?
- unit test
Don't treat host/target duplicates as duplicates
### What does this PR try to resolve?
close https://github.com/rust-lang/cargo/issues/9519
Don't treat host/target duplicates as duplicates.
### How should we test and review this PR?
- Unit test.
- Create a manifest where a dependency shows up as both a normal dependency and a build-dependency. Run `cargo tree -d --target x86_64-unknown-linux-gnu`
Unstable --keep-going flag
## Summary
This PR adds an unstable `--keep-going` flag documented as follows:
> `cargo build --keep-going` (and similarly for `check`, `test` etc) will build as many crates in the dependency graph as possible, rather than aborting the build at the first one that fails to build.
>
> For example if the current package depends on dependencies `fails` and `works`, one of which fails to build, `cargo check -j1` may or may not build the one that succeeds (depending on which one of the two builds Cargo picked to run first), whereas `cargo check -j1 --keep-going` would definitely run both builds, even if the one run first fails.
>
> The `-Z unstable-options` command-line option must be used in order to use `--keep-going` while it is not yet stable:
>
> ```console
> cargo check --keep-going -Z unstable-options
> ```
## Prior art
[Buck](https://buck.build/) and [Bazel](https://bazel.build/) and Make all have this flag (though Bazel calls it `--keep_going` 🤮) with exactly this behavior.
## Motivation
I need this in order to make https://github.com/dtolnay/trybuild not super slow.
Trybuild wants to run Cargo on a bunch of test cases, each of which is a bin crate. The bad options currently available are:
- Give each test case its own target dir and run build on them in parallel. This is bad because all the test cases have the same dependencies in common (whatever `dev-dependencies` are declared by the project). If there are 100 test cases, all the dependencies would end up getting built 100 times, which is 100x slower than necessary despite the parallelism.
- Reuse a single target dir for all the test cases. Two Cargos can't operate in parallel on the same target directory, so this forces the test cases to be built serially. This is much slower than necessary on a many-core system, and compounds all of the overheads in Cargo because the project structure must be reloaded by each invocation.
The good option I'd like to switch to is:
- Run `cargo build --bins --keep-going --message-format=json` to build *all* the test cases in parallel. Use the filepaths in the JSON messages to ascribe diagnostics to which bin they're from.
If we interpret `cargo install` and `cargo install --bins` as "install
the binaries that are available", it should not be considered an error
if no binaries ended up being installed due to required features.
Instead, this should provide the user with a warning that this may not
have been what they intended.
Fixes#10289.
Part 1 of RFC2906 - Packages can inherit fields from their root workspace
Tracking issue: #8415
RFC: rust-lang/rfcs#2906
All changes were heavily inspired by #8664 and #9684
A big thanks to both `@yaymukund` and `@ParkMyCar.` I pulled a lot of inspiration from their branches.
I would also like to thank `@alexcrichton` for all the feedback on the first attempt. It has brought this into a much better state.
All changes have been made according to the RFC as well as `@alexcrichton's` [comment](https://github.com/rust-lang/cargo/pull/8664#issuecomment-704878103).
This is part 1 of many, as described by [this comment](https://github.com/rust-lang/cargo/pull/9684#issuecomment-943567692), [this comment](https://github.com/rust-lang/cargo/pull/9684#pullrequestreview-779070283) and redefined [by this one](https://github.com/rust-lang/cargo/pull/10497#issuecomment-1076301312).
This PR focuses on inheriting in root package, including:
- Add `MaybeWorkspace<T>` to allow for `{ workspace = true }`
- Add a new variant to `TomlDependency` to allow inheriting dependencies from a workspace
- Add `InheritableFields` so that we have somewhere to get a value from when we `resolve` a `MaybeWorkspace<T>`
- `license_file` and `readme` are in `InheritableFields` in this part but are not `MaybeWorkspace` for reasons [described here](https://github.com/rust-lang/cargo/pull/10497#issuecomment-1076470424)
- Add a method to `resolve` a `MaybeWorkspace<T>` into a `T` that can fail if we have nowhere to pull from or the workspace does not define that field
- Disallow inheriting from a different `Cargo.toml`
- This means that you can only inherit from inside the same `Cargo.toml`, i.e. it has both a `[workspace]` and a `[package]`
- Forcing this requirement allows us to test the implementations of `MaybeWorkspace<T>` and the new `TomlDependency` variant without having to add searching for a workspace root and the complexity with it
Remaining implementation work for the RFC
- Support inheriting in any workspace member
- Correctly inherit fields that are relative paths
- Including adding support for inheriting `license_file`, `readme`, and path-dependencies
- Path dependencies infer version directive
- Lock workspace dependencies and warn when unused
- Optimizations, as needed
- Evaluate any new fields for being inheritable (e.g. `rust-version`)
- Evaluate making users of `{ workspace = true }` define the workspace they pull from in `[package]`
Areas of concern:
- `TomlDependency` Deserialization is a mess, and I could not figure out how to do it in a cleaner way without significant headaches. I ended up having to do the same thing as last time [which was an issue](https://github.com/rust-lang/cargo/pull/9684#discussion_r728444103).
- Resolving on a `MaybeWorkspace` feels extremely verbose currently:
```rust
project.homepage.clone().map(|mw| mw.resolve(&features, "homepage", || get_ws(inheritable)?.homepage())).transpose()?,
```
This way allows for lazy resolution of inheritable fields and finding a workspace (in part 2) so you do not pay a penalty for not using this feature. The other bit of good news is `&features` will go away as it is just feature checking currently.
- This feature requires some level of duplication of code as well as remaking the original `TomlManifest` which adds extra length to the changes.
- This feature also takes a linear process and makes it potentially non-linear which adds lots of complexity (which will get worse in Part 2)
Please let me know if you have any feedback or suggestions!
Remove unused profile support for -Zpanic-abort-tests
This removes the vestigial `PanicSetting::Inherit` setting.
This was initially introduced in #7460 which added `-Zpanic-abort-tests`. This was needed at the time because `test` and `dev` profiles were separate, but they were inter-mixed when running `cargo test`. That would cause a problem if the unwind/abort settings were mixed. However, with named profiles, `test` now inherits from `dev`, so this is no longer necessary. Now that named profiles are stable, support for the old form is no longer necessary.
HTTP registry implementation
Implement HTTP registry support described in [RFC 2789](https://github.com/rust-lang/rfcs/pull/2789).
Adds a new unstable flag `-Z http-registry` which allows cargo to interact with remote registries served over http rather than git. These registries can be identified by urls starting with `sparse+http://` or `sparse+https://`.
When fetching index metadata over http, cargo only downloads the metadata for needed crates, which can save significant time and bandwidth over git.
The format of the http index is identical to a checkout of a git-based index.
This change is based on `@jonhoo's` PR #8890.
cc `@Eh2406`
Remaining items:
- [x] Performance measurements
- [x] Make unstable only
- [x] Investigate unification of download system. Probably best done in separate change.
- [x] Unify registry tests (code duplication in `http_registry.rs`)
- [x] Use existing on-disk cache, rather than adding a new one.
Add tests for ignoring symlinks
This adds tests for the expected behavior in https://github.com/rust-lang/cargo/issues/10032. Interestingly, these tests pass (🎉). Will update that issue with more details shortly, but figured these tests were worthwhile to add to the testsuite anyway now that I've written them.
-- `resolve` not takes `get_ws: impl FnOnce() -> CargoResult<T>`
-- removed `MaybeWorkspace` from `readme` and `license-file`
-- changed error messages and test names
Update doc string for deps_of/compute_deps.
I noticed the `compute_deps` doc string was outdated due to some recent refactorings. This updates the doc comments to try to clarify them and make them more accurate.
