Commit graph

60 commits

Author SHA1 Message Date
Ed Page
675224b3a0 test(config): Shift to config.toml 2024-01-26 13:40:46 -06:00
Ed Page
633929d3aa refactor(schemas): Pull out cargo-util-schemas crate
Fixes #12801
2023-12-15 13:22:25 -06:00
Ed Page
378502bf20 refactor: Try to make PackageId conversion clearer 2023-12-01 10:48:14 -06:00
Ed Page
e46df98241 refactor(schemas): Pull out mod for proposed schemas package
Originally for #12801 we talked about a `cargo-util-manifest-schema` package
- `util` in the name to not clash with plugins
- manifest specific to keep the scope down

The problem is we have types that aren't manifest specific, like
- `PartialVersion` (currently slated for `cargo-util-semverext`)
- `RustVersion`
- `PackageIdSpec`
- `SourceKind` (soon)

Things get messy if we try to break things down into common packages.
Instead, I think it'd be useful to have a schemas package that has mods
for each type of schema, re-exporting what is needed.

Normally, componentizing your package by the layer in the stack is a
recipe for pain.
I don't think that'll apply here because these are meant to be so low
level.

The other big concern could be compile times.  My hope is it won't be
too bad.

So this moves the `util/toml` types into the module and we can add more
in the future.
2023-12-01 13:31:30 -06:00
Ed Page
9a1bbc9625 refactor(toml): Pull out schema
Remaining steps
- Decouple `toml/mod.rs` functionality from `toml/schema.rs`
- Pull out `core/` types referenced by `toml/schema.rs`
2023-11-01 20:59:46 -05:00
Ed Page
e4e10f2393 chore: Fix typos
This is a repeat of #11561
2023-09-19 15:28:48 -05:00
Weihang Lo
e575448574
refactor: flatten module path of SourceId 2023-09-07 21:06:35 +08:00
Weihang Lo
64a1f204f3
refactor: only when -C debuginfo > 0 will spilt-debuginfo be passed
It was unnecessary to pass `spilt-debuginfo` if there is no debuginfo.
Tests are touched here only for matching rustflags invocation stderr
in the original test suite.
2023-05-31 20:43:57 +01:00
jyn
e2eacc6414 Don't distinguish Debuginfo::None and Debuginfo::Explicit(None)
Previously, `Debuginfo::None` meant "don't pass -C debuginfo" and `Explicit(None)` meant
"-C debuginfo=0", which occasionally led to caching bugs where cargo would sometimes pass
`-C debuginfo=0` and sometimes not. There are no such bugs currently that we know of, but
representing them the same within cargo avoids the possibility of the bug popping up again in the
future.

I tested the `with_stderr_does_not_contain_tests` with this diff to ensure they did not pass:
```diff
diff --git a/src/cargo/core/compiler/mod.rs b/src/cargo/core/compiler/mod.rs
index 55ec17182..c186dd00a 100644
--- a/src/cargo/core/compiler/mod.rs
+++ b/src/cargo/core/compiler/mod.rs
@@ -1073,9 +1073,7 @@ fn build_base_args(

     let debuginfo = debuginfo.into_inner();
     // Shorten the number of arguments if possible.
-    if debuginfo != TomlDebugInfo::None {
         cmd.arg("-C").arg(format!("debuginfo={}", debuginfo));
-    }

     cmd.args(unit.pkg.manifest().lint_rustflags());
     if !rustflags.is_empty() {
```
2023-05-30 17:06:50 -05:00
Joshua Nelson
d861dcf792 Allow named debuginfo options in Cargo.toml
Rustc supports these since rust-lang/rust#109808. It's technically
possible to set a named debuginfo level through `RUSTFLAGS`, but in
practice cargo always passes its own opinion of what the debuginfo level
is, so allow it to be configured through cargo too.
2023-04-11 15:06:02 -05:00
Rémy Rakic
79dd5117da introduce dedicated DebugInfo enum in Profiles
This enum will be used to model the current Option<u32> value in
profiles, as the explicitly set value, and also allow to model a
deferred value: one that can be ignored for optimization purposes,
and used in all other cases.

This allows to have a default debuginfo for build dependencies, that can
change:
- if a dependency is shared between the build and runtime subgraphs, the
  default value will be used, to that re-use between units will
  continue.
- otherwise, a build dependency is only used in a context where
  debuginfo is not very useful. We can optimize build times in this
  situation by not asking rustc to emit debuginfo.
2023-01-31 19:05:20 +00:00
Weihang Lo
629f763be8
Support codegen-backend and rustflags in profiles in config file 2023-01-14 11:14:40 +00:00
Ed Page
cdec3838a8 Extend pkgid syntax with @ support
In addition to `foo:1.2.3`, we now support `foo@1.2.3` for pkgids.  We
are also making it the default way of rendering pkgid's for the user.

With cargo-add in #10472, we've decided to only use `@` in it and to add
it as an alternative to `:` in the rest of cargo.  `cargo-add`
originally used `@`.  When preparing it for merge, I switched to `:` to
be consistent with pkgids. When discussing this, it was felt `@` has
precedence in too many tools to switch to `:` but that we should instead
switch pkgid's to use `@`, in a backwards compatible way.

See also https://internals.rust-lang.org/t/feedback-on-cargo-add-before-its-merged/16024/26?u=epage
2022-04-19 16:00:15 -05:00
Weihang Lo
555209ce06
Stop gating feature "named-profiles" 2022-02-28 23:23:26 +08:00
bors
5aad9b302a Auto merge of #9992 - Byron:rfc-3028, r=ehuss
Implement "artifact dependencies" (RFC-3028)

Tracking issue: #9096

#### Tasks

* [x] -Z unstable option called `bindeps`
*  **(config)** allow parsing of newly introduced 'artifact' fields
   * [x] into `TomlManifest`
   * [x] into `Manifest`
      - [x] ~~abort~~ warn if artifacts are used on stable
*  **resolver** : assure artifact dependencies are part of the resolution process and unified into the dependency tree
* 🔬**compiler**: make it understand 'artifact' dependencies and pass new environment variables to the crate being build
  * [x] `lib=false` should not be considered a rust library for the dependent, in unit and possibly resolve graph
  * [x] assure profile settings are applied correctly
  * [x] target overrides work
      * [x] `target = "target"` in build deps
      * [x] other targets on build deps
      * [x] other targets on non-build deps
      * [x] 'no-cross doc tests' seems like a constraint we should apply as well maybe
  * [x] more confidence with `resolver = "2"`
  * [x] assure artifact placement is correct (bin and various forms of lib)
*  **serialization**: rewriting manifests (i.e. for publishing) does not discard artifact information
   * [x] publishing keeps `artifact` and `lib` values
* **Other cargo subcommands**
   * [x] `cargo metadata`
        * leave unchanged
   * [x] artifacts work with `cargo check`
   * [x] artifacts work with rustdoc, such that it doesn't document them unless `lib=true`
   * [x] `cargo tree` maybe?
   * [x] `cargo clean` should clean artifacts - even though it's more complex, ultimately it deletes the `target` directory.
   * [x] artifacts work with `cargo test` (and dev-dependencies)
       * [x] doctests
       * [x] try `reproducible` repository as well.
* 🧪 **tests** for more subtle RFC constraints
   - [x] build scripts cannot access artifact environment variables at compile time, only at runtime)
   - [x] assure 'examples' which also support crate-type fields like [[lib]] won't become artifacts themselves.
   - [x] assure `--out-dir` does not leak artifacts - tested manually, it seemed to niche to add a test.
   - [x] try `target="foo"` in artifact and assure it sees a decent error message
   - [x] Assure RFC 3176 _doesn't_ work
* 🧹cleanup and finalization
    - [x] assure no `TODO(ST)` markers are left in code
    - [x] assure no tests are ignored
    - [x] use `resolver = "1"` once to assert everything also works with the previous resolver, but leave it on "2".

#### Implementation and review notes

- artifacts and unstable options are only checked when transforming them from `TomlManifest` to `Manifest`, discarding artifact information if the unstable flag is not set. Nowhere else in code will the CLI options be checked again.
- `If no binaries are specified, all the binaries in the package will be built and made available.` - this should only refer to `[[bin]]` targets, not examples for instance.
- artifact binaries won't be uplifted, hence won't be present outside of their output directory
- ️We don't know how [package links](00e925f61f/src/cargo/core/compiler/unit_dependencies.rs (L380)) will affect artifacts for build dependencies. Should probably be thought through.
- ️The location of artifacts is only tested roughly to avoid having to deal with different output names on the four platforms that seem to matter (gnu, macos, windows msvc, windows gnu).
- `cargo tree` doesn't handle artifacts specifically, and it might be interesting to make clear if an artifact is only an artifact, or both artifact and dependency.
- Most error and warning messages can probably be more cargo-matic.

#### Questions

* Does `cargo` without the feature enabled have to complain about the "artifact" field in a dependency, like it does right now?  It doesn't look like machinery for that exists in `do_read_manifest()`.
   - ✔️It warns now
*  Should parsing of artifact values, like "bin" be case sensitive?
   - ✔️ It's case sensitive now, which should help with serde roundtripping.

#### Review Progress

* [x] address Josh's review notes one by one
   * [x] introduce `IsArtifact` (see [this answer](https://github.com/rust-lang/cargo/pull/9992#discussion_r774928102)) (76cd48a2d62d74e043a1a482199c5bb920f50311)
   * [x] prefer uplifting artifact deps that were written into the `deps` directory, but prefer to do that in [this PR instead](https://github.com/rust-lang/cargo/pull/9992)
   * [x] add extra-tests as described in Josh's comment: " features get unified between a Rust library and a binary, and one that confirms features don't get unified between a Rust library and a binary for a different target?"
   * [x] Make target-based artifact splitting work by porting parts of RFC-3176
       * [x] test-support/cross-compile
       * [x] namespace separation
* [x] re-create RFC-3176 what's not in RFC-3028, namely multidep support and all related tests
* [x] Address Eh2406 's review comments
* [x] Address Eric's comments
   * [x] Unstable features need to be documented in unstable.md.
   * [x] sort out [`target_data`](https://github.com/rust-lang/cargo/pull/9992#discussion_r787316229)
   * [x] figure out [cargo metadata](https://github.com/rust-lang/cargo/pull/9992#discussion_r787320923)
   * [x] See if target-data can work without an [index format update](https://github.com/rust-lang/cargo/pull/9992#issuecomment-1021697124).
   * [x] truncate comments at 80-90 lines and remove unused methods, remove -Z unstable-options, use `cfg!(target_env = "msvc")`
   * [x] add missing doc comments to newly added methods and funtions
   * [x] simplify method parameters and inline some functions
   * [x] add test and extend test to show additional issues
* [x] assure current set of tests works consistently, also on windows

Sponsored by [Profian](https://www.profian.com)
2022-02-22 03:56:46 +00:00
Sebastian Thiel
7248f4b70d
add support for artifact dependencies (#9096)
Tracking issue: https://github.com/rust-lang/cargo/issues/9096
Original PR: https://github.com/rust-lang/cargo/pull/9992

Add 'bindeps' -Z flag for later use

A test to validate artifact dependencies aren't currently parsed.

Parse 'artifact' and 'lib' fields.

Note that this isn't behind a feature toggle so 'unused' messages will
disappear.

Transfer artifact dependencies from toml- into manifest-dependencies

There are a few premises governing the operation.

- if unstable features are not set, warn when 'artifact' or 'lib' is
  encountered.
- bail if 'lib' is encountered alone, but warn that this WOULD happen
  with nightly.
- artifact parsing checks for all invariants, but some aren't tested.

Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing

This should be the only place were these fields matter and where a cargo
manifest is actually produced. These are only for internal use, no user
is typically going to see or edit them.

Place all artifact dependency tests inta their own module

This facilitates deduplication later and possibly redistribution into
other modules if there is a better fit.

Represent artifacts that are rust libraries as another ArtifactKind

This is more consistent and probably simpler for later use.
No need to reflect the TOML data structure.

Add tests to assure only 'lib = true' artifact deps are documented

RFC-3028 doesn't talk about documentation, but for lib=true it's clear
what the desired behaviour should be.
If an artifact isn't a library though, then for now, it's transparent,
maybe.

Many more tests, more documentation, mild `Artifact` refactor

The latter seems to be a better fit for what being an artifact
really means within cargo, as it literally turns being a library
on or off, and thus only optionally becoming a normal library.

refactor to prepare for artifact related checks

Don't show a no-lib warning for artifact dependencies (with lib = false)

Tests for more artifact dependency invariants

These are merely a proof of concept to show that we are not in
a position to actually figure out everything about artifacts
right after resolution.

However, the error message looks more like a fatal error and less
like something that can happen with a more elaborate error message
with causes.

This might show that these kind of checks might be better done later
right before trying to use the information for create compile units.

Validate that artifact deps with lib=true still trigger no-lib warnings

This triggers the same warning as before, for now without any
customization to indicate it's an artifact dependency.

Use warnings instead of errors
------------------------------

This avoids the kind of harsh end of compilation in favor of something
that can be recovered from. Since warnings are annoying, users will
probably avoid re-declaring artifact dependencies.

Hook in artifact dependencies into build script runs

Even though we would still have to see what happens if they have a lib
as well. Is it built twice?

Also
----

- fly-by refactor: fix typo; use ? in method returning option
- Propagate artifact information into Units; put artifacts into place

  This means artifacts now have their own place in the 'artifact'
  directory and uplifts won't happen for them.

- refactor and fix cippy suggestion
- fix build after rebasing onto master

Create directories when executing the job, and not when preparing it.

also: Get CI to work on windows the easy way, for now.

Set directories for artifact dependencies in build script runtimes

Test remaining kinds of build-script runtime environment variables

Also
----
- Fix windows tests, the quick way.
- Try to fix windows assertions, and generalize them
- Fix second test for windows, hopefully

test for available library dependency in build scripts with lib = true

probably generally exclude all artifact dependencies with lib=false.

Pass renamed dep names along with unit deps to allow proper artifact env names

Test for selective bin:<name> syntax, as well as binaries with dashes

Test to assure dependency names are transformed correctly

assure advertised binaries and directories are actually present

This wouldn't be the case if dependencies are not setup correctly,
for instance.

Also
----
 - make it easier to see actual values even on failure

   This should help figure out why on CI something fails that works
   locally no matter what.
   Turns out this is a race condition, with my machine being on the good
   side of it so it doesn't show in testing. Fortunately it still can be
   reproduced and easily tested for.

 - refactor test; the race condition is still present though

 - Force CI to pass here by avoiding checks triggering race.

 - Fix windows build, maybe?

More tolerant is_file() checks to account for delay on CI

This _should_ help CI to test for the presence which is better than
not testing at all.

This appears to be needed as the output file isn't ready/present in time
for some reason.

The root cause of this issue is unknown, but it's definitely a race
as it rarely happens locally. When it happened, the file was always
present after the run.
Now we will learn if it is truly not present, ever, or if it's maybe
something very else.

Validate libs also don't see artifact dependencies as libraries with lib=false

Also
----

 - Add prelimiary test for validating build-time artifacts
 - Try to fix CI on gnu windows

   Which apparently generates paths similar to linux, but with .exe suffix.
   The current linux patterns should match that.

 - refactor

   Help sharing code across modules

allow rustc to use artifact dep environment variables, but…

…it needs some adjustments to actually setup the unit dependency graph
with artifacts as well.
Right now it will only setup dependencies for artifacts that are libs,
but not the artifacts themselves, completely ignoring them when they
are not libs.

Make artifact dependencies available in main loop

This is the commit message #2:
------------------------------

rough cut of support for artifact dependencies at build time…

…which unfortunately already shows that the binary it is supposed to
include is reproducibly not ready in time even though the path is
correct and it's present right after the run.

Could it be related to rmeta?

This is the commit message #3:
------------------------------

Fix test expectations as failure is typical than the warning we had before…

…and add some tolerance to existing test to avoid occasional failures.

This doesn't change the issue that it also doens't work at all for
libraries, which is nicely reproducable and hopefully helps to fix
this issue.

This is the commit message #4:
------------------------------

Probably the fix for the dependency issue in the scheduler

This means that bin() targets are now properly added to the job graph
to cause proper syncing, whereas previously apparently it would
still schedule binaries, but somehow consider them rmeta and thus
start their dependents too early, leading to races.

This is the commit message #5:
------------------------------

Don't accidentally include non-gnu windows tests in gnu windows.

Support cargo doc and cargo check

The major changes here are…

- always compile artifacts in build mode, as we literally want the
  build output, always, which the dependent might rely on being present.
- share code between the rather similar looking paths for rustdoc and
  rustc.

Make artifact messages appear more in line with cargo by using backticks

Also: Add first test for static lib support in build scripts

build-scripts with support for cdylib and staticlib

 - Fix windows msvc build

   No need to speculate why the staticlib has hashes in the name even
   though nothing else.

staticlib and cdylib support for libraries

test staticlib and cdylibs for rustdoc as well.

Also catch a seemingly untested special case/warning about the lack
of linkable items, which probably shouldn't be an issue for artifacts
as they are not linkable in the traditional sense.

more useful test for 'cargo check'

`cargo check` isn't used very consistently in tests, so when we use it
we should be sure to actually try to use an artifact based feature
to gain some coverage.

verify that multiple versions are allowed for artifact deps as well.

also: remove redundant test

This is the commit message #2:
------------------------------

Properly choose which dependencies take part in artifact handling

Previously it would include them very generously without considering
the compatible dependency types.

This is the commit message #3:
------------------------------

a more complex test which includes dev-dependencies

It also shows that doc-tests don't yet work as rustdoc is run outside of
the system into which we integrate right now.

It should be possible to write our environment variable configuration
in terms of this 'finished compilation' though, hopefully with
most code reused.

This is the commit message #4:
------------------------------

A first stab at storing artifact environment variables for packages…

…however, it seems like the key for this isn't necessarily correct
under all circumstances. Maybe it should be something more specific,
don't know.

This is the commit message #5:
------------------------------

Adjust key for identifying units to Metadata

This one is actually unique and feels much better.

This is the commit message #6:
------------------------------

Attempt to make use of artifact environment information…

…but fail as the metadata won't match as the doctest unit is, of course,
its separate unit. Now I wonder if its possible to find the artifact
units in question that have the metadata.

Properly use metadata to use artifact environment variables in doctests

This is the commit message #2:
------------------------------

Add test for resolver = "2" and build dependencies

Interestingly the 'host-features' flag must be set (as is seemingly
documented in the flags documentation as well), even though I am not
quite sure if this is the 100% correct solution. Should it rather
have an entry with this flag being false in its map? Probably not…
but I am not quite certain.

This is the commit message #3:
------------------------------

set most if not all tests to use resolver = "2"

This allows to keep it working with the most recent version while
allowing to quickly test with "1" as well (which thus far was working
fine).

All tests I could imagine (excluding target and profiles) are working now

Crossplatform tests now run on architecture aarm64 as well.

More stringent negative testing

Fix incorrect handling of dependency directory computation

Previously it would just 'hack' the deps-dir to become something very
different for artifacts.

This could easily be fixed by putting the logic for artifact output
directories into the right spot.

A test for cargo-tree to indicate artifacts aren't handled specifically

Assure build-scripts can't access artifacts at build time

Actual doc-tests with access to artifact env vars

All relevant parsing of `target = [..]`

Next step is to actually take it into consideration.

A failing test for adjusting the target for build script artifacts using --target

Check for unknown artifact target triple in a place that exists for a year

The first test showing that `target="target"` deps seemingly work

For now only tested for build scripts, but it won't be much different
for non-build dependencies.

build scripts accept custom targets unconditionally

Support target setting for non-build dependencies

This is the commit message #2:
------------------------------

Add doc-test cross compile related test

Even though there is no artifact code specific to doc testing, it's
worth to try testing it with different target settings to validate
it still works despite doc tests having some special caseing around
target settings.

This is the commit message #3:
------------------------------

A test to validate profiles work as expected for build-deps and non-build deps

No change is required to make this work and artifact dependencies 'just work'
based on the typical rules of their non-artifact counterarts.

This is the commit message #4:
------------------------------

Adjust `cargo metadata` to deal with artifact dependencies

This commit was squashed and there is probably more that changed.

This is the commit message #5:
------------------------------

Show bin-only artifacts in "resolve" of metadata as well.

This is the commit message #6:
------------------------------

minor refactoring during research for RFC-3176

This will soon need to return multiple extern-name/dep-name pairs.

This is the commit message #7:
------------------------------

See if opt-level 3 works on win-msvc in basic profile test for artifacts

This is the same value as is used in the other test of the same name,
which certainly runs on windows.

This is the commit message #8:
------------------------------

refactor

Assure the type for targets reflect that they cannot be the host target,
which removes a few unreachable!() expressions.

Put `root_unit_compile_kind` into `UnitFor`

Previously that wasn't done because of the unused `all_values()`
method which has now been deleted as its not being used anyomre.

This allows for the root unit compile kind to be passed as originally
intended, instead of working around the previous lack of extendability
of UnitFor due to ::all_values().

This is also the basis for better/correct feature handling once
feature resolution can be depending on the artifact target as well,
resulting in another extension to UnitFor for that matter.

Also
----

 - Fix ordering

   Previously the re-created target_mode was used due to the reordering
   in code, and who knows what kind of effects that might have
   (despite the test suite being OK with it).

   Let's put it back in place.

 - Deactivate test with filename collision on MSVC until RFC-3176 lands

Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/'

This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834

Don't rely on operator precedence for boolean operations

Now it should be clear that no matter what the first term is,
if the unit is an artifact, we should enqueue it.

Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)`

fix `doc::doc_lib_false()` test

It broke due to major breakage in the way dependencies are calculated.

Now we differentiate between deps computation for docs and for building.

Avoid testing for doctest cross-compilation message

It seems to be present on my machine, but isn't on linux and it's
probably better to leave it out entirely and focus on the portions
of consecutive output that we want to see at least.

A test to validate features are unified across libraries and those in artifact deps in the same target

Allow aarch64 MacOS to crosscompile to an easily executable alternative target

That way more tests can run locally.

Support for feature resolution per target

The implementation is taken directly from RFC-3176 and notably lacks
the 'multidep' part.

Doing this definitely has the benefit of making entirely clear
'what is what' and helps to greatly reduce the scope of RFC-3176
when it's rebuilt based on the latest RF-3028, what we are implementing
right now.

Also
----
- A test which prooves that artifact deps with different target don't have a feature namespace yet

- Add a test to validate features are namespaced by target

  Previously it didn't work because it relies on resolver = "2".

- 'cargo metadata' test to see how artifact-deps are presented

- Missed an opportunity for using the newly introduced `PackageFeaturesKey`

- Use a HashMap to store name->value relations for artifact environment variables

  This is semantically closer to what's intended.

  also: Remove a by now misleading comment

Prevent resolver crash if `target = "target"` is encountered in non-build dependencies

A warning was emitted before, now we also apply a fix.

Previously the test didn't fail as it accidentally used the old
resolver, which now has been removed.

Abort in parsing stage if nightly flag is not set and 'artifact' is used

There is no good reason to delay errors to a later stage when code
tries to use artifacts via environment variables which are not present.

Change wording of warning message into what's expected for an error message

remove unnecessary `Result` in `collect()` call

Improve logic to warn if dependencie are ignored due to missing libraries

The improvement here is to trigger correctly if any dependency of a
crate is potentially a library, without having an actual library target
as part of the package specification.

Due to artifact dependencies it's also possible to have a dependency
to the same crate of the same version, hence the package name
isn't necessarily a unique name anymore. Now the name of the actual
dependency in the toml file is used to alleviate this.

Various small changes for readability and consistency

A failing test to validate artifacts work in published crates as well

Originally this should have been a test to see target acquisition works
but this more pressing issue surfaced instead.

Make artifacts known to the registry data (backwards compatible)

Now artifacts are serialized into the registry on publish (at least
if this code is actually used in the real crates-io registry) which
allows the resolve stage to contain artifact information.

This seems to be in line with the idea to provide cargo with all
information it needs to do package resolution without downloading
the actual manifest.

Pick up all artifact targets into target info once resolve data is available

Even though this works in the test at hand, it clearly shows there
is a cyclic dependency between the resolve and the target data.

In theory, one would have to repeat resolution until it settles
while avoiding cycles.

Maybe there is a better way.

Add `bindeps`/artifact dependencies to `unstsable.md` with examples

Fix tests

Various small improvements

Greatly simplify artifact environment propagation to commands

Remove all adjustments to cargo-metadata, but leave tests

The tests are to record the status quo with the current code
when artifact dependencies are present and assure the information
is not entirely non-sensical.

Revert "Make artifacts known to the registry data (backwards compatible)"

This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0.

Ideally we are able to make it work without altering the registry
storage format. This could work if information from the package
set is added to the resolve information.

Enrich resolves information with additional information from downloaded manifests

Resolve information comes from the registry, and it's only as rich as
needed to know which packages take part in the build.

Artifacts, however, don't influence dependency resolution, hence it
shouldn't be part of it.

For artifact information being present nonetheless when it matters,
we port it back to the resolve graph where it will be needed later.

Collect 'forced-target' information from non-workspace members as well

This is needed as these targets aren't present in the registry and
thus can't be picked up by traversing non-workspce members.

The mechanism used to pick up artifact targets can also be used
to pick up these targets.

Remove unnecessary adjustment of doc test

refactor `State::deps()` to have filter; re-enable accidentally disabled test

The initial rebasing started out with a separted `deps_filtered()`
method to retain the original capabilities while minimizing the chance
for surprises. It turned out that the all changes combined in this PR
make heavy use of filtering capabilities to the point where
`deps(<without filter>)` was unused. This suggested that it's required
to keep it as is without a way to inline portions of it.

For the original change that triggered this rebase, see

bd45ac81ba

The fix originally made was reapplied by allowing to re-use the
required filter, but without inlining it.

Always error on invalid artifact setup, with or without enabled bindeps feature

Clarify how critical resolver code around artifact is working

Remove workaround in favor of deferring a proper implementation

See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197
for reference and the TODO in the ignored test for more information.

truncate comments at 80-90c; cleanup

- remove unused method
- remove '-Z unstable-options'
- improve error message
- improve the way MSVC special cases are targetted in tests
- improve how executables are found on non MSVC

Avoid depending on output of rustc

There is cyclic dependency between rustc and cargo which makes it
impossible to adjust cargo's expectations on rustc without leaving
broken commits in rustc and cargo.

Add missing documentation

fix incorrect removal of non-artifact libs

This is also the first step towards cleaning up the filtering logic
which is still making some logic harder to understand than needs be.

The goal is to get it to be closer to what's currently on master.

Another test was added to have more safety regarding the overall
library inclusion logic.

inline `build_artifact_requirements_to_units()`

Simplify filtering

This adds a default filter to `state.deps(…)` making it similar to
what's currently in master, while creating another version of it
to allow setting a custom filter. This is needed as the default filter
won't allow build dependencies, which we need in this particular case.

`calc_artifact_deps(…)` now hard-codes the default filter which is
needed due to the use of `any` here:
c0e6abe384/src/cargo/core/compiler/unit_dependencies.rs (L1119)
.

Simplify filtering.
2022-02-22 08:08:42 +08:00
Yerkebulan Tulibergenov
eca63cfd4e Merge branch 'master' into yerke/no-run-test-executable-path 2022-01-30 23:13:35 -08:00
Vaibhav
bef4d79ff2 Print executable name on cargo test --no-run.
Closes #9957

Signed-off-by: Vaibhav <vrongmeal@gmail.com>
2021-10-05 23:47:38 +05:30
Eric Huss
895f52714c Stabilize named profiles. 2021-09-24 10:00:42 -07:00
Eric Huss
2c99f654c9 Update nightly failure notification. 2021-07-05 16:08:36 -07:00
Alex Crichton
a4f0988ef8 Default macOS targets to unpacked debuginfo
This commit continues the work from #9112 to enable `unpacked` split
debuginfo on macOS targets by default. This has been discussed on [internals]
for awhile now and no breakage has emerged while significant speedups
have. This is expected to be a compile-time and `target`-directory size
win for almost all macOS Rust projects.

While breakage is possible it's possible to mitigate this with
project-local or global cargo configuration of the `dev` and `test` profiles.

[internals]: https://internals.rust-lang.org/t/help-test-faster-incremental-debug-macos-builds-on-nightly/14016/9
2021-03-26 07:38:17 -07:00
Joshua Nelson
ecfdced0d8 Fix test that assumed tests always were run on the stable channel
Note that this has to be set in the builder, before
`config.configure()` gets run.
2021-02-24 15:11:34 -05:00
Joshua Nelson
4b096beaed Fix masquerade_as_nightly_cargo in work threads
Previously, since `ENABLE_NIGHTLY_FEATURES` and
`NIGHTLY_FEATURES_ENABLED` were thread locals, reading them in any other
thread would always say nightly features were disabled. Now, they are
tied to the `Context` itself, so it is both more clear how the variables
are being set and fixes the behavior within work threads.

Note that `Context` is not thread-safe, so this passes a boolean through
to `BuildOutput::parse`.
2021-02-24 14:43:00 -05:00
Alex Crichton
ffa9dbd348 Don't change the macOS default just yet 2021-02-03 14:06:41 -08:00
Alex Crichton
ed4568e108 Add split-debuginfo profile option
This commit adds a new `split-debuginfo` option to Cargo compilation
profiles which gets forwarded to the `-Csplit-debuginfo` codegen option
in rustc. This commit also sets the default, only on macOS, to be
`-Csplit-debuginfo=unpacked`. The purpose of this change is to leverage
rust-lang/rust#79570 to avoid running `dsymutil` on incremental builds
while also preserving a pleasant debugging experience by default. This
should lead to much faster incremental build times on macOS since
`dsymutil` isn't exactly the speediest tool in the world.

This is technically a breaking change in Cargo because we're no longer
by-default producing the `*.dSYM` folders on macOS. If those are still
desired, however, authors can always run `dsymutil` themselves or
otherwise configure `split-debuginfo = 'packed'` in their
manifest/profile configuration.
2021-02-01 09:21:36 -08:00
Eric Huss
6f8c7d5a87 Normalize raw string indentation. 2020-09-26 17:59:58 -07:00
Alex Crichton
dc4b695f41 Build host dependencies with opt-level 0 by default
This commit updates Cargo's build of host dependencies to build them
with optimization level 0 by default instead of matching the profile of
the final binary.

Since Cargo's inception build dependencies have, by default, been built
in a profile that largely matches the profile of the final target
artifact. Build dependencies, however, rarely actually need to be
optimized and are often executing very small tasks, which means that
optimizing them often wastes a lot of build time. A great example of
this is procedural macros where `syn` and friends are pretty heavyweight
to optimize, and the amount of Rust code they're parsing is typically
quite small, so the time spent optimizing rarely comes as a benefit.

The goal of this PR is to improve build times on average in the
community by not spending time optimizing build dependencies (build
scripts, procedural macros, and their transitive dependencies). The PR
will not be a universal win for everyone, however. There's some
situations where your build time may actually increase:

* In some cases build scripts and procedural macros can take quite a
  long time to run!
* Cargo may not build dependencies more than once if they're shared with
  the main build. This only applies to builds without `--target` where
  the same crate is used in the final binary as in a build script.

In these cases, however, the `build-override` profile has existed for
some time know and allows giving a knob to tweak this behavior. For
example to get back the previous build behavior of Cargo you would
specify, in `Cargo.toml`:

    [profile.release.build-override]
    opt-level = 3

or you can configure this via the environment:

    export CARGO_PROFILE_RELEASE_BUILD_OVERRIDE_OPT_LEVEL=3

There are two notable features we would like to add in the future which
would make the impact of a change like this smaller, but they're not
implemented at this time (nor do we have concrete plans to implement
them). First we would like crates to have a way of specifying they
should be optimized by default, despite default profile options. Often
crates, like lalrpop historically, have abysmal performance in debug
mode and almost always (even in debug builds) want to be built in
release mode. The second feature is that ideally crate authors would be
able to tell Cargo to minimize the number of crates built, unifying
profiles where possible to avoid double-compiling crates.

At this time though the Cargo team feels that the benefit of changing
the defaults is well worth this change. Neither today nor directly after
this change will be a perfect world, but it's hoped that this change
makes things less bad!
2020-07-17 12:39:41 -07:00
est31
7f73a6c782 Move string interning to util
Code that handles string interning is rather an util functionality than
a core functionality.
2020-06-26 19:55:29 +02:00
Alex Crichton
6514c289d2 Improve git error messages a bit
This commit is targeted at further improving the error messages
generated from git errors. For authentication errors the actual URL
fetched is now printed out as well if it's different from the original
URL. This should help handle `insteadOf` logic where SSH urls are used
instead of HTTPS urls and users can know to track that down.

Otherwise the logic about recommending `net.git-fetch-with-cli` was
tweaked a bit and moved to the same location as the rest of our error
reporting.

Note that a change piggy-backed here as well is that `Caused by:` errors
are now automatically all tabbed over a bit instead of only having the
first line tabbed over. This required a good number of tests to be
updated, but it's just an updated in renderings.
2020-06-25 08:47:15 -07:00
Eric Huss
b6a4b074ed build-std: Don't treat std like a "local" package. 2020-04-28 13:05:55 -07:00
Alex Crichton
bac300bda0 Add support for -Cembed-bitcode=no
This commit is the Cargo half of support necessary for
rust-lang/rust#70458. Today the compiler emits embedded bytecode in
rlibs by default, but compresses it. This is both extraneous disk space
and wasted build time for almost all builds, so the PR in question there
is changing rustc to have a `-Cembed-bitcode` flag which, when enabled,
places the bitcode in the object file rather than an auxiliary file (no
extra compression), but also enables `-Cembed-bitcode=no` to disable
bitcode emission entirely.

This Cargo support changes Cargo to pass `-Cembed-bitcode=no` for almost
all compilations. Cargo will keep `lto = true` and such working by not
passing this flag (and thus allowing bitcode to get embedded), but by
default `cargo build` and `cargo build --release` will no longer have
any bitcode in rlibs which should result in speedier builds!

Most of the changes here were around the test suite and various
assertions about the `rustc` command lines we spit out. One test was
hard-disabled until we can get `-Cembed-bitcode=no` into nightly, and
then we can make it a nightly-only test. The test will then be stable
again once `-Cembed-bitcode=no` hits stable.

Note that this is intended to land before the upstream `-Cembed-bitcode`
change. The thinking is that we'll land everything in rust-lang/rust all
at once so there's no build time regressions for anyone. If we were to
land the `-Cembed-bitcode` PR first then there would be a build time
regression until we land Cargo changes because rustc would be emitting
uncompressed bitcode by default and Cargo wouldn't be turning it off.
2020-04-01 14:31:06 -07:00
bors
92d0ce8069 Auto merge of #8003 - ehuss:proc-macro-index, r=alexcrichton
Add proc-macro to index, and new feature resolver.

This adds the "pm" field to the index so that Cargo can detect which packages contain a proc-macro without downloading the package.

The second commit builds on that to support proc-macros in the new "de-unification" of the new feature resolver. This prevents dependencies shared between proc-macros and other dependency kinds from having features unified.

cc #7915
2020-03-18 14:00:07 +00:00
Eric Huss
51f3253238 Fix config profiles using "dev" in cargo test. 2020-03-17 10:29:27 -07:00
Eric Huss
0b115f57aa -Zfeatures=host_dep: Support decoupling proc-macro features. 2020-03-15 15:59:42 -07:00
Eric Huss
6154645186 Rewrite new feature resolver to simplify DepKind handling.
Now that dev-dependencies are a global setting, only build-dependencies
matter. This is maybe a little simpler to understand?
2020-02-20 12:04:28 -08:00
Eric Huss
8e935d4d20 Stabilize config-profile. 2020-01-23 12:05:02 -08:00
Eric Huss
dafacbb76b Update tests for formatting changes due to anyhow changes. 2020-01-13 13:36:20 -08:00
Eric Huss
77ee608de3 Add named config profiles. 2020-01-13 13:27:33 -08:00
Eric Huss
91015d52ce Add --config CLI option. 2019-12-19 09:44:02 -08:00
Eric Huss
2e5796d4e8 Store Definition in ConfigValue. 2019-12-19 09:44:02 -08:00
bors
b35bb1b9b7 Auto merge of #7591 - ehuss:stabilize-profile-overrides, r=alexcrichton
Stabilize profile-overrides.

This stabilizes the profile-overrides feature. This was proposed in [RFC 2282](https://github.com/rust-lang/rfcs/pull/2282) and implemented in #5384. Tracking issue is https://github.com/rust-lang/rust/issues/48683.

This is intended to land in 1.41 which will reach the stable channel on Jan 30th.

This includes a new documentation chapter on profiles. See the ["Overrides" section](9c993a92ce/src/doc/src/reference/profiles.md (overrides)) in `profiles.md` file for details on what is being stabilized.

Note: The `config-profile` and `named-profiles` features are still unstable.

Closes #6214

**Concerns**
- There is some risk that `build-override` may be confusing with the [proposed future dedicated `build` profile](https://github.com/rust-lang/cargo/pull/6577). There is some more discussion about the differences at https://github.com/rust-lang/rust/issues/48683#issuecomment-445571286. I don't expect it to be a significant drawback. If we proceed later with a dedicated `build` profile, I think we can handle explaining the differences in the documentation. (The linked PR is designed to work with profile-overrides.)
- I don't anticipate any unexpected interactions with `config-profiles` or `named-profiles`.
- Some of the syntax like `"*"` or `build-override` may not be immediately obvious what it means without reading the documentation. Nobody suggested any alternatives, though.
- Configuring overrides for multiple packages is currently a pain, as you have to repeat the settings separately for each package. I propose that we can extend the syntax in the future to allow a comma-separated list of package names to alleviate this concern if it is deemed worthwhile.
- The user may not know which packages to override, particularly for some dependencies that are split across multiple crates. I think, since this is an advanced feature, the user will likely be comfortable with using things like `cargo tree` to understand what needs to be overridden. There is [some discussion](https://github.com/rust-lang/rust/issues/48683#issuecomment-473356415) in the tracking issue about automatically including a package's dependencies, but this is somewhat complex.
- There is some possibly confusing interaction with the test/bench profile. Because dependencies are always built with the dev/release profiles, overridding test/bench usually does not have an effect (unless specifying a workspace member that is being tested/benched). Overriding test/bench was previously prohibited, but was relaxed when named profiles were added.
- We may want to allow overriding the `panic`, `lto`, and `rpath` settings in the future. I can imagine a case where someone has a large workspace, and wants to override those settings for just one package in the workspace. They are currently not allowed because it doesn't make sense to change those settings for rlibs, and `panic` usually needs to be in sync for the whole profile.
- There are some strange interactions with `dylib` crates detailed in https://github.com/rust-lang/rust/issues/64319. A fix was attempted, but later reverted. Since `dylib` crates are rare (this mostly applied to `libstd`), and a workaround was implemented for `build-std` (it no longer builds a dylib), I'm not too worried about this.
- The interaction with `share-generics` can be quite confusing (see https://github.com/rust-lang/rust/issues/63484). I added a section in the docs that tries to address this concern. It's also possible future versions of `rustc` may handle this better.
- The new documentation duplicates some of the information in the rustc book.  I think it's fine, as there are subtle differences, and it avoids needing to flip back and forth between the two books to learn what the settings do.
2019-12-02 15:54:59 +00:00
Eric Huss
83571aee56 Minor testsuite organization. 2019-11-24 18:42:45 -08:00
Eric Huss
dda81d3177 Stabilize profile-overrides. 2019-11-15 17:13:55 -08:00
Eric Huss
fcfe0b8988 Rename overrides to package in profiles. 2019-10-10 14:39:30 -07:00
Alex Crichton
61fb34b0c0 Allow a number of tests to run on stable
These tests all relied on support which has now ridden to stable at this
point, so let's let them run on stable!
2019-09-26 11:31:57 -07:00
Alex Crichton
9115b2c326 Extract support directory to its own crate
Extract out all our test support code to its own standalone crate so it
can be shared between multiple test suites if necessary.
2019-09-16 11:47:09 -07:00
Jethro Beekman
0e0d968825 Update #[test] attribute on all tests in the testsuite
sed -i 's/^#\[test\]/#[cargo_test]/' $(rg -l '^#\[test\]')

Manual fixes:
* proc_macro::proc_macro_doctest
2019-06-07 12:41:26 -07:00
Eric Huss
2b6fd6f0ff Incremental profile cleanup. 2019-02-20 09:12:27 -08:00
Alex Crichton
fecb724643 Format with cargo fmt 2018-12-08 03:19:47 -08:00
Dale Wijnand
04ddd4d0fc
Upgrade to Rust 2018 2018-12-06 20:18:35 +01:00