cargo/tests/testsuite/artifact_dep.rs

2906 lines
83 KiB
Rust
Raw Normal View History

add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
//! Tests specific to artifact dependencies, designated using
//! the new `dep = { artifact = "bin", … }` syntax in manifests.
use cargo_test_support::compare::match_exact;
use cargo_test_support::registry::Package;
use cargo_test_support::{
basic_bin_manifest, basic_manifest, cross_compile, project, publish, registry, rustc_host,
Project,
};
#[cargo_test]
fn check_with_invalid_artifact_dependency() {
// invalid name
let p = project()
.file(
"Cargo.toml",
r#"
[package]
name = "foo"
version = "0.0.0"
authors = []
resolver = "2"
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
[dependencies]
bar = { path = "bar/", artifact = "unknown" }
"#,
)
.file("src/lib.rs", "extern crate bar;") // this would fail but we don't get there, artifacts are no libs
.file("bar/Cargo.toml", &basic_manifest("bar", "0.0.1"))
.file("bar/src/lib.rs", "")
.build();
p.cargo("check -Z bindeps")
.masquerade_as_nightly_cargo(&["bindeps"])
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
.with_stderr(
"\
[ERROR] failed to parse manifest at `[..]/Cargo.toml`
Caused by:
'unknown' is not a valid artifact specifier
",
)
.with_status(101)
.run();
fn run_cargo_with_and_without_bindeps_feature(
p: &Project,
cmd: &str,
assert: &dyn Fn(&mut cargo_test_support::Execs),
) {
assert(
p.cargo(&format!("{} -Z bindeps", cmd))
.masquerade_as_nightly_cargo(&["bindeps"]),
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
);
assert(&mut p.cargo(cmd));
}
// lib specified without artifact
let p = project()
.file(
"Cargo.toml",
r#"
[package]
name = "foo"
version = "0.0.0"
authors = []
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
[dependencies]
bar = { path = "bar/", lib = true }
"#,
)
.file("src/lib.rs", "")
.file("bar/Cargo.toml", &basic_manifest("bar", "0.0.1"))
.file("bar/src/lib.rs", "")
.build();
run_cargo_with_and_without_bindeps_feature(&p, "check", &|cargo| {
cargo
.with_stderr(
"\
[ERROR] failed to parse manifest at `[..]/Cargo.toml`
Caused by:
'lib' specifier cannot be used without an 'artifact = ' value (bar)
",
)
.with_status(101)
.run();
});
// target specified without artifact
let p = project()
.file(
"Cargo.toml",
r#"
[package]
name = "foo"
version = "0.0.0"
authors = []
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
[dependencies]
bar = { path = "bar/", target = "target" }
"#,
)
.file("src/lib.rs", "")
.file("bar/Cargo.toml", &basic_manifest("bar", "0.0.1"))
.file("bar/src/lib.rs", "")
.build();
run_cargo_with_and_without_bindeps_feature(&p, "check", &|cargo| {
cargo
.with_stderr(
"\
[ERROR] failed to parse manifest at `[..]/Cargo.toml`
Caused by:
'target' specifier cannot be used without an 'artifact = ' value (bar)
",
)
.with_status(101)
.run();
})
}
#[cargo_test]
fn check_with_invalid_target_triple() {
// invalid name
let p = project()
.file(
"Cargo.toml",
r#"
[package]
name = "foo"
version = "0.0.0"
authors = []
resolver = "2"
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
[dependencies]
bar = { path = "bar/", artifact = "bin", target = "unknown-target-triple" }
"#,
)
.file("src/lib.rs", "")
.file("bar/Cargo.toml", &basic_manifest("bar", "0.0.1"))
.file("bar/src/main.rs", "fn main() {}")
.build();
p.cargo("check -Z bindeps")
.masquerade_as_nightly_cargo(&["bindeps"])
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
.with_stderr_contains(
r#"[..]Could not find specification for target "unknown-target-triple"[..]"#,
)
.with_status(101)
.run();
}
#[cargo_test]
fn build_without_nightly_aborts_with_error() {
let p = project()
.file(
"Cargo.toml",
r#"
[package]
name = "foo"
version = "0.0.0"
authors = []
resolver = "2"
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
[dependencies]
bar = { path = "bar/", artifact = "bin" }
"#,
)
.file("src/lib.rs", "extern crate bar;")
.file("bar/Cargo.toml", &basic_manifest("bar", "0.0.1"))
.file("bar/src/lib.rs", "")
.build();
p.cargo("check")
.with_status(101)
.with_stderr(
"\
[ERROR] failed to parse manifest at [..]
Caused by:
`artifact = ` requires `-Z bindeps` (bar)
",
)
.run();
}
#[cargo_test]
fn disallow_artifact_and_no_artifact_dep_to_same_package_within_the_same_dep_category() {
let p = project()
.file(
"Cargo.toml",
r#"
[package]
name = "foo"
version = "0.0.0"
authors = []
resolver = "2"
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
[dependencies]
bar = { path = "bar/", artifact = "bin" }
bar_stable = { path = "bar/", package = "bar" }
"#,
)
.file("src/lib.rs", "")
.file("bar/Cargo.toml", &basic_bin_manifest("bar"))
.file("bar/src/main.rs", "fn main() {}")
.build();
p.cargo("check -Z bindeps")
.masquerade_as_nightly_cargo(&["bindeps"])
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
.with_status(101)
.with_stderr("\
[WARNING] foo v0.0.0 ([CWD]) ignoring invalid dependency `bar_stable` which is missing a lib target
[ERROR] the crate `foo v0.0.0 ([CWD])` depends on crate `bar v0.5.0 ([CWD]/bar)` multiple times with different names",
)
.run();
}
#[cargo_test]
fn features_are_unified_among_lib_and_bin_dep_of_same_target() {
let p = project()
.file(
"Cargo.toml",
r#"
[package]
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
name = "foo"
version = "0.0.1"
authors = []
resolver = "2"
[dependencies.d1]
path = "d1"
features = ["d1f1"]
artifact = "bin"
lib = true
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
[dependencies.d2]
path = "d2"
features = ["d2f2"]
"#,
)
.file(
"src/main.rs",
r#"
fn main() {
d1::f1();
d1::f2();
d2::f1();
d2::f2();
}
"#,
)
.file(
"d1/Cargo.toml",
r#"
[package]
name = "d1"
version = "0.0.1"
authors = []
[features]
d1f1 = ["d2"]
[dependencies.d2]
path = "../d2"
features = ["d2f1"]
optional = true
"#,
)
.file(
"d1/src/main.rs",
r#"fn main() {
#[cfg(feature = "d1f1")]
d2::f1();
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
// Using f2 is only possible as features are unififed across the same target.
// Our own manifest would only enable f1, and f2 comes in because a parent crate
// enables the feature in its manifest.
#[cfg(feature = "d1f1")]
d2::f2();
}"#,
)
.file(
"d1/src/lib.rs",
r#"
#[cfg(feature = "d2")]
extern crate d2;
/// Importing f2 here shouldn't be possible as unless features are unified.
#[cfg(feature = "d1f1")]
pub use d2::{f1, f2};
"#,
)
.file(
"d2/Cargo.toml",
r#"
[package]
name = "d2"
version = "0.0.1"
authors = []
[features]
d2f1 = []
d2f2 = []
"#,
)
.file(
"d2/src/lib.rs",
r#"
#[cfg(feature = "d2f1")] pub fn f1() {}
#[cfg(feature = "d2f2")] pub fn f2() {}
"#,
)
.build();
p.cargo("build -Z bindeps")
.masquerade_as_nightly_cargo(&["bindeps"])
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
.with_stderr(
"\
[COMPILING] d2 v0.0.1 ([CWD]/d2)
[COMPILING] d1 v0.0.1 ([CWD]/d1)
[COMPILING] foo v0.0.1 ([CWD])
[FINISHED] dev [unoptimized + debuginfo] target(s) in [..]
",
)
.run();
}
#[cargo_test]
fn features_are_not_unified_among_lib_and_bin_dep_of_different_target() {
if cross_compile::disabled() {
return;
}
let target = cross_compile::alternate();
let p = project()
.file(
"Cargo.toml",
&r#"
[package]
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
name = "foo"
version = "0.0.1"
authors = []
resolver = "2"
[dependencies.d1]
path = "d1"
features = ["d1f1"]
artifact = "bin"
lib = true
target = "$TARGET"
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
[dependencies.d2]
path = "d2"
features = ["d2f2"]
"#
.replace("$TARGET", target),
)
.file(
"src/main.rs",
r#"
fn main() {
// the lib = true part always builds for our current target, unifying dependencies
d1::d2::f1();
d1::d2::f2();
d2::f1();
d2::f2();
}
"#,
)
.file(
"d1/Cargo.toml",
r#"
[package]
name = "d1"
version = "0.0.1"
authors = []
[features]
d1f1 = ["d2"]
[dependencies.d2]
path = "../d2"
features = ["d2f1"]
optional = true
"#,
)
.file("d1/src/main.rs", r#"fn main() {
// f1 we set ourselves
d2::f1();
// As 'main' is only compiled as part of the artifact dependency and since that is not unified
// if the target differs, trying to access f2 is a compile time error as the feature isn't enabled in our dependency tree.
d2::f2();
}"#)
.file(
"d1/src/lib.rs",
r#"
#[cfg(feature = "d2")]
pub extern crate d2;
"#,
)
.file(
"d2/Cargo.toml",
r#"
[package]
name = "d2"
version = "0.0.1"
authors = []
[features]
d2f1 = []
d2f2 = []
"#,
)
.file(
"d2/src/lib.rs",
r#"
#[cfg(feature = "d2f1")] pub fn f1() {}
#[cfg(feature = "d2f2")] pub fn f2() {}
"#,
)
.build();
p.cargo("build -Z bindeps")
.masquerade_as_nightly_cargo(&["bindeps"])
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
.with_status(101)
.with_stderr_contains(
"error[E0425]: cannot find function `f2` in crate `d2`\n --> d1/src/main.rs:6:17",
)
.run();
}
2022-02-28 02:28:32 +00:00
#[cargo_test]
fn feature_resolution_works_for_cfg_target_specification() {
if cross_compile::disabled() {
return;
}
let target = cross_compile::alternate();
let p = project()
.file(
"Cargo.toml",
&r#"
[package]
2022-02-28 02:28:32 +00:00
name = "foo"
version = "0.0.1"
authors = []
resolver = "2"
[dependencies.d1]
path = "d1"
artifact = "bin"
target = "$TARGET"
"#
.replace("$TARGET", target),
)
.file(
"src/main.rs",
r#"
fn main() {
let _b = include_bytes!(env!("CARGO_BIN_FILE_D1"));
}
"#,
)
.file(
"d1/Cargo.toml",
&r#"
[package]
name = "d1"
version = "0.0.1"
authors = []
[target.'$TARGET'.dependencies]
d2 = { path = "../d2" }
2022-02-28 02:28:32 +00:00
"#
.replace("$TARGET", target),
2022-02-28 02:28:32 +00:00
)
.file(
"d1/src/main.rs",
r#"fn main() {
d1::f();
}"#,
)
.file("d1/build.rs", r#"fn main() { }"#)
2022-02-28 02:28:32 +00:00
.file(
"d1/src/lib.rs",
&r#"pub fn f() {
#[cfg(target = "$TARGET")]
d2::f();
2022-02-28 02:28:32 +00:00
}
"#
.replace("$TARGET", target),
2022-02-28 02:28:32 +00:00
)
.file(
"d2/Cargo.toml",
r#"
[package]
name = "d2"
version = "0.0.1"
authors = []
"#,
)
.file("d2/build.rs", r#"fn main() { }"#)
2022-02-28 02:28:32 +00:00
.file("d2/src/lib.rs", "pub fn f() {}")
.build();
p.cargo("test -Z bindeps")
.masquerade_as_nightly_cargo(&["bindeps"])
2022-02-28 02:28:32 +00:00
.run();
}
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
#[cargo_test]
fn build_script_with_bin_artifacts() {
let p = project()
.file(
"Cargo.toml",
r#"
[package]
name = "foo"
version = "0.0.0"
authors = []
resolver = "2"
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
[build-dependencies]
bar = { path = "bar/", artifact = ["bin", "staticlib", "cdylib"] }
"#,
)
.file("src/lib.rs", "")
.file("build.rs", r#"
fn main() {
let baz: std::path::PathBuf = std::env::var("CARGO_BIN_FILE_BAR_baz").expect("CARGO_BIN_FILE_BAR_baz").into();
println!("{}", baz.display());
assert!(&baz.is_file());
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
let lib: std::path::PathBuf = std::env::var("CARGO_STATICLIB_FILE_BAR_bar").expect("CARGO_STATICLIB_FILE_BAR_bar").into();
println!("{}", lib.display());
assert!(&lib.is_file());
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
let lib: std::path::PathBuf = std::env::var("CARGO_CDYLIB_FILE_BAR_bar").expect("CARGO_CDYLIB_FILE_BAR_bar").into();
println!("{}", lib.display());
assert!(&lib.is_file());
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
let dir: std::path::PathBuf = std::env::var("CARGO_BIN_DIR_BAR").expect("CARGO_BIN_DIR_BAR").into();
println!("{}", dir.display());
assert!(dir.is_dir());
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
let bar: std::path::PathBuf = std::env::var("CARGO_BIN_FILE_BAR").expect("CARGO_BIN_FILE_BAR").into();
println!("{}", bar.display());
assert!(&bar.is_file());
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
let bar2: std::path::PathBuf = std::env::var("CARGO_BIN_FILE_BAR_bar").expect("CARGO_BIN_FILE_BAR_bar").into();
println!("{}", bar2.display());
assert_eq!(bar, bar2);
}
"#)
.file(
"bar/Cargo.toml",
r#"
[package]
name = "bar"
version = "0.5.0"
authors = []
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
[lib]
crate-type = ["staticlib", "cdylib"]
"#,
)
// compilation target is native for build scripts unless overridden
.file("bar/src/bin/bar.rs", &format!(r#"fn main() {{ assert_eq!(std::env::var("TARGET").unwrap(), "{}"); }}"#, cross_compile::native()))
.file("bar/src/bin/baz.rs", "fn main() {}")
.file("bar/src/lib.rs", "")
.build();
p.cargo("build -Z bindeps")
.masquerade_as_nightly_cargo(&["bindeps"])
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
.with_stderr_contains("[COMPILING] foo [..]")
.with_stderr_contains("[COMPILING] bar v0.5.0 ([CWD]/bar)")
.with_stderr_contains("[FINISHED] dev [unoptimized + debuginfo] target(s) in [..]")
.run();
let build_script_output = build_script_output_string(&p, "foo");
let msg = "we need the binary directory for this artifact along with all binary paths";
if cfg!(target_env = "msvc") {
match_exact(
"[..]/artifact/bar-[..]/bin/baz.exe\n\
[..]/artifact/bar-[..]/staticlib/bar-[..].lib\n\
[..]/artifact/bar-[..]/cdylib/bar.dll\n\
[..]/artifact/bar-[..]/bin\n\
[..]/artifact/bar-[..]/bin/bar.exe\n\
[..]/artifact/bar-[..]/bin/bar.exe",
&build_script_output,
msg,
"",
None,
)
.unwrap();
} else {
match_exact(
"[..]/artifact/bar-[..]/bin/baz-[..]\n\
[..]/artifact/bar-[..]/staticlib/libbar-[..].a\n\
[..]/artifact/bar-[..]/cdylib/[..]bar.[..]\n\
[..]/artifact/bar-[..]/bin\n\
[..]/artifact/bar-[..]/bin/bar-[..]\n\
[..]/artifact/bar-[..]/bin/bar-[..]",
&build_script_output,
msg,
"",
None,
)
.unwrap();
}
assert!(
!p.bin("bar").is_file(),
"artifacts are located in their own directory, exclusively, and won't be lifted up"
);
assert!(!p.bin("baz").is_file(),);
assert_artifact_executable_output(&p, "debug", "bar", "bar");
}
#[cargo_test]
fn build_script_with_bin_artifact_and_lib_false() {
let p = project()
.file(
"Cargo.toml",
r#"
[package]
name = "foo"
version = "0.0.0"
authors = []
resolver = "2"
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
[build-dependencies]
bar = { path = "bar/", artifact = "bin" }
"#,
)
.file("src/lib.rs", "")
.file(
"build.rs",
r#"
fn main() {
bar::doit()
}
"#,
)
.file("bar/Cargo.toml", &basic_bin_manifest("bar"))
.file("bar/src/main.rs", "fn main() { bar::doit(); }")
.file(
"bar/src/lib.rs",
r#"
pub fn doit() {
panic!("sentinel");
}
"#,
)
.build();
p.cargo("build -Z bindeps")
.masquerade_as_nightly_cargo(&["bindeps"])
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
.with_status(101)
.with_stderr_does_not_contain("[..]sentinel[..]")
.run();
}
#[cargo_test]
fn lib_with_bin_artifact_and_lib_false() {
let p = project()
.file(
"Cargo.toml",
r#"
[package]
name = "foo"
version = "0.0.0"
authors = []
resolver = "2"
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
[dependencies]
bar = { path = "bar/", artifact = "bin" }
"#,
)
.file(
"src/lib.rs",
r#"
pub fn foo() {
bar::doit()
}"#,
)
.file("bar/Cargo.toml", &basic_bin_manifest("bar"))
.file("bar/src/main.rs", "fn main() { bar::doit(); }")
.file(
"bar/src/lib.rs",
r#"
pub fn doit() {
panic!("sentinel");
}
"#,
)
.build();
p.cargo("build -Z bindeps")
.masquerade_as_nightly_cargo(&["bindeps"])
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
.with_status(101)
.with_stderr_does_not_contain("[..]sentinel[..]")
.run();
}
#[cargo_test]
fn build_script_with_selected_dashed_bin_artifact_and_lib_true() {
let p = project()
.file(
"Cargo.toml",
r#"
[package]
name = "foo"
version = "0.0.0"
authors = []
resolver = "2"
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
[build-dependencies]
bar-baz = { path = "bar/", artifact = "bin:baz-suffix", lib = true }
"#,
)
.file("src/lib.rs", "")
.file("build.rs", r#"
fn main() {
bar_baz::print_env()
}
"#)
.file(
"bar/Cargo.toml",
r#"
[package]
name = "bar-baz"
version = "0.5.0"
authors = []
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
[[bin]]
name = "bar"
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
[[bin]]
name = "baz-suffix"
"#,
)
.file("bar/src/main.rs", "fn main() {}")
.file("bar/src/lib.rs", r#"
pub fn print_env() {
let dir: std::path::PathBuf = std::env::var("CARGO_BIN_DIR_BAR_BAZ").expect("CARGO_BIN_DIR_BAR_BAZ").into();
let bin: std::path::PathBuf = std::env::var("CARGO_BIN_FILE_BAR_BAZ_baz-suffix").expect("CARGO_BIN_FILE_BAR_BAZ_baz-suffix").into();
println!("{}", dir.display());
println!("{}", bin.display());
assert!(dir.is_dir());
assert!(&bin.is_file());
assert!(std::env::var("CARGO_BIN_FILE_BAR_BAZ").is_err(), "CARGO_BIN_FILE_BAR_BAZ isn't set due to name mismatch");
assert!(std::env::var("CARGO_BIN_FILE_BAR_BAZ_bar").is_err(), "CARGO_BIN_FILE_BAR_BAZ_bar isn't set as binary isn't selected");
}
"#)
.build();
p.cargo("build -Z bindeps")
.masquerade_as_nightly_cargo(&["bindeps"])
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
.with_stderr(
"\
[COMPILING] bar-baz v0.5.0 ([CWD]/bar)
[COMPILING] foo [..]
[FINISHED] dev [unoptimized + debuginfo] target(s) in [..]",
)
.run();
let build_script_output = build_script_output_string(&p, "foo");
let msg = "we need the binary directory for this artifact and the binary itself";
if cfg!(target_env = "msvc") {
cargo_test_support::compare::match_exact(
&format!(
"[..]/artifact/bar-baz-[..]/bin\n\
[..]/artifact/bar-baz-[..]/bin/baz_suffix{}",
std::env::consts::EXE_SUFFIX,
),
&build_script_output,
msg,
"",
None,
)
.unwrap();
} else {
cargo_test_support::compare::match_exact(
"[..]/artifact/bar-baz-[..]/bin\n\
[..]/artifact/bar-baz-[..]/bin/baz_suffix-[..]",
&build_script_output,
msg,
"",
None,
)
.unwrap();
}
assert!(
!p.bin("bar").is_file(),
"artifacts are located in their own directory, exclusively, and won't be lifted up"
);
assert_artifact_executable_output(&p, "debug", "bar", "baz_suffix");
}
#[cargo_test]
fn lib_with_selected_dashed_bin_artifact_and_lib_true() {
let p = project()
.file(
"Cargo.toml",
r#"
[package]
name = "foo"
version = "0.0.0"
authors = []
resolver = "2"
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
[dependencies]
bar-baz = { path = "bar/", artifact = ["bin:baz-suffix", "staticlib", "cdylib"], lib = true }
"#,
)
.file(
"src/lib.rs",
r#"
pub fn foo() {
bar_baz::exists();
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
env!("CARGO_BIN_DIR_BAR_BAZ");
let _b = include_bytes!(env!("CARGO_BIN_FILE_BAR_BAZ_baz-suffix"));
let _b = include_bytes!(env!("CARGO_STATICLIB_FILE_BAR_BAZ"));
let _b = include_bytes!(env!("CARGO_STATICLIB_FILE_BAR_BAZ_bar-baz"));
let _b = include_bytes!(env!("CARGO_CDYLIB_FILE_BAR_BAZ"));
let _b = include_bytes!(env!("CARGO_CDYLIB_FILE_BAR_BAZ_bar-baz"));
}
"#,
)
.file(
"bar/Cargo.toml",
r#"
[package]
name = "bar-baz"
version = "0.5.0"
authors = []
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
[lib]
crate-type = ["rlib", "staticlib", "cdylib"]
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
[[bin]]
name = "bar"
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
[[bin]]
name = "baz-suffix"
"#,
)
.file("bar/src/main.rs", "fn main() {}")
.file("bar/src/lib.rs", "pub fn exists() {}")
.build();
p.cargo("build -Z bindeps")
.masquerade_as_nightly_cargo(&["bindeps"])
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
.with_stderr(
"\
[COMPILING] bar-baz v0.5.0 ([CWD]/bar)
[COMPILING] foo [..]
[FINISHED] dev [unoptimized + debuginfo] target(s) in [..]",
)
.run();
assert!(
!p.bin("bar").is_file(),
"artifacts are located in their own directory, exclusively, and won't be lifted up"
);
assert_artifact_executable_output(&p, "debug", "bar", "baz_suffix");
}
#[cargo_test]
fn allow_artifact_and_no_artifact_dep_to_same_package_within_different_dep_categories() {
let p = project()
.file(
"Cargo.toml",
r#"
[package]
name = "foo"
version = "0.0.0"
authors = []
resolver = "2"
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
[dependencies]
bar = { path = "bar/", artifact = "bin" }
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
[dev-dependencies]
bar = { path = "bar/", package = "bar" }
"#,
)
.file(
"src/lib.rs",
r#"
#[cfg(test)] extern crate bar;
pub fn foo() {
env!("CARGO_BIN_DIR_BAR");
let _b = include_bytes!(env!("CARGO_BIN_FILE_BAR"));
}"#,
)
.file("bar/Cargo.toml", &basic_bin_manifest("bar"))
.file("bar/src/main.rs", "fn main() {}")
.file("bar/src/lib.rs", "")
.build();
p.cargo("test -Z bindeps")
.masquerade_as_nightly_cargo(&["bindeps"])
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
.with_stderr_contains("[COMPILING] bar v0.5.0 ([CWD]/bar)")
.with_stderr_contains("[FINISHED] test [unoptimized + debuginfo] target(s) in [..]")
.run();
}
#[cargo_test]
fn normal_build_deps_are_picked_up_in_presence_of_an_artifact_build_dep_to_the_same_package() {
let p = project()
.file(
"Cargo.toml",
r#"
[package]
name = "foo"
version = "0.0.0"
authors = []
resolver = "2"
[dependencies]
bar = { path = "bar", artifact = "bin:bar" }
[build-dependencies]
bar = { path = "bar" }
"#,
)
.file("build.rs", "fn main() { bar::f(); }")
.file(
"src/lib.rs",
r#"
pub fn foo() {
env!("CARGO_BIN_DIR_BAR");
let _b = include_bytes!(env!("CARGO_BIN_FILE_BAR"));
}"#,
)
.file("bar/Cargo.toml", &basic_bin_manifest("bar"))
.file("bar/src/main.rs", "fn main() {}")
.file("bar/src/lib.rs", "pub fn f() {}")
.build();
p.cargo("check -Z bindeps")
.masquerade_as_nightly_cargo(&["bindeps"])
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
.run();
}
#[cargo_test]
fn disallow_using_example_binaries_as_artifacts() {
let p = project()
.file(
"Cargo.toml",
r#"
[package]
name = "foo"
version = "0.0.0"
authors = []
resolver = "2"
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
[dependencies]
bar = { path = "bar/", artifact = "bin:one-example" }
"#,
)
.file("src/lib.rs", "")
.file("bar/Cargo.toml", &basic_bin_manifest("bar"))
.file("bar/src/main.rs", "fn main() {}")
.file("bar/examples/one-example.rs", "fn main() {}")
.build();
p.cargo("build -Z bindeps")
.masquerade_as_nightly_cargo(&["bindeps"])
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
.with_status(101)
.with_stderr(r#"[ERROR] dependency `bar` in package `foo` requires a `bin:one-example` artifact to be present."#)
.run();
}
/// From RFC 3028
///
/// > You may also specify separate dependencies with different artifact values, as well as
/// dependencies on the same crate without artifact specified; for instance, you may have a
/// build dependency on the binary of a crate and a normal dependency on the Rust library of the same crate.
#[cargo_test]
fn allow_artifact_and_non_artifact_dependency_to_same_crate() {
let p = project()
.file(
"Cargo.toml",
r#"
[package]
name = "foo"
version = "0.0.0"
authors = []
resolver = "2"
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
[build-dependencies]
bar = { path = "bar/", artifact = "bin" }
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
[dependencies]
bar = { path = "bar/" }
"#,
)
.file("src/lib.rs", r#"
pub fn foo() {
bar::doit();
assert!(option_env!("CARGO_BIN_FILE_BAR").is_none());
}"#)
.file(
"build.rs",
r#"
fn main() {
assert!(option_env!("CARGO_BIN_FILE_BAR").is_none(), "no environment variables at build time");
std::process::Command::new(std::env::var("CARGO_BIN_FILE_BAR").expect("BAR present")).status().unwrap();
}"#,
)
.file("bar/Cargo.toml", &basic_bin_manifest("bar"))
.file("bar/src/main.rs", "fn main() {}")
.file("bar/src/lib.rs", "pub fn doit() {}")
.build();
p.cargo("check -Z bindeps")
.masquerade_as_nightly_cargo(&["bindeps"])
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
.with_stderr_contains("[COMPILING] bar [..]")
.with_stderr_contains("[COMPILING] foo [..]")
.run();
}
#[cargo_test]
fn build_script_deps_adopt_specified_target_unconditionally() {
if cross_compile::disabled() {
return;
}
let target = cross_compile::alternate();
let p = project()
.file(
"Cargo.toml",
&format!(
r#"
[package]
name = "foo"
version = "0.0.0"
authors = []
resolver = "2"
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
[build-dependencies.bar]
path = "bar/"
artifact = "bin"
target = "{}"
"#,
target
),
)
.file("src/lib.rs", "")
.file("build.rs", r#"
fn main() {
let bar: std::path::PathBuf = std::env::var("CARGO_BIN_FILE_BAR").expect("CARGO_BIN_FILE_BAR").into();
assert!(&bar.is_file());
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
}"#)
.file("bar/Cargo.toml", &basic_bin_manifest("bar"))
.file("bar/src/main.rs", "fn main() {}")
.file("bar/src/lib.rs", "pub fn doit() {}")
.build();
p.cargo("check -v -Z bindeps")
.masquerade_as_nightly_cargo(&["bindeps"])
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
.with_stderr_does_not_contain(format!(
"[RUNNING] `rustc --crate-name build_script_build build.rs [..]--target {} [..]",
target
))
.with_stderr_contains("[RUNNING] `rustc --crate-name build_script_build build.rs [..]")
.with_stderr_contains(format!(
"[RUNNING] `rustc --crate-name bar bar/src/lib.rs [..]--target {} [..]",
target
))
.with_stderr_contains(format!(
"[RUNNING] `rustc --crate-name bar bar/src/main.rs [..]--target {} [..]",
target
))
.with_stderr_does_not_contain(format!(
"[RUNNING] `rustc --crate-name foo [..]--target {} [..]",
target
))
.with_stderr_contains("[RUNNING] `rustc --crate-name foo [..]")
.run();
}
/// inverse RFC-3176
#[cargo_test]
fn build_script_deps_adopt_do_not_allow_multiple_targets_under_different_name_and_same_version() {
if cross_compile::disabled() {
return;
}
let alternate = cross_compile::alternate();
let native = cross_compile::native();
let p = project()
.file(
"Cargo.toml",
&format!(
r#"
[package]
name = "foo"
version = "0.0.0"
authors = []
resolver = "2"
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
[build-dependencies.bar]
path = "bar/"
artifact = "bin"
target = "{}"
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
[build-dependencies.bar-native]
package = "bar"
path = "bar/"
artifact = "bin"
target = "{}"
"#,
alternate,
native
),
)
.file("src/lib.rs", "")
.file("build.rs", r#"
fn main() {
let bar: std::path::PathBuf = std::env::var("CARGO_BIN_FILE_BAR").expect("CARGO_BIN_FILE_BAR").into();
assert!(&bar.is_file());
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
let bar_native: std::path::PathBuf = std::env::var("CARGO_BIN_FILE_BAR_NATIVE_bar").expect("CARGO_BIN_FILE_BAR_NATIVE_bar").into();
assert!(&bar_native.is_file());
assert_ne!(bar_native, bar, "should build different binaries due to different targets");
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
}"#)
.file("bar/Cargo.toml", &basic_bin_manifest("bar"))
.file("bar/src/main.rs", "fn main() {}")
.build();
p.cargo("check -v -Z bindeps")
.masquerade_as_nightly_cargo(&["bindeps"])
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
.with_status(101)
.with_stderr(format!(
"error: the crate `foo v0.0.0 ([CWD])` depends on crate `bar v0.5.0 ([CWD]/bar)` multiple times with different names",
))
.run();
}
#[cargo_test]
fn non_build_script_deps_adopt_specified_target_unconditionally() {
if cross_compile::disabled() {
return;
}
let target = cross_compile::alternate();
let p = project()
.file(
"Cargo.toml",
&format!(
r#"
[package]
name = "foo"
version = "0.0.0"
authors = []
resolver = "2"
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
[dependencies.bar]
path = "bar/"
artifact = "bin"
target = "{}"
"#,
target
),
)
.file(
"src/lib.rs",
r#"pub fn foo() { let _b = include_bytes!(env!("CARGO_BIN_FILE_BAR")); }"#,
)
.file("bar/Cargo.toml", &basic_bin_manifest("bar"))
.file("bar/src/main.rs", "fn main() {}")
.file("bar/src/lib.rs", "pub fn doit() {}")
.build();
p.cargo("check -v -Z bindeps")
.masquerade_as_nightly_cargo(&["bindeps"])
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
.with_stderr_contains(format!(
"[RUNNING] `rustc --crate-name bar bar/src/lib.rs [..]--target {} [..]",
target
))
.with_stderr_contains(format!(
"[RUNNING] `rustc --crate-name bar bar/src/main.rs [..]--target {} [..]",
target
))
.with_stderr_does_not_contain(format!(
"[RUNNING] `rustc --crate-name foo [..]--target {} [..]",
target
))
.with_stderr_contains("[RUNNING] `rustc --crate-name foo [..]")
.run();
}
#[cargo_test]
fn no_cross_doctests_works_with_artifacts() {
if cross_compile::disabled() {
return;
}
let p = project()
.file(
"Cargo.toml",
r#"
[package]
name = "foo"
version = "0.0.1"
authors = []
resolver = "2"
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
[dependencies]
bar = { path = "bar/", artifact = "bin", lib = true }
"#,
)
.file(
"src/lib.rs",
r#"
//! ```
//! env!("CARGO_BIN_DIR_BAR");
//! let _b = include_bytes!(env!("CARGO_BIN_FILE_BAR"));
//! ```
pub fn foo() {
env!("CARGO_BIN_DIR_BAR");
let _b = include_bytes!(env!("CARGO_BIN_FILE_BAR"));
}
"#,
)
.file("bar/Cargo.toml", &basic_bin_manifest("bar"))
.file("bar/src/lib.rs", r#"pub extern "C" fn c() {}"#)
.file("bar/src/main.rs", "fn main() {}")
.build();
let target = rustc_host();
p.cargo("test -Z bindeps --target")
.arg(&target)
.masquerade_as_nightly_cargo(&["bindeps"])
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
.with_stderr(&format!(
"\
[COMPILING] bar v0.5.0 ([CWD]/bar)
[COMPILING] foo v0.0.1 ([CWD])
[FINISHED] test [unoptimized + debuginfo] target(s) in [..]
[RUNNING] [..] (target/{triple}/debug/deps/foo-[..][EXE])
[DOCTEST] foo
",
triple = target
))
.run();
println!("c");
let target = cross_compile::alternate();
// This will build the library, but does not build or run doc tests.
// This should probably be a warning or error.
p.cargo("test -Z bindeps -v --doc --target")
.arg(&target)
.masquerade_as_nightly_cargo(&["bindeps"])
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
.with_stderr_contains(format!(
"[COMPILING] bar v0.5.0 ([CWD]/bar)
[RUNNING] `rustc --crate-name bar bar/src/lib.rs [..]--target {triple} [..]
[RUNNING] `rustc --crate-name bar bar/src/main.rs [..]--target {triple} [..]
[COMPILING] foo v0.0.1 ([CWD])
[RUNNING] `rustc --crate-name foo [..]
[FINISHED] test [unoptimized + debuginfo] target(s) in [..]",
triple = target
))
.run();
if !cross_compile::can_run_on_host() {
return;
}
// This tests the library, but does not run the doc tests.
p.cargo("test -Z bindeps -v --target")
.arg(&target)
.masquerade_as_nightly_cargo(&["bindeps"])
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
.with_stderr_contains(&format!(
"[FRESH] bar v0.5.0 ([CWD]/bar)
[COMPILING] foo v0.0.1 ([CWD])
[RUNNING] `rustc --crate-name foo [..]--test[..]
[FINISHED] test [unoptimized + debuginfo] target(s) in [..]
[RUNNING] `[CWD]/target/{triple}/debug/deps/foo-[..][EXE]`",
triple = target
))
.run();
}
#[cargo_test]
fn build_script_deps_adopts_target_platform_if_target_equals_target() {
if cross_compile::disabled() {
return;
}
let p = project()
.file(
"Cargo.toml",
r#"
[package]
name = "foo"
version = "0.0.0"
authors = []
resolver = "2"
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
[build-dependencies]
bar = { path = "bar/", artifact = "bin", target = "target" }
"#,
)
.file("src/lib.rs", "")
.file("build.rs", r#"
fn main() {
let bar: std::path::PathBuf = std::env::var("CARGO_BIN_FILE_BAR").expect("CARGO_BIN_FILE_BAR").into();
assert!(&bar.is_file());
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
}"#)
.file("bar/Cargo.toml", &basic_bin_manifest("bar"))
.file("bar/src/main.rs", "fn main() {}")
.file("bar/src/lib.rs", "pub fn doit() {}")
.build();
let alternate_target = cross_compile::alternate();
p.cargo("check -v -Z bindeps --target")
.arg(alternate_target)
.masquerade_as_nightly_cargo(&["bindeps"])
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
.with_stderr_does_not_contain(format!(
"[RUNNING] `rustc --crate-name build_script_build build.rs [..]--target {} [..]",
alternate_target
))
.with_stderr_contains("[RUNNING] `rustc --crate-name build_script_build build.rs [..]")
.with_stderr_contains(format!(
"[RUNNING] `rustc --crate-name bar bar/src/lib.rs [..]--target {} [..]",
alternate_target
))
.with_stderr_contains(format!(
"[RUNNING] `rustc --crate-name bar bar/src/main.rs [..]--target {} [..]",
alternate_target
))
.with_stderr_contains(format!(
"[RUNNING] `rustc --crate-name foo [..]--target {} [..]",
alternate_target
))
.run();
}
#[cargo_test]
2022-08-02 19:12:13 +00:00
// TODO(ST): rename bar (dependency) to something else and un-ignore this with RFC-3176
#[cfg_attr(target_env = "msvc", ignore = "msvc not working")]
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
fn profile_override_basic() {
let p = project()
.file(
"Cargo.toml",
r#"
[package]
name = "foo"
version = "0.0.1"
authors = []
[build-dependencies]
bar = { path = "bar", artifact = "bin" }
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
[dependencies]
bar = { path = "bar", artifact = "bin" }
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
[profile.dev.build-override]
opt-level = 1
[profile.dev]
opt-level = 3
"#,
)
.file("build.rs", "fn main() {}")
.file("src/lib.rs", "")
.file("bar/Cargo.toml", &basic_bin_manifest("bar"))
.file("bar/src/main.rs", "fn main() {}")
.file("bar/src/lib.rs", "pub fn bar() {}")
.build();
p.cargo("build -v -Z bindeps")
.masquerade_as_nightly_cargo(&["bindeps"])
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
.with_stderr_contains(
"[RUNNING] `rustc --crate-name build_script_build [..] -C opt-level=1 [..]`",
)
.with_stderr_contains(
"[RUNNING] `rustc --crate-name bar bar/src/main.rs [..] -C opt-level=3 [..]`",
)
.with_stderr_contains(
"[RUNNING] `rustc --crate-name bar bar/src/main.rs [..] -C opt-level=1 [..]`",
)
.with_stderr_contains(
"[RUNNING] `rustc --crate-name bar bar/src/lib.rs [..] -C opt-level=1 [..]`",
)
.with_stderr_contains(
"[RUNNING] `rustc --crate-name bar bar/src/lib.rs [..] -C opt-level=3 [..]`",
)
.with_stderr_contains("[RUNNING] `rustc --crate-name foo [..] -C opt-level=3 [..]`")
.run();
}
#[cargo_test]
fn dependencies_of_dependencies_work_in_artifacts() {
Package::new("baz", "1.0.0")
.file("src/lib.rs", "pub fn baz() {}")
.publish();
let p = project()
.file(
"Cargo.toml",
r#"
[package]
name = "foo"
version = "0.0.0"
authors = []
resolver = "2"
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
[build-dependencies]
bar = { path = "bar/", artifact = "bin" }
"#,
)
.file("src/lib.rs", "")
.file(
"build.rs",
r#"
fn main() {
std::process::Command::new(std::env::var("CARGO_BIN_FILE_BAR").expect("BAR present")).status().unwrap();
}
"#,
)
.file(
"bar/Cargo.toml",
r#"
[package]
name = "bar"
version = "0.5.0"
authors = []
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
[dependencies]
baz = "1.0.0"
"#,
)
.file("bar/src/lib.rs", r#"pub fn bar() {baz::baz()}"#)
.file("bar/src/main.rs", r#"fn main() {bar::bar()}"#)
.build();
p.cargo("build -Z bindeps")
.masquerade_as_nightly_cargo(&["bindeps"])
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
.run();
// cargo tree sees artifacts as the dependency kind they are in and doesn't do anything special with it.
p.cargo("tree -Z bindeps")
.masquerade_as_nightly_cargo(&["bindeps"])
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
.with_stdout(
"\
foo v0.0.0 ([CWD])
[build-dependencies]
bar v0.5.0 ([CWD]/bar)
baz v1.0.0
",
)
.run();
}
// TODO: Fix this potentially by reverting 887562bfeb8c540594d7d08e6e9a4ab7eb255865 which adds artifact information to the registry
// followed by 0ff93733626f7cbecaf9dce9ab62b4ced0be088e which picks it up.
// For reference, see comments by ehuss https://github.com/rust-lang/cargo/pull/9992#discussion_r801086315 and
// joshtriplett https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 .
#[cargo_test]
2022-08-02 19:12:13 +00:00
#[ignore = "broken, need artifact info in index"]
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
fn targets_are_picked_up_from_non_workspace_artifact_deps() {
if cross_compile::disabled() {
return;
}
let target = cross_compile::alternate();
Package::new("artifact", "1.0.0")
.file("src/main.rs", r#"fn main() {}"#)
.file("src/lib.rs", r#"pub fn lib() {}"#)
.publish();
let mut dep = registry::Dependency::new("artifact", "1.0.0");
Package::new("uses-artifact", "1.0.0")
.file(
"src/lib.rs",
r#"pub fn uses_artifact() { let _b = include_bytes!(env!("CARGO_BIN_FILE_ARTIFACT")); }"#,
)
.add_dep(dep.artifact("bin", Some(target.to_string())))
.publish();
let p = project()
.file(
"Cargo.toml",
r#"
[package]
name = "foo"
version = "0.0.0"
authors = []
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
[dependencies]
uses-artifact = { version = "1.0.0" }
"#,
)
.file(
"src/lib.rs",
r#"pub fn foo() { uses_artifact::uses_artifact(); }"#,
)
.build();
p.cargo("build -Z bindeps")
.masquerade_as_nightly_cargo(&["bindeps"])
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
.run();
}
#[cargo_test]
fn allow_dep_renames_with_multiple_versions() {
Package::new("bar", "1.0.0")
.file("src/main.rs", r#"fn main() {println!("1.0.0")}"#)
.publish();
let p = project()
.file(
"Cargo.toml",
r#"
[package]
name = "foo"
version = "0.0.0"
authors = []
resolver = "2"
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
[build-dependencies]
bar = { path = "bar/", artifact = "bin" }
bar_stable = { package = "bar", version = "1.0.0", artifact = "bin" }
"#,
)
.file("src/lib.rs", "")
.file(
"build.rs",
r#"
fn main() {
std::process::Command::new(std::env::var("CARGO_BIN_FILE_BAR").expect("BAR present")).status().unwrap();
std::process::Command::new(std::env::var("CARGO_BIN_FILE_BAR_STABLE_bar").expect("BAR STABLE present")).status().unwrap();
}
"#,
)
.file("bar/Cargo.toml", &basic_bin_manifest("bar"))
.file("bar/src/main.rs", r#"fn main() {println!("0.5.0")}"#)
.build();
p.cargo("check -Z bindeps")
.masquerade_as_nightly_cargo(&["bindeps"])
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
.with_stderr_contains("[COMPILING] bar [..]")
.with_stderr_contains("[COMPILING] foo [..]")
.run();
let build_script_output = build_script_output_string(&p, "foo");
match_exact(
"0.5.0\n1.0.0",
&build_script_output,
"build script output",
"",
None,
)
.unwrap();
}
#[cargo_test]
fn allow_artifact_and_non_artifact_dependency_to_same_crate_if_these_are_not_the_same_dep_kind() {
let p = project()
.file(
"Cargo.toml",
r#"
[package]
name = "foo"
version = "0.0.0"
authors = []
resolver = "2"
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
[build-dependencies]
bar = { path = "bar/", artifact = "bin", lib = false }
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
[dependencies]
bar = { path = "bar/" }
"#,
)
.file("src/lib.rs", r#"
pub fn foo() {
bar::doit();
assert!(option_env!("CARGO_BIN_FILE_BAR").is_none());
}"#)
.file(
"build.rs",
r#"fn main() {
println!("{}", std::env::var("CARGO_BIN_FILE_BAR").expect("CARGO_BIN_FILE_BAR"));
println!("{}", std::env::var("CARGO_BIN_FILE_BAR_bar").expect("CARGO_BIN_FILE_BAR_bar"));
}"#,
)
.file("bar/Cargo.toml", &basic_manifest("bar", "0.0.1"))
.file("bar/src/lib.rs", "pub fn doit() {}")
.file("bar/src/main.rs", "fn main() {}")
.build();
p.cargo("build -Z bindeps")
.masquerade_as_nightly_cargo(&["bindeps"])
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
.with_stderr(
"\
[COMPILING] bar [..]
[COMPILING] foo [..]
[FINISHED] dev [unoptimized + debuginfo] target(s) in [..]
",
)
.run();
}
#[cargo_test]
fn prevent_no_lib_warning_with_artifact_dependencies() {
let p = project()
.file(
"Cargo.toml",
r#"
[package]
name = "foo"
version = "0.0.0"
authors = []
resolver = "2"
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
[dependencies]
bar = { path = "bar/", artifact = "bin" }
"#,
)
.file(
"src/lib.rs",
r#"pub fn foo() { let _b = include_bytes!(env!("CARGO_BIN_FILE_BAR")); }"#,
)
.file("bar/Cargo.toml", &basic_bin_manifest("bar"))
.file("bar/src/main.rs", "fn main() {}")
.build();
p.cargo("check -Z bindeps")
.masquerade_as_nightly_cargo(&["bindeps"])
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
.with_stderr(
"\
[COMPILING] bar v0.5.0 ([CWD]/bar)\n\
[CHECKING] foo v0.0.0 ([CWD])\n\
[FINISHED] dev [unoptimized + debuginfo] target(s) in [..]",
)
.run();
}
#[cargo_test]
fn show_no_lib_warning_with_artifact_dependencies_that_have_no_lib_but_lib_true() {
let p = project()
.file(
"Cargo.toml",
r#"
[package]
name = "foo"
version = "0.0.0"
authors = []
resolver = "2"
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
[build-dependencies]
bar = { path = "bar/", artifact = "bin" }
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
[dependencies]
bar = { path = "bar/", artifact = "bin", lib = true }
"#,
)
.file("src/lib.rs", "")
.file("src/build.rs", "fn main() {}")
.file("bar/Cargo.toml", &basic_bin_manifest("bar"))
.file("bar/src/main.rs", "fn main() {}")
.build();
p.cargo("check -Z bindeps")
.masquerade_as_nightly_cargo(&["bindeps"])
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
.with_stderr_contains("[WARNING] foo v0.0.0 ([CWD]) ignoring invalid dependency `bar` which is missing a lib target")
.with_stderr_contains("[COMPILING] bar v0.5.0 ([CWD]/bar)")
.with_stderr_contains("[CHECKING] foo [..]")
.with_stderr_contains("[FINISHED] dev [unoptimized + debuginfo] target(s) in [..]")
.run();
}
#[cargo_test]
fn resolver_2_build_dep_without_lib() {
let p = project()
.file(
"Cargo.toml",
r#"
[package]
name = "foo"
version = "0.0.0"
authors = []
edition = "2021"
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
[build-dependencies]
bar = { path = "bar/", artifact = "bin" }
"#,
)
.file("src/lib.rs", "")
.file("build.rs", r#"
fn main() {
let bar: std::path::PathBuf = std::env::var("CARGO_BIN_FILE_BAR").expect("CARGO_BIN_FILE_BAR").into();
assert!(&bar.is_file());
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
}"#)
.file("bar/Cargo.toml", &basic_bin_manifest("bar"))
.file("bar/src/main.rs", "fn main() {}")
.build();
p.cargo("check -Z bindeps")
.masquerade_as_nightly_cargo(&["bindeps"])
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
.run();
}
#[cargo_test]
fn check_missing_crate_type_in_package_fails() {
for crate_type in &["cdylib", "staticlib", "bin"] {
let p = project()
.file(
"Cargo.toml",
&format!(
r#"
[package]
name = "foo"
version = "0.0.0"
authors = []
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
[dependencies]
bar = {{ path = "bar/", artifact = "{}" }}
"#,
crate_type
),
)
.file("src/lib.rs", "")
.file("bar/Cargo.toml", &basic_manifest("bar", "0.0.1")) //no bin, just rlib
.file("bar/src/lib.rs", "")
.build();
p.cargo("check -Z bindeps")
.masquerade_as_nightly_cargo(&["bindeps"])
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
.with_status(101)
.with_stderr(
"[ERROR] dependency `bar` in package `foo` requires a `[..]` artifact to be present.",
)
.run();
}
}
#[cargo_test]
fn check_target_equals_target_in_non_build_dependency_errors() {
let p = project()
.file(
"Cargo.toml",
r#"
[package]
name = "foo"
version = "0.0.0"
authors = []
resolver = "2"
[dependencies]
bar = { path = "bar/", artifact = "bin", target = "target" }
"#,
)
.file("src/lib.rs", "")
.file("bar/Cargo.toml", &basic_manifest("bar", "0.0.1"))
.file("bar/src/main.rs", "fn main() {}")
.build();
p.cargo("check -Z bindeps")
.masquerade_as_nightly_cargo(&["bindeps"])
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
.with_status(101)
.with_stderr_contains(
" `target = \"target\"` in normal- or dev-dependencies has no effect (bar)",
)
.run();
}
#[cargo_test]
fn env_vars_and_build_products_for_various_build_targets() {
let p = project()
.file(
"Cargo.toml",
r#"
[package]
name = "foo"
version = "0.0.0"
authors = []
resolver = "2"
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
[lib]
doctest = true
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
[build-dependencies]
bar = { path = "bar/", artifact = ["cdylib", "staticlib"] }
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
[dependencies]
bar = { path = "bar/", artifact = "bin", lib = true }
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
[dev-dependencies]
bar = { path = "bar/", artifact = "bin:baz" }
"#,
)
.file("build.rs", r#"
fn main() {
let file: std::path::PathBuf = std::env::var("CARGO_CDYLIB_FILE_BAR").expect("CARGO_CDYLIB_FILE_BAR").into();
assert!(&file.is_file());
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
let file: std::path::PathBuf = std::env::var("CARGO_STATICLIB_FILE_BAR").expect("CARGO_STATICLIB_FILE_BAR").into();
assert!(&file.is_file());
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
assert!(std::env::var("CARGO_BIN_FILE_BAR").is_err());
assert!(std::env::var("CARGO_BIN_FILE_BAR_baz").is_err());
}
"#)
.file(
"src/lib.rs",
r#"
//! ```
//! bar::c();
//! env!("CARGO_BIN_DIR_BAR");
//! let _b = include_bytes!(env!("CARGO_BIN_FILE_BAR"));
//! let _b = include_bytes!(env!("CARGO_BIN_FILE_BAR_bar"));
//! let _b = include_bytes!(env!("CARGO_BIN_FILE_BAR_baz"));
//! assert!(option_env!("CARGO_STATICLIB_FILE_BAR").is_none());
//! assert!(option_env!("CARGO_CDYLIB_FILE_BAR").is_none());
//! ```
pub fn foo() {
bar::c();
env!("CARGO_BIN_DIR_BAR");
let _b = include_bytes!(env!("CARGO_BIN_FILE_BAR"));
let _b = include_bytes!(env!("CARGO_BIN_FILE_BAR_bar"));
let _b = include_bytes!(env!("CARGO_BIN_FILE_BAR_baz"));
assert!(option_env!("CARGO_STATICLIB_FILE_BAR").is_none());
assert!(option_env!("CARGO_CDYLIB_FILE_BAR").is_none());
}
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
#[cfg(test)]
#[test]
fn env_unit() {
env!("CARGO_BIN_DIR_BAR");
let _b = include_bytes!(env!("CARGO_BIN_FILE_BAR"));
let _b = include_bytes!(env!("CARGO_BIN_FILE_BAR_bar"));
let _b = include_bytes!(env!("CARGO_BIN_FILE_BAR_baz"));
assert!(option_env!("CARGO_STATICLIB_FILE_BAR").is_none());
assert!(option_env!("CARGO_CDYLIB_FILE_BAR").is_none());
}
"#,
)
.file(
"tests/main.rs",
r#"
#[test]
fn env_integration() {
env!("CARGO_BIN_DIR_BAR");
let _b = include_bytes!(env!("CARGO_BIN_FILE_BAR"));
let _b = include_bytes!(env!("CARGO_BIN_FILE_BAR_bar"));
let _b = include_bytes!(env!("CARGO_BIN_FILE_BAR_baz"));
}"#,
)
.file("build.rs", "fn main() {}")
.file(
"bar/Cargo.toml",
r#"
[package]
name = "bar"
version = "0.5.0"
authors = []
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
[lib]
crate-type = ["staticlib", "cdylib", "rlib"]
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
[[bin]]
name = "bar"
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
[[bin]]
name = "baz"
"#,
)
.file("bar/src/lib.rs", r#"pub extern "C" fn c() {}"#)
.file("bar/src/main.rs", "fn main() {}")
.build();
p.cargo("test -Z bindeps")
.masquerade_as_nightly_cargo(&["bindeps"])
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
.with_stderr(
"\
[COMPILING] bar [..]
[COMPILING] foo [..]
[FINISHED] test [unoptimized + debuginfo] target(s) in [..]
[RUNNING] unittests [..]
[RUNNING] tests/main.rs [..]
[DOCTEST] foo
",
)
.run();
}
#[cargo_test]
fn publish_artifact_dep() {
// HACK below allows us to use a local registry
let registry = registry::init();
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
Package::new("bar", "1.0.0").publish();
Package::new("baz", "1.0.0").publish();
let p = project()
.file(
"Cargo.toml",
r#"
[package]
name = "foo"
version = "0.1.0"
authors = []
license = "MIT"
description = "foo"
documentation = "foo"
homepage = "foo"
repository = "foo"
resolver = "2"
[dependencies]
bar = { version = "1.0", artifact = "bin", lib = true }
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
[build-dependencies]
baz = { version = "1.0", artifact = ["bin:a", "cdylib", "staticlib"], target = "target" }
"#,
)
.file("src/lib.rs", "")
.build();
// HACK: Inject `foo` directly into the index so `publish` won't block for it to be in
// the index.
//
// This is to ensure we can verify the Summary we post to the registry as doing so precludes
// the registry from processing the publish.
Package::new("foo", "0.1.0")
.file("src/lib.rs", "")
.publish();
p.cargo("publish -Z bindeps --no-verify")
.replace_crates_io(registry.index_url())
.masquerade_as_nightly_cargo(&["bindeps"])
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
.with_stderr(
"\
[UPDATING] [..]
[PACKAGING] foo v0.1.0 [..]
[PACKAGED] [..]
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
[UPLOADING] foo v0.1.0 [..]
fix(publish): Block until it is in index Originally, crates.io would block on publish requests until the publish was complete, giving `cargo publish` this behavior by extension. When crates.io switched to asynchronous publishing, this intermittently broke people's workflows when publishing multiple crates. I say interittent because it usually works until it doesn't and it is unclear why to the end user because it will be published by the time they check. In the end, callers tend to either put in timeouts (and pray), poll the server's API, or use `crates-index` crate to poll the index. This isn't sufficient because - For any new interested party, this is a pit of failure they'll fall into - crates-index has re-implemented index support incorrectly in the past, currently doesn't handle auth, doesn't support `git-cli`, etc. - None of these previous options work if we were to implement workspace-publish support (#1169) - The new sparse registry might increase the publish times, making the delay easier to hit manually - The new sparse registry goes through CDNs so checking the server's API might not be sufficient - Once the sparse registry is available, crates-index users will find out when the package is ready in git but it might not be ready through the sparse registry because of CDNs So now `cargo` will block until it sees the package in the index. - This is checking via the index instead of server APIs in case there are propagation delays. This has the side effect of being noisy because of all of the "Updating index" messages. - This is done unconditionally because cargo used to block and that didn't seem to be a problem, blocking by default is the less error prone case, and there doesn't seem to be enough justification for a "don't block" flag. The timeout was 5min but I dropped it to 1m. Unfortunately, I don't have data from `cargo-release` to know what a reasonable timeout is, so going ahead and dropping to 60s and assuming anything more is an outage. Fixes #9507
2022-07-21 16:48:29 +00:00
[UPDATING] [..]
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
",
)
.run();
publish::validate_upload_with_contents(
r#"
{
"authors": [],
"badges": {},
"categories": [],
"deps": [{
"default_features": true,
"features": [],
"kind": "normal",
"name": "bar",
"optional": false,
"target": null,
"version_req": "^1.0"
},
{
"default_features": true,
"features": [],
"kind": "build",
"name": "baz",
"optional": false,
"target": null,
"version_req": "^1.0"
}
],
"description": "foo",
"documentation": "foo",
"features": {},
"homepage": "foo",
"keywords": [],
"license": "MIT",
"license_file": null,
"links": null,
"name": "foo",
"readme": null,
"readme_file": null,
"repository": "foo",
"vers": "0.1.0"
}
"#,
"foo-0.1.0.crate",
&["Cargo.toml", "Cargo.toml.orig", "src/lib.rs"],
&[(
"Cargo.toml",
&format!(
r#"{}
[package]
name = "foo"
version = "0.1.0"
authors = []
description = "foo"
homepage = "foo"
documentation = "foo"
license = "MIT"
repository = "foo"
resolver = "2"
[dependencies.bar]
version = "1.0"
artifact = ["bin"]
lib = true
[build-dependencies.baz]
version = "1.0"
artifact = [
"bin:a",
"cdylib",
"staticlib",
]
target = "target""#,
cargo::core::package::MANIFEST_PREAMBLE
),
)],
);
}
#[cargo_test]
fn doc_lib_true() {
let p = project()
.file(
"Cargo.toml",
r#"
[package]
name = "foo"
version = "0.0.1"
authors = []
resolver = "2"
[dependencies.bar]
path = "bar"
artifact = "bin"
lib = true
"#,
)
.file("src/lib.rs", "extern crate bar; pub fn foo() {}")
.file("bar/Cargo.toml", &basic_manifest("bar", "0.0.1"))
.file("bar/src/lib.rs", "pub fn bar() {}")
.file("bar/src/main.rs", "fn main() {}")
.build();
p.cargo("doc -Z bindeps")
.masquerade_as_nightly_cargo(&["bindeps"])
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
.with_stderr(
"\
[COMPILING] bar v0.0.1 ([CWD]/bar)
[DOCUMENTING] bar v0.0.1 ([CWD]/bar)
[DOCUMENTING] foo v0.0.1 ([CWD])
[FINISHED] dev [unoptimized + debuginfo] target(s) in [..]
",
)
.run();
assert!(p.root().join("target/doc").is_dir());
assert!(p.root().join("target/doc/foo/index.html").is_file());
assert!(p.root().join("target/doc/bar/index.html").is_file());
// Verify that it emits rmeta for the bin and lib dependency.
assert_eq!(p.glob("target/debug/artifact/*.rlib").count(), 0);
assert_eq!(p.glob("target/debug/deps/libbar-*.rmeta").count(), 2);
p.cargo("doc -Z bindeps")
.masquerade_as_nightly_cargo(&["bindeps"])
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
.env("CARGO_LOG", "cargo::ops::cargo_rustc::fingerprint")
.with_stdout("")
.run();
assert!(p.root().join("target/doc").is_dir());
assert!(p.root().join("target/doc/foo/index.html").is_file());
assert!(p.root().join("target/doc/bar/index.html").is_file());
}
#[cargo_test]
fn rustdoc_works_on_libs_with_artifacts_and_lib_false() {
let p = project()
.file(
"Cargo.toml",
r#"
[package]
name = "foo"
version = "0.0.1"
authors = []
resolver = "2"
[dependencies.bar]
path = "bar"
artifact = ["bin", "staticlib", "cdylib"]
"#,
)
.file(
"src/lib.rs",
r#"
pub fn foo() {
env!("CARGO_BIN_DIR_BAR");
let _b = include_bytes!(env!("CARGO_BIN_FILE_BAR"));
let _b = include_bytes!(env!("CARGO_CDYLIB_FILE_BAR"));
let _b = include_bytes!(env!("CARGO_CDYLIB_FILE_BAR_bar"));
let _b = include_bytes!(env!("CARGO_STATICLIB_FILE_BAR"));
let _b = include_bytes!(env!("CARGO_STATICLIB_FILE_BAR_bar"));
}"#,
)
.file(
"bar/Cargo.toml",
r#"
[package]
name = "bar"
version = "0.5.0"
authors = []
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
[lib]
crate-type = ["staticlib", "cdylib"]
"#,
)
.file("bar/src/lib.rs", "pub fn bar() {}")
.file("bar/src/main.rs", "fn main() {}")
.build();
p.cargo("doc -Z bindeps")
.masquerade_as_nightly_cargo(&["bindeps"])
add support for artifact dependencies (#9096) Tracking issue: https://github.com/rust-lang/cargo/issues/9096 Original PR: https://github.com/rust-lang/cargo/pull/9992 Add 'bindeps' -Z flag for later use A test to validate artifact dependencies aren't currently parsed. Parse 'artifact' and 'lib' fields. Note that this isn't behind a feature toggle so 'unused' messages will disappear. Transfer artifact dependencies from toml- into manifest-dependencies There are a few premises governing the operation. - if unstable features are not set, warn when 'artifact' or 'lib' is encountered. - bail if 'lib' is encountered alone, but warn that this WOULD happen with nightly. - artifact parsing checks for all invariants, but some aren't tested. Assure serialization of 'artifact' and 'lib' fields produces suitable values during publishing This should be the only place were these fields matter and where a cargo manifest is actually produced. These are only for internal use, no user is typically going to see or edit them. Place all artifact dependency tests inta their own module This facilitates deduplication later and possibly redistribution into other modules if there is a better fit. Represent artifacts that are rust libraries as another ArtifactKind This is more consistent and probably simpler for later use. No need to reflect the TOML data structure. Add tests to assure only 'lib = true' artifact deps are documented RFC-3028 doesn't talk about documentation, but for lib=true it's clear what the desired behaviour should be. If an artifact isn't a library though, then for now, it's transparent, maybe. Many more tests, more documentation, mild `Artifact` refactor The latter seems to be a better fit for what being an artifact really means within cargo, as it literally turns being a library on or off, and thus only optionally becoming a normal library. refactor to prepare for artifact related checks Don't show a no-lib warning for artifact dependencies (with lib = false) Tests for more artifact dependency invariants These are merely a proof of concept to show that we are not in a position to actually figure out everything about artifacts right after resolution. However, the error message looks more like a fatal error and less like something that can happen with a more elaborate error message with causes. This might show that these kind of checks might be better done later right before trying to use the information for create compile units. Validate that artifact deps with lib=true still trigger no-lib warnings This triggers the same warning as before, for now without any customization to indicate it's an artifact dependency. Use warnings instead of errors ------------------------------ This avoids the kind of harsh end of compilation in favor of something that can be recovered from. Since warnings are annoying, users will probably avoid re-declaring artifact dependencies. Hook in artifact dependencies into build script runs Even though we would still have to see what happens if they have a lib as well. Is it built twice? Also ---- - fly-by refactor: fix typo; use ? in method returning option - Propagate artifact information into Units; put artifacts into place This means artifacts now have their own place in the 'artifact' directory and uplifts won't happen for them. - refactor and fix cippy suggestion - fix build after rebasing onto master Create directories when executing the job, and not when preparing it. also: Get CI to work on windows the easy way, for now. Set directories for artifact dependencies in build script runtimes Test remaining kinds of build-script runtime environment variables Also ---- - Fix windows tests, the quick way. - Try to fix windows assertions, and generalize them - Fix second test for windows, hopefully test for available library dependency in build scripts with lib = true probably generally exclude all artifact dependencies with lib=false. Pass renamed dep names along with unit deps to allow proper artifact env names Test for selective bin:<name> syntax, as well as binaries with dashes Test to assure dependency names are transformed correctly assure advertised binaries and directories are actually present This wouldn't be the case if dependencies are not setup correctly, for instance. Also ---- - make it easier to see actual values even on failure This should help figure out why on CI something fails that works locally no matter what. Turns out this is a race condition, with my machine being on the good side of it so it doesn't show in testing. Fortunately it still can be reproduced and easily tested for. - refactor test; the race condition is still present though - Force CI to pass here by avoiding checks triggering race. - Fix windows build, maybe? More tolerant is_file() checks to account for delay on CI This _should_ help CI to test for the presence which is better than not testing at all. This appears to be needed as the output file isn't ready/present in time for some reason. The root cause of this issue is unknown, but it's definitely a race as it rarely happens locally. When it happened, the file was always present after the run. Now we will learn if it is truly not present, ever, or if it's maybe something very else. Validate libs also don't see artifact dependencies as libraries with lib=false Also ---- - Add prelimiary test for validating build-time artifacts - Try to fix CI on gnu windows Which apparently generates paths similar to linux, but with .exe suffix. The current linux patterns should match that. - refactor Help sharing code across modules allow rustc to use artifact dep environment variables, but… …it needs some adjustments to actually setup the unit dependency graph with artifacts as well. Right now it will only setup dependencies for artifacts that are libs, but not the artifacts themselves, completely ignoring them when they are not libs. Make artifact dependencies available in main loop This is the commit message #2: ------------------------------ rough cut of support for artifact dependencies at build time… …which unfortunately already shows that the binary it is supposed to include is reproducibly not ready in time even though the path is correct and it's present right after the run. Could it be related to rmeta? This is the commit message #3: ------------------------------ Fix test expectations as failure is typical than the warning we had before… …and add some tolerance to existing test to avoid occasional failures. This doesn't change the issue that it also doens't work at all for libraries, which is nicely reproducable and hopefully helps to fix this issue. This is the commit message #4: ------------------------------ Probably the fix for the dependency issue in the scheduler This means that bin() targets are now properly added to the job graph to cause proper syncing, whereas previously apparently it would still schedule binaries, but somehow consider them rmeta and thus start their dependents too early, leading to races. This is the commit message #5: ------------------------------ Don't accidentally include non-gnu windows tests in gnu windows. Support cargo doc and cargo check The major changes here are… - always compile artifacts in build mode, as we literally want the build output, always, which the dependent might rely on being present. - share code between the rather similar looking paths for rustdoc and rustc. Make artifact messages appear more in line with cargo by using backticks Also: Add first test for static lib support in build scripts build-scripts with support for cdylib and staticlib - Fix windows msvc build No need to speculate why the staticlib has hashes in the name even though nothing else. staticlib and cdylib support for libraries test staticlib and cdylibs for rustdoc as well. Also catch a seemingly untested special case/warning about the lack of linkable items, which probably shouldn't be an issue for artifacts as they are not linkable in the traditional sense. more useful test for 'cargo check' `cargo check` isn't used very consistently in tests, so when we use it we should be sure to actually try to use an artifact based feature to gain some coverage. verify that multiple versions are allowed for artifact deps as well. also: remove redundant test This is the commit message #2: ------------------------------ Properly choose which dependencies take part in artifact handling Previously it would include them very generously without considering the compatible dependency types. This is the commit message #3: ------------------------------ a more complex test which includes dev-dependencies It also shows that doc-tests don't yet work as rustdoc is run outside of the system into which we integrate right now. It should be possible to write our environment variable configuration in terms of this 'finished compilation' though, hopefully with most code reused. This is the commit message #4: ------------------------------ A first stab at storing artifact environment variables for packages… …however, it seems like the key for this isn't necessarily correct under all circumstances. Maybe it should be something more specific, don't know. This is the commit message #5: ------------------------------ Adjust key for identifying units to Metadata This one is actually unique and feels much better. This is the commit message #6: ------------------------------ Attempt to make use of artifact environment information… …but fail as the metadata won't match as the doctest unit is, of course, its separate unit. Now I wonder if its possible to find the artifact units in question that have the metadata. Properly use metadata to use artifact environment variables in doctests This is the commit message #2: ------------------------------ Add test for resolver = "2" and build dependencies Interestingly the 'host-features' flag must be set (as is seemingly documented in the flags documentation as well), even though I am not quite sure if this is the 100% correct solution. Should it rather have an entry with this flag being false in its map? Probably not… but I am not quite certain. This is the commit message #3: ------------------------------ set most if not all tests to use resolver = "2" This allows to keep it working with the most recent version while allowing to quickly test with "1" as well (which thus far was working fine). All tests I could imagine (excluding target and profiles) are working now Crossplatform tests now run on architecture aarm64 as well. More stringent negative testing Fix incorrect handling of dependency directory computation Previously it would just 'hack' the deps-dir to become something very different for artifacts. This could easily be fixed by putting the logic for artifact output directories into the right spot. A test for cargo-tree to indicate artifacts aren't handled specifically Assure build-scripts can't access artifacts at build time Actual doc-tests with access to artifact env vars All relevant parsing of `target = [..]` Next step is to actually take it into consideration. A failing test for adjusting the target for build script artifacts using --target Check for unknown artifact target triple in a place that exists for a year The first test showing that `target="target"` deps seemingly work For now only tested for build scripts, but it won't be much different for non-build dependencies. build scripts accept custom targets unconditionally Support target setting for non-build dependencies This is the commit message #2: ------------------------------ Add doc-test cross compile related test Even though there is no artifact code specific to doc testing, it's worth to try testing it with different target settings to validate it still works despite doc tests having some special caseing around target settings. This is the commit message #3: ------------------------------ A test to validate profiles work as expected for build-deps and non-build deps No change is required to make this work and artifact dependencies 'just work' based on the typical rules of their non-artifact counterarts. This is the commit message #4: ------------------------------ Adjust `cargo metadata` to deal with artifact dependencies This commit was squashed and there is probably more that changed. This is the commit message #5: ------------------------------ Show bin-only artifacts in "resolve" of metadata as well. This is the commit message #6: ------------------------------ minor refactoring during research for RFC-3176 This will soon need to return multiple extern-name/dep-name pairs. This is the commit message #7: ------------------------------ See if opt-level 3 works on win-msvc in basic profile test for artifacts This is the same value as is used in the other test of the same name, which certainly runs on windows. This is the commit message #8: ------------------------------ refactor Assure the type for targets reflect that they cannot be the host target, which removes a few unreachable!() expressions. Put `root_unit_compile_kind` into `UnitFor` Previously that wasn't done because of the unused `all_values()` method which has now been deleted as its not being used anyomre. This allows for the root unit compile kind to be passed as originally intended, instead of working around the previous lack of extendability of UnitFor due to ::all_values(). This is also the basis for better/correct feature handling once feature resolution can be depending on the artifact target as well, resulting in another extension to UnitFor for that matter. Also ---- - Fix ordering Previously the re-created target_mode was used due to the reordering in code, and who knows what kind of effects that might have (despite the test suite being OK with it). Let's put it back in place. - Deactivate test with filename collision on MSVC until RFC-3176 lands Avoid clashes with binaries called 'artifact' by putting 'artifact/' into './deps/' This commit addresses review comment https://github.com/rust-lang/cargo/pull/9992#discussion_r772939834 Don't rely on operator precedence for boolean operations Now it should be clear that no matter what the first term is, if the unit is an artifact, we should enqueue it. Replace boolean and `/*artifact*/ <bool>` with `IsArtifact::(Yes/No)` fix `doc::doc_lib_false()` test It broke due to major breakage in the way dependencies are calculated. Now we differentiate between deps computation for docs and for building. Avoid testing for doctest cross-compilation message It seems to be present on my machine, but isn't on linux and it's probably better to leave it out entirely and focus on the portions of consecutive output that we want to see at least. A test to validate features are unified across libraries and those in artifact deps in the same target Allow aarch64 MacOS to crosscompile to an easily executable alternative target That way more tests can run locally. Support for feature resolution per target The implementation is taken directly from RFC-3176 and notably lacks the 'multidep' part. Doing this definitely has the benefit of making entirely clear 'what is what' and helps to greatly reduce the scope of RFC-3176 when it's rebuilt based on the latest RF-3028, what we are implementing right now. Also ---- - A test which prooves that artifact deps with different target don't have a feature namespace yet - Add a test to validate features are namespaced by target Previously it didn't work because it relies on resolver = "2". - 'cargo metadata' test to see how artifact-deps are presented - Missed an opportunity for using the newly introduced `PackageFeaturesKey` - Use a HashMap to store name->value relations for artifact environment variables This is semantically closer to what's intended. also: Remove a by now misleading comment Prevent resolver crash if `target = "target"` is encountered in non-build dependencies A warning was emitted before, now we also apply a fix. Previously the test didn't fail as it accidentally used the old resolver, which now has been removed. Abort in parsing stage if nightly flag is not set and 'artifact' is used There is no good reason to delay errors to a later stage when code tries to use artifacts via environment variables which are not present. Change wording of warning message into what's expected for an error message remove unnecessary `Result` in `collect()` call Improve logic to warn if dependencie are ignored due to missing libraries The improvement here is to trigger correctly if any dependency of a crate is potentially a library, without having an actual library target as part of the package specification. Due to artifact dependencies it's also possible to have a dependency to the same crate of the same version, hence the package name isn't necessarily a unique name anymore. Now the name of the actual dependency in the toml file is used to alleviate this. Various small changes for readability and consistency A failing test to validate artifacts work in published crates as well Originally this should have been a test to see target acquisition works but this more pressing issue surfaced instead. Make artifacts known to the registry data (backwards compatible) Now artifacts are serialized into the registry on publish (at least if this code is actually used in the real crates-io registry) which allows the resolve stage to contain artifact information. This seems to be in line with the idea to provide cargo with all information it needs to do package resolution without downloading the actual manifest. Pick up all artifact targets into target info once resolve data is available Even though this works in the test at hand, it clearly shows there is a cyclic dependency between the resolve and the target data. In theory, one would have to repeat resolution until it settles while avoiding cycles. Maybe there is a better way. Add `bindeps`/artifact dependencies to `unstsable.md` with examples Fix tests Various small improvements Greatly simplify artifact environment propagation to commands Remove all adjustments to cargo-metadata, but leave tests The tests are to record the status quo with the current code when artifact dependencies are present and assure the information is not entirely non-sensical. Revert "Make artifacts known to the registry data (backwards compatible)" This reverts commit adc5f8ad04840af9fd06c964cfcdffb8c30769b0. Ideally we are able to make it work without altering the registry storage format. This could work if information from the package set is added to the resolve information. Enrich resolves information with additional information from downloaded manifests Resolve information comes from the registry, and it's only as rich as needed to know which packages take part in the build. Artifacts, however, don't influence dependency resolution, hence it shouldn't be part of it. For artifact information being present nonetheless when it matters, we port it back to the resolve graph where it will be needed later. Collect 'forced-target' information from non-workspace members as well This is needed as these targets aren't present in the registry and thus can't be picked up by traversing non-workspce members. The mechanism used to pick up artifact targets can also be used to pick up these targets. Remove unnecessary adjustment of doc test refactor `State::deps()` to have filter; re-enable accidentally disabled test The initial rebasing started out with a separted `deps_filtered()` method to retain the original capabilities while minimizing the chance for surprises. It turned out that the all changes combined in this PR make heavy use of filtering capabilities to the point where `deps(<without filter>)` was unused. This suggested that it's required to keep it as is without a way to inline portions of it. For the original change that triggered this rebase, see bd45ac81ba062a7daa3b0178dfcb6fd5759a943c The fix originally made was reapplied by allowing to re-use the required filter, but without inlining it. Always error on invalid artifact setup, with or without enabled bindeps feature Clarify how critical resolver code around artifact is working Remove workaround in favor of deferring a proper implementation See https://github.com/rust-lang/cargo/pull/9992#issuecomment-1033394197 for reference and the TODO in the ignored test for more information. truncate comments at 80-90c; cleanup - remove unused method - remove '-Z unstable-options' - improve error message - improve the way MSVC special cases are targetted in tests - improve how executables are found on non MSVC Avoid depending on output of rustc There is cyclic dependency between rustc and cargo which makes it impossible to adjust cargo's expectations on rustc without leaving broken commits in rustc and cargo. Add missing documentation fix incorrect removal of non-artifact libs This is also the first step towards cleaning up the filtering logic which is still making some logic harder to understand than needs be. The goal is to get it to be closer to what's currently on master. Another test was added to have more safety regarding the overall library inclusion logic. inline `build_artifact_requirements_to_units()` Simplify filtering This adds a default filter to `state.deps(…)` making it similar to what's currently in master, while creating another version of it to allow setting a custom filter. This is needed as the default filter won't allow build dependencies, which we need in this particular case. `calc_artifact_deps(…)` now hard-codes the default filter which is needed due to the use of `any` here: https://github.com/rust-lang/cargo/blob/c0e6abe384c2c6282bdd631e2f2a3b092043e6c6/src/cargo/core/compiler/unit_dependencies.rs#L1119 . Simplify filtering.
2021-10-21 09:57:23 +00:00
.with_stderr(
"\
[COMPILING] bar v0.5.0 ([CWD]/bar)
[DOCUMENTING] foo v0.0.1 ([CWD])
[FINISHED] dev [unoptimized + debuginfo] target(s) in [..]
",
)
.run();
assert!(p.root().join("target/doc").is_dir());
assert!(p.root().join("target/doc/foo/index.html").is_file());
assert!(
!p.root().join("target/doc/bar/index.html").is_file(),
"bar is not a lib dependency and thus remains undocumented"
);
}
fn assert_artifact_executable_output(
p: &Project,
target_name: &str,
dep_name: &str,
bin_name: &str,
) {
if cfg!(target_env = "msvc") {
assert_eq!(
p.glob(format!(
"target/{}/deps/artifact/{}-*/bin/{}{}",
target_name,
dep_name,
bin_name,
std::env::consts::EXE_SUFFIX
))
.count(),
1,
"artifacts are placed into their own output directory to not possibly clash"
);
} else {
assert_eq!(
p.glob(format!(
"target/{}/deps/artifact/{}-*/bin/{}-*{}",
target_name,
dep_name,
bin_name,
std::env::consts::EXE_SUFFIX
))
.filter_map(Result::ok)
.filter(|f| f.extension().map_or(true, |ext| ext != "o" && ext != "d"))
.count(),
1,
"artifacts are placed into their own output directory to not possibly clash"
);
}
}
fn build_script_output_string(p: &Project, package_name: &str) -> String {
let paths = p
.glob(format!("target/debug/build/{}-*/output", package_name))
.collect::<Result<Vec<_>, _>>()
.unwrap();
assert_eq!(paths.len(), 1);
std::fs::read_to_string(&paths[0]).unwrap()
}
#[cargo_test]
fn build_script_features_for_shared_dependency() {
// When a build script is built and run, its features should match. Here:
//
// foo
// -> artifact on d1 with target
// -> common with features f1
//
// d1
// -> common with features f2
//
// common has features f1 and f2, with a build script.
//
// When common is built as a dependency of d1, it should have features
// `f2` (for the library and the build script).
//
// When common is built as a dependency of foo, it should have features
// `f1` (for the library and the build script).
if cross_compile::disabled() {
return;
}
let target = cross_compile::alternate();
let p = project()
.file(
"Cargo.toml",
&r#"
[package]
name = "foo"
version = "0.0.1"
resolver = "2"
[dependencies]
d1 = { path = "d1", artifact = "bin", target = "$TARGET" }
common = { path = "common", features = ["f1"] }
"#
.replace("$TARGET", target),
)
.file(
"src/main.rs",
r#"
fn main() {
let _b = include_bytes!(env!("CARGO_BIN_FILE_D1"));
common::f1();
}
"#,
)
.file(
"d1/Cargo.toml",
r#"
[package]
name = "d1"
version = "0.0.1"
[dependencies]
common = { path = "../common", features = ["f2"] }
"#,
)
.file(
"d1/src/main.rs",
r#"fn main() {
common::f2();
}"#,
)
.file(
"common/Cargo.toml",
r#"
[package]
name = "common"
version = "0.0.1"
[features]
f1 = []
f2 = []
"#,
)
.file(
"common/src/lib.rs",
r#"
#[cfg(feature = "f1")]
pub fn f1() {}
#[cfg(feature = "f2")]
pub fn f2() {}
"#,
)
.file(
"common/build.rs",
&r#"
use std::env::var_os;
fn main() {
assert_eq!(var_os("CARGO_FEATURE_F1").is_some(), cfg!(feature="f1"));
assert_eq!(var_os("CARGO_FEATURE_F2").is_some(), cfg!(feature="f2"));
if std::env::var("TARGET").unwrap() == "$TARGET" {
assert!(var_os("CARGO_FEATURE_F1").is_none());
assert!(var_os("CARGO_FEATURE_F2").is_some());
} else {
assert!(var_os("CARGO_FEATURE_F1").is_some());
assert!(var_os("CARGO_FEATURE_F2").is_none());
}
}
"#
.replace("$TARGET", target),
)
.build();
p.cargo("build -Z bindeps -v")
.masquerade_as_nightly_cargo(&["bindeps"])
.run();
}
#[cargo_test]
fn calc_bin_artifact_fingerprint() {
// See rust-lang/cargo#10527
let p = project()
.file(
"Cargo.toml",
r#"
[package]
name = "foo"
version = "0.1.0"
resolver = "2"
[dependencies]
bar = { path = "bar/", artifact = "bin" }
"#,
)
.file(
"src/main.rs",
r#"
fn main() {
let _b = include_bytes!(env!("CARGO_BIN_FILE_BAR"));
}
"#,
)
.file("bar/Cargo.toml", &basic_bin_manifest("bar"))
.file("bar/src/main.rs", r#"fn main() { println!("foo") }"#)
.build();
p.cargo("check -Z bindeps")
.masquerade_as_nightly_cargo(&["bindeps"])
.with_stderr(
"\
[COMPILING] bar v0.5.0 ([CWD]/bar)
[CHECKING] foo v0.1.0 ([CWD])
[FINISHED] dev [unoptimized + debuginfo] target(s) in [..]
",
)
.run();
p.change_file("bar/src/main.rs", r#"fn main() { println!("bar") }"#);
// Change in artifact bin dep `bar` propagates to `foo`, triggering recompile.
p.cargo("check -v -Z bindeps")
.masquerade_as_nightly_cargo(&["bindeps"])
.with_stderr(
"\
[DIRTY] bar v0.5.0 ([CWD]/bar): the file `bar/src/main.rs` has changed ([..])
[COMPILING] bar v0.5.0 ([CWD]/bar)
[RUNNING] `rustc --crate-name bar [..]`
[DIRTY] foo v0.1.0 ([CWD]): the dependency bar was rebuilt
[CHECKING] foo v0.1.0 ([CWD])
[RUNNING] `rustc --crate-name foo [..]`
[FINISHED] dev [unoptimized + debuginfo] target(s) in [..]
",
)
.run();
// All units are fresh. No recompile.
p.cargo("check -v -Z bindeps")
.masquerade_as_nightly_cargo(&["bindeps"])
.with_stderr(
"\
[FRESH] bar v0.5.0 ([CWD]/bar)
[FRESH] foo v0.1.0 ([CWD])
[FINISHED] dev [unoptimized + debuginfo] target(s) in [..]
",
)
.run();
}
#[cargo_test]
fn with_target_and_optional() {
// See rust-lang/cargo#10526
if cross_compile::disabled() {
return;
}
let target = cross_compile::alternate();
let p = project()
.file(
"Cargo.toml",
&r#"
[package]
name = "foo"
version = "0.0.1"
edition = "2021"
[dependencies]
d1 = { path = "d1", artifact = "bin", optional = true, target = "$TARGET" }
"#
.replace("$TARGET", target),
)
.file(
"src/main.rs",
r#"
fn main() {
let _b = include_bytes!(env!("CARGO_BIN_FILE_D1"));
}
"#,
)
.file(
"d1/Cargo.toml",
r#"
[package]
name = "d1"
version = "0.0.1"
edition = "2021"
"#,
)
.file("d1/src/main.rs", "fn main() {}")
.build();
p.cargo("check -Z bindeps -F d1 -v")
.masquerade_as_nightly_cargo(&["bindeps"])
.with_stderr(
"\
[COMPILING] d1 v0.0.1 [..]
[RUNNING] `rustc --crate-name d1 [..]--crate-type bin[..]
[CHECKING] foo v0.0.1 [..]
[RUNNING] `rustc --crate-name foo [..]--cfg[..]d1[..]
[FINISHED] dev [..]
",
)
.run();
}
#[cargo_test]
fn with_assumed_host_target_and_optional_build_dep() {
let p = project()
.file(
"Cargo.toml",
r#"
[package]
name = "foo"
version = "0.0.1"
edition = "2021"
[build-dependencies]
d1 = { path = "d1", artifact = "bin", optional = true, target = "target" }
"#,
)
.file("src/main.rs", "fn main() {}")
.file(
"build.rs",
r#"
fn main() {
std::env::var("CARGO_BIN_FILE_D1").unwrap();
}
"#,
)
.file(
"d1/Cargo.toml",
r#"
[package]
name = "d1"
version = "0.0.1"
edition = "2021"
"#,
)
.file("d1/src/main.rs", "fn main() {}")
.build();
p.cargo("check -Z bindeps -F d1 -v")
.masquerade_as_nightly_cargo(&["bindeps"])
.with_stderr_unordered(
"\
[COMPILING] foo v0.0.1 ([CWD])
[COMPILING] d1 v0.0.1 ([CWD]/d1)
[RUNNING] `rustc --crate-name build_script_build [..]--crate-type bin[..]
[RUNNING] `rustc --crate-name d1 [..]--crate-type bin[..]
[RUNNING] `[CWD]/target/debug/build/foo-[..]/build-script-build`
[RUNNING] `rustc --crate-name foo [..]--cfg[..]d1[..]
[FINISHED] dev [..]
",
)
.run();
}
#[cargo_test]
fn decouple_same_target_transitive_dep_from_artifact_dep() {
// See https://github.com/rust-lang/cargo/issues/11463
let target = rustc_host();
let p = project()
.file(
"Cargo.toml",
&format!(
r#"
[package]
name = "foo"
version = "0.1.0"
edition = "2021"
[dependencies]
a = {{ path = "a" }}
bar = {{ path = "bar", artifact = "bin", target = "{target}" }}
"#
),
)
.file(
"src/main.rs",
r#"
fn main() {}
"#,
)
.file(
"bar/Cargo.toml",
r#"
[package]
name = "bar"
version = "0.1.0"
[dependencies]
a = { path = "../a", features = ["feature"] }
"#,
)
.file(
"bar/src/main.rs",
r#"
fn main() {}
"#,
)
.file(
"a/Cargo.toml",
r#"
[package]
name = "a"
version = "0.1.0"
edition = "2021"
[dependencies]
b = { path = "../b" }
c = { path = "../c" }
[features]
feature = ["c/feature"]
"#,
)
.file(
"a/src/lib.rs",
r#"
use b::Trait as _;
pub fn use_b_trait(x: &impl c::Trait) {
x.b();
}
"#,
)
.file(
"b/Cargo.toml",
r#"
[package]
name = "b"
version = "0.1.0"
[dependencies]
c = { path = "../c" }
"#,
)
.file(
"b/src/lib.rs",
r#"
pub trait Trait {
fn b(&self) {}
}
impl<T: c::Trait> Trait for T {}
"#,
)
.file(
"c/Cargo.toml",
r#"
[package]
name = "c"
version = "0.1.0"
[features]
feature = []
"#,
)
.file(
"c/src/lib.rs",
r#"
pub trait Trait {}
"#,
)
.build();
p.cargo("build -Z bindeps")
.masquerade_as_nightly_cargo(&["bindeps"])
.with_stderr(
"\
[COMPILING] c v0.1.0 ([CWD]/c)
[COMPILING] b v0.1.0 ([CWD]/b)
[COMPILING] a v0.1.0 ([CWD]/a)
[COMPILING] bar v0.1.0 ([CWD]/bar)
[COMPILING] foo v0.1.0 ([CWD])
[FINISHED] dev [unoptimized + debuginfo] target(s) in [..]
",
)
.run();
}
#[cargo_test]
fn decouple_same_target_transitive_dep_from_artifact_dep_lib() {
// See https://github.com/rust-lang/cargo/issues/10837
let target = rustc_host();
let p = project()
.file(
"Cargo.toml",
&format!(
r#"
[package]
name = "foo"
version = "0.1.0"
edition = "2021"
[dependencies]
a = {{ path = "a" }}
b = {{ path = "b", features = ["feature"] }}
bar = {{ path = "bar", artifact = "bin", lib = true, target = "{target}" }}
"#
),
)
.file("src/lib.rs", "")
.file(
"bar/Cargo.toml",
r#"
[package]
name = "bar"
version = "0.1.0"
edition = "2021"
[dependencies]
a = { path = "../a", features = ["b"] }
b = { path = "../b" }
"#,
)
.file("bar/src/lib.rs", "")
.file(
"bar/src/main.rs",
r#"
use b::Trait;
fn main() {
a::A.b()
}
"#,
)
.file(
"a/Cargo.toml",
r#"
[package]
name = "a"
version = "0.1.0"
[dependencies]
b = { path = "../b", optional = true }
"#,
)
.file(
"a/src/lib.rs",
r#"
pub struct A;
#[cfg(feature = "b")]
impl b::Trait for A {}
"#,
)
.file(
"b/Cargo.toml",
r#"
[package]
name = "b"
version = "0.1.0"
[features]
feature = []
"#,
)
.file(
"b/src/lib.rs",
r#"
pub trait Trait {
fn b(&self) {}
}
"#,
)
.build();
p.cargo("build -Z bindeps")
.masquerade_as_nightly_cargo(&["bindeps"])
.with_stderr(
"\
[COMPILING] b v0.1.0 ([CWD]/b)
[COMPILING] a v0.1.0 ([CWD]/a)
[COMPILING] bar v0.1.0 ([CWD]/bar)
[COMPILING] foo v0.1.0 ([CWD])
[FINISHED] dev [unoptimized + debuginfo] target(s) in [..]
",
)
.run();
}
2022-12-15 13:41:04 +00:00
#[cargo_test]
fn decouple_same_target_transitive_dep_from_artifact_dep_and_proc_macro() {
2022-12-15 13:41:04 +00:00
let target = rustc_host();
let p = project()
.file(
"Cargo.toml",
&format!(
r#"
[package]
name = "foo"
version = "0.1.0"
2022-12-15 13:41:04 +00:00
edition = "2021"
[dependencies]
c = {{ path = "c" }}
bar = {{ path = "bar", artifact = "bin", target = "{target}" }}
2022-12-15 13:41:04 +00:00
"#
),
)
.file("src/lib.rs", "")
2022-12-15 13:41:04 +00:00
.file(
"bar/Cargo.toml",
2022-12-15 13:41:04 +00:00
r#"
[package]
name = "bar"
version = "0.1.0"
2022-12-15 13:41:04 +00:00
[dependencies]
b = { path = "../b" }
2022-12-15 13:41:04 +00:00
"#,
)
.file("bar/src/main.rs", "fn main() {}")
2022-12-15 13:41:04 +00:00
.file(
"b/Cargo.toml",
2022-12-15 13:41:04 +00:00
r#"
[package]
name = "b"
version = "0.1.0"
2022-12-15 13:41:04 +00:00
edition = "2021"
[dependencies]
a = { path = "../a" }
2022-12-15 13:41:04 +00:00
[lib]
proc-macro = true
"#,
)
.file("b/src/lib.rs", "")
2022-12-15 13:41:04 +00:00
.file(
"c/Cargo.toml",
2022-12-15 13:41:04 +00:00
r#"
[package]
name = "c"
version = "0.1.0"
2022-12-15 13:41:04 +00:00
edition = "2021"
[dependencies]
d = { path = "../d", features = ["feature"] }
a = { path = "../a" }
2022-12-15 13:41:04 +00:00
[lib]
proc-macro = true
"#,
)
.file(
"c/src/lib.rs",
2022-12-15 13:41:04 +00:00
r#"
use a::Trait;
2022-12-15 13:41:04 +00:00
fn _c() {
d::D.a()
2022-12-15 13:41:04 +00:00
}
"#,
)
.file(
"a/Cargo.toml",
2022-12-15 13:41:04 +00:00
r#"
[package]
name = "a"
version = "0.1.0"
2022-12-15 13:41:04 +00:00
[dependencies]
d = { path = "../d" }
2022-12-15 13:41:04 +00:00
"#,
)
.file(
"a/src/lib.rs",
2022-12-15 13:41:04 +00:00
r#"
pub trait Trait {
fn a(&self) {}
2022-12-15 13:41:04 +00:00
}
impl Trait for d::D {}
2022-12-15 13:41:04 +00:00
"#,
)
.file(
"d/Cargo.toml",
2022-12-15 13:41:04 +00:00
r#"
[package]
name = "d"
version = "0.1.0"
2022-12-15 13:41:04 +00:00
[features]
feature = []
2022-12-15 13:41:04 +00:00
"#,
)
.file("d/src/lib.rs", "pub struct D;")
2022-12-15 13:41:04 +00:00
.build();
p.cargo("build -Z bindeps")
.masquerade_as_nightly_cargo(&["bindeps"])
.with_stderr_unordered(
"\
[COMPILING] d v0.1.0 ([CWD]/d)
[COMPILING] a v0.1.0 ([CWD]/a)
[COMPILING] b v0.1.0 ([CWD]/b)
[COMPILING] c v0.1.0 ([CWD]/c)
[COMPILING] bar v0.1.0 ([CWD]/bar)
[COMPILING] foo v0.1.0 ([CWD])
[FINISHED] dev [unoptimized + debuginfo] target(s) in [..]
2022-12-15 13:41:04 +00:00
",
)
.run();
}
#[cargo_test]
fn same_target_artifact_dep_sharing() {
let target = rustc_host();
let p = project()
.file(
"Cargo.toml",
&format!(
r#"
[package]
name = "foo"
version = "0.1.0"
[dependencies]
a = {{ path = "a" }}
bar = {{ path = "bar", artifact = "bin", target = "{target}" }}
"#
),
)
.file("src/lib.rs", "")
.file(
"bar/Cargo.toml",
r#"
[package]
name = "bar"
version = "0.1.0"
[dependencies]
a = { path = "../a" }
"#,
)
.file(
"bar/src/main.rs",
r#"
fn main() {}
"#,
)
.file(
"a/Cargo.toml",
r#"
[package]
name = "a"
version = "0.1.0"
"#,
)
.file("a/src/lib.rs", "")
.build();
p.cargo(&format!("build -Z bindeps --target {target}"))
.masquerade_as_nightly_cargo(&["bindeps"])
.with_stderr(
"\
[COMPILING] a v0.1.0 ([CWD]/a)
[COMPILING] bar v0.1.0 ([CWD]/bar)
[COMPILING] foo v0.1.0 ([CWD])
[FINISHED] dev [unoptimized + debuginfo] target(s) in [..]
",
)
.run();
}
#[cargo_test]
fn check_transitive_artifact_dependency_with_different_target() {
let p = project()
.file(
"Cargo.toml",
r#"
[package]
name = "foo"
version = "0.0.0"
[dependencies]
bar = { path = "bar/" }
"#,
)
.file("src/lib.rs", "")
.file(
"bar/Cargo.toml",
r#"
[package]
name = "bar"
version = "0.0.0"
[dependencies]
baz = { path = "baz/", artifact = "bin", target = "custom" }
"#,
)
.file("bar/src/lib.rs", "")
.file(
"bar/baz/Cargo.toml",
r#"
[package]
name = "baz"
version = "0.0.0"
[dependencies]
"#,
)
.file("bar/baz/src/main.rs", "fn main() {}")
.build();
p.cargo("check -Z bindeps")
.masquerade_as_nightly_cargo(&["bindeps"])
.with_stderr_contains(r#"thread 'main' panicked at 'no entry found for key'[..]"#)
.with_status(101)
.run();
}