There is a soundness issue with atty when building on Windows with a
custom allocator.
This PR switches direct dependencies on atty to is-terminal. New semver
compatible versions of clap and snapbox remove atty. #11417 upgrades
env_logger to remove it from there.
Fixes#11415
This removes one path to `atty`.
Others:
- clap: fixed in 4.0.27
- pretty-env-logger: seanmonstar/pretty-env-logger#52 needs to be resolved first
- snapbox: this will be fixed soonish but is also only a test dependency
- direct dependency
This is part of #11416
Expose libgit2-sys/vendored feature as vendored-libgit2
### What does this PR try to resolve?
Compiling cargo on MacOS can be problematic due present libraries, `cargo` already supports vendoring for openssl (and it helps https://github.com/pacak/cargo-show-asm/pull/51 to some users). This pull request attempts to extend vendoring support to libgit2 in order to fix more foreign library issues: https://github.com/pacak/cargo-show-asm/issues/50
While it is possible for me to enable it on my side (https://github.com/pacak/cargo-show-asm/pull/52) a special care needs to keep versions in sync. Exposing this feature in `cargo` directly will make such fixes for my and other crates more robust.
### How should we test and review this PR?
Disabled by default means existing behavior is unchanged, testing with it enabled to confirm if it still works should do the trick.
Improve errors for TOML fields that support workspace inheritance
Fixes#10997
This also addresses the issue with `MaybeWorkspace<VecStringOrBool>` mentioned in #10942:
```
Caused by:
invalid type: string "foo", expected a boolean or vector of strings for key `package.publish`
```
I removed the `maybe_workspace_vec_string` deserializer in 7a50c0c718 because the error message from the inner `Vec<String>` is now surfaced, but I can revert that if it's preferable to keep those changes.
I tried to base the `deserialize` implementation off of the derived impl for an untagged enum from `cargo expand`. This approach [ultimately led me](https://github.com/serde-rs/serde/blob/v1.0.144/serde/src/private/de.rs#L218) to adding the `serde-value` dependency.
Before, we had hacks to intercept raw arguments and to intercept clap
errors and assume what their intention was to be able to implement our
help system.
This flips it around and makes help like any other subcommand,
simplifying cargo initialization.
Bump git2 to 0.15 and libgit2-sys to 0.14
This will allow cargo to avoid vendored builds of git2 in up-to-date
environments going forward, and brings in the [libgit2 1.4.4 CVE fix].
[libgit2 1.4.4 CVE fix]: https://github.com/libgit2/libgit2/releases/tag/v1.4.4
This will allow cargo to avoid vendored builds of git2 in up-to-date
environments going forward, and brings in the [libgit2 1.4.4 CVE fix].
[libgit2 1.4.4 CVE fix]: https://github.com/libgit2/libgit2/releases/tag/v1.4.4
This reverts commit 8345cf5037. Since that
time, there are now multiple calls to get the number of CPUs, to handle
the `-j -NUM` case, so factor out a helper function.
Bump cargo-util version.
#10546 made a semver-incompatible change to the API of `ProcessBuilder::get_args`. Unfortunately we did not catch that until it was published. This bumps the version of cargo-util to 0.2.1 to accommodate that change. Stable will get version 0.2.0 so that the changes on beta can be released as 0.2.1 in their own time.
cc #10803
Retry command invocation with argfile
### What does this PR try to resolve?
Fixes#10381
The goal is to invoke `rustc` and `rustdoc` with ``@path`` arg file as a fallback.
Since the `-Zcheck-cfg-features`[^1] has already been implemented, in the foreseeable future cargo will probably hit the “**command line too big**” more often on Windows. `rustdoc` and `rustc` both support ``@path`` argfile [^2][^3], so we can leverage the feature for the fallback logic.
The idea behind this PR is that we put the **retry logic** in `ProcessBuilder::exec*` functions and reuse them as much as possible.
### How should we test and review this PR?
Review it commit by commit is recommended.
Argfile fallback is hard to tested with integration tests, so I added some unit tests for `cargo_util::ProcessBuilder` and `cargo::ops::fix::FixArgs`.
If you do want to test the support of argfile manually, a new environment variable `__CARGO_TEST_FORCE_ARGFILE` is added for the sake of forcing cargo to use argfile for rustc invocations. For example, `__CARGO_TEST_FORCE_ARGFILE cargo test --test testsuite -- fix::` is usually a good start to test this feature.
[^1]: https://doc.rust-lang.org/beta/cargo/reference/unstable.html?highlight=check#check-cfg-features
[^2]: https://doc.rust-lang.org/rustc/command-line-arguments.html#path-load-command-line-flags-from-a-path
[^3]: https://doc.rust-lang.org/rustdoc/command-line-arguments.html#path-load-command-line-flags-from-a-path
Part 3 of RFC2906 - Add support for inheriting `license-path`, and `depednency.path`
Tracking issue: #8415
RFC: rust-lang/rfcs#2906
[Part 1](https://github.com/rust-lang/cargo/pull/10497)
[Part 2](https://github.com/rust-lang/cargo/pull/10517)
This PR focuses on adding support for inheriting `license-path`, and `depednency.path`:
- To adjust the relative paths from being workspace-relative to package-relative, we use `pathdiff` which `cargo-add` is also going to be using for a similar purpose
- `ws_path` was added to `InheritableFields` so we can resolve relative paths from workspace-relative to package-relative
- Moved `resolve` for toml dependencies from `TomlDependency::<P>` to `TomlDependency`
- This was done since resolving a relative path should be a string
- You should never inherit from a `.cargo/config.toml` which is the reason `P` was added
Remaining implementation work for the RFC
- Relative paths for `readme`
- Path dependencies infer version directive
- Lock workspace dependencies and warn when unused
- Optimizations, as needed
- Evaluate any new fields for being inheritable (e.g. `rust-version`)
Update git2 dependencies
This pull-request update git2 to 0.14.1 and git2-curl to 0.15.0 and libgit2-sys to 0.13.1.
This fix a memory corruption that I have locally when running the testsuite:
```
==2338650== Uninitialised value was created by a stack allocation
==2338650== at 0x11FE3A0: git2::remote::Remote::fetch (remote.rs:276)
```
`std:🧵:available_parallelism` has been stabilized since 1.59.0.
Also, we don't want to block timing data output, so if parallelism
data is not available the table will display `ncpu=n/a` instead.
`toml_edit` fixed a bug in 0.13.4 that this test was relying on that is
meant to help with rust-lang/cargo#10349. This basically restores us
back to the pre-toml_edit behavior for published manifests.
Port cargo from toml-rs to toml_edit
Benefits:
- A TOML 1.0 compliant parser
- Unblock future work
- Have `cargo init` add the current crate to the workspace, rather
than error
- #5586: Upstream `cargo-add`
TODO
- [x] Analyze performance and address regressions
- [x] Identify and resolve incompatibiies
- [x] Resolve remaining test failures, see
https://github.com/ordian/toml_edit/labels/cargo
- [x] ~~Switch the code from https://github.com/rust-lang/cargo/pull/10176 to only parse once~~ (this PR is being merged first)
Benefits:
- A TOML 1.0 compliant parser
- Unblock future work
- Have `cargo init` add the current crate to the workspace, rather
than error
- #5586: Upstream `cargo-add`
Stabilize namespaced and weak dependency features.
This stabilizes the namespaced and weak dependency features. Support is now enabled on crates.io, so this should be ready to go.
As a part of this change, the new feature resolver is now enabled all of the time. This is fairly risky, since there are likely edge cases that haven't been exercised.
NOTE: Projects using `resolver="1"` *should* continue to have the same behavior, the old resolver behavior is emulated.
Closes#8813Closes#8832
- One parser change found by `cargo_config::includes` is that clap 2
would ignore any values after a `=` for flags.
`cargo config --show-origin` is a flag but the test passed `--show-origin=yes` which
happens to give the desired result for that test but is the same as
`--show-origin=no` or `--show-origin=alien-invasion`.
- The parser now panics when accessing an undefined attribute but clap
takes advantage of that for sharing code across commands that have
different subsets of arguments defined. I've extended clap so we can
"look before you leap" and put the checks at the argument calls to
start off with so its very clear what is tenuously shared. This
allows us to go in either direction in the future, either addressing
how we are sharing between commands or by moving this down into the
extension methods and pretending this clap feature doesn't exist
- On that topic, a test found clap-rs/clap#3263. For now, there is a
hack in clap. Depending on how we fix that in clap for clap 4.0, we
might need to re-address things in cargo.
- `value_of_os` now requires setting `allow_invalid_utf8`, otherwise it
asserts. To help catch this, I updated the argument definitions
associated with lookups reported by:
- `rg 'values?_os' src/`
- `rg 'values?_of_os' src/`
- clap now reports `2` for usage errors, so we had to bypass clap's
`exit` call to keep the same exit code.
BREAKING CHANGE: API now uses clap3
Update cargo-platform to 0.1.2
This preps cargo-platform for a release. The only substantial change is that this should include the license files into the archive that is uploaded to crates.io.
Closes#9758
This preps cargo-platform for a release. The only substantial change is that
this should include the license files into the archive that is uploaded to
crates.io.
Closes#9758
Update opener requirement from 0.4 to 0.5
Updates the requirements on [opener](https://github.com/Seeker14491/opener) to permit the latest version.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href="https://github.com/Seeker14491/opener/blob/master/CHANGELOG.md">opener's changelog</a>.</em></p>
<blockquote>
<h2>[0.5.0] - 2021-06-11</h2>
<h3>Added</h3>
<ul>
<li><code>open_browser()</code>, which uses the <code>$BROWSER</code> environment variable before falling back to <code>open()</code>.</li>
<li>WSL-specific implementation. Previously, WSL used the same implementation as Linux. Now the strategy on WSL is to use
the system's <code>wslview</code> command from <a href="https://github.com/wslutilities/wslu"><code>wslu</code></a> if available, falling back to the
system <code>xdg-open</code>, if available.</li>
</ul>
<h3>Changed</h3>
<ul>
<li>On Linux (non-WSL), the system <code>xdg-open</code> is now used if present. Otherwise, the bundled version is used, as before.</li>
<li>Avoid blocking the thread on Linux and WSL.</li>
</ul>
<h3>Removed</h3>
<ul>
<li><code>impl From<io::Error> for OpenError</code>.</li>
</ul>
<h2>[0.4.1] - 2019-09-30</h2>
<h3>Changed</h3>
<ul>
<li>Update <code>xdg-open</code>.</li>
</ul>
<h2>[0.4.0] - 2019-05-02</h2>
<h3>Added</h3>
<ul>
<li><code>OpenError</code> now implements <code>std::error::Error</code>.</li>
</ul>
<h3>Changed</h3>
<ul>
<li><code>OpenError</code>'s <code>failure::Fail</code> impl was removed from this crate, but the failure crate provides a blanket impl of
<code>failure::Fail</code> for types implementing <code>std::error::Error</code>, so this shouldn't break anything.</li>
</ul>
<h2>[0.3.0] - 2018-08-18</h2>
<h3>Added</h3>
<ul>
<li><code>stderr</code> field to <code>OpenError::ExitStatus</code> variant, which captures anything the failed process wrote to stderr.</li>
</ul>
<h2>[0.2.0] - 2018-08-08</h2>
<h3>Removed</h3>
<ul>
<li>The <code>open_sys</code> function, which was erroneously pub on non-Windows builds.</li>
</ul>
<h2>[0.1.0] - 2018-08-08</h2>
<ul>
<li>Initial release.</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="86e0a154d2"><code>86e0a15</code></a> Fix Drakefile.ts</li>
<li><a href="4261c2be99"><code>4261c2b</code></a> Release v0.5.0</li>
<li><a href="5d24271a65"><code>5d24271</code></a> Revert command name in error type to <code>&'static str</code></li>
<li><a href="9a36c6e656"><code>9a36c6e</code></a> Remove dependency on <code>wsl</code> crate</li>
<li><a href="165b933775"><code>165b933</code></a> Avoid blocking</li>
<li><a href="5c1e99a1c1"><code>5c1e99a</code></a> Update changelog</li>
<li><a href="2da976eb48"><code>2da976e</code></a> Refactor <code>linux_and_more</code> module</li>
<li><a href="f5c5564a9c"><code>f5c5564</code></a> Fix newline being included after WSL path conversion</li>
<li><a href="bd30011d61"><code>bd30011</code></a> [WSL] Try <code>wslview</code> first instead of second</li>
<li><a href="e16875d797"><code>e16875d</code></a> Misc formatting changes</li>
<li>Additional commits viewable in <a href="https://github.com/Seeker14491/opener/compare/v0.4.1...v0.5.0">compare view</a></li>
</ul>
</details>
<br />
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting ``@dependabot` rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- ``@dependabot` rebase` will rebase this PR
- ``@dependabot` recreate` will recreate this PR, overwriting any edits that have been made to it
- ``@dependabot` merge` will merge this PR after your CI passes on it
- ``@dependabot` squash and merge` will squash and merge this PR after your CI passes on it
- ``@dependabot` cancel merge` will cancel a previously requested merge and block automerging
- ``@dependabot` reopen` will reopen this PR if it is closed
- ``@dependabot` close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- ``@dependabot` ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- ``@dependabot` ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- ``@dependabot` ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
</details>
Update to semver 1.0.0
I am working on a 1.0.0 of the `semver` crate some time this week. It would be good to confirm Cargo will be able to use it, beforehand!
It's a from-scratch rewrite, but https://github.com/dtolnay/semver/issues/237 has code to compare against 0.10.0 (currently used by Cargo) how every possible version requirement currently published to crates.io matches against every possible crate version. The differences are all broken syntax like `^0-.11.0` previously parsing with ".11.0" as a pre-release string (which is invalid, because pre-release are not allowed to contain empty dot-separated identifiers) and `~2.0-2.2` previously parsing with "2.2" as a pre-release string, when the user almost certainly meant `>=2.0, <=2.2`. I'm not sure how much of those you want to add code into Cargo to preserve behavior, but I would be happy to do it.
cc rust-lang/rust#71249
This implements the Cargo side of 'Cargo report future-incompat'
Based on feedback from alexcrichton and est31, I'm implemented this a
flag `--future-compat-report` on `cargo check/build/rustc`, rather than
a separate `cargo describe-future-incompatibilities` command. This
allows us to avoid writing additional information to disk (beyond the
pre-existing recording of rustc command outputs).
This PR contains:
* Gating of all functionality behind `-Z report-future-incompat`.
Without this flag, all user output is unchanged.
* Passing `-Z emit-future-incompat-report` to rustc when
`-Z report-future-incompat` is enabled
* Parsing the rustc JSON future incompat report, and displaying it
it a user-readable format.
* Emitting a warning at the end of a build if any crates had
future-incompat reports
* A `--future-incompat-report` flag, which shows the full report for
each affected crate.
* Tests for all of the above.
At the moment, we can use the `array_into_iter` to write a test.
However, we might eventually get to a point where rustc is not currently
emitting future-incompat reports for any lints. What would we want the
cargo tests to do in this situation?
This functionality didn't require any significant internal changes to
Cargo, with one exception: we now process captured command output for
all units, not just ones where we want to display warnings. This may
result in a slightly longer time to run `cargo build/check/rustc` from
a full cache. since we do slightly more work for each upstream
dependency. Doing this seems unavoidable with the current architecture,
since we need to process captured command outputs to detect
any future-incompat-report messages that were emitted.
Update crossbeam-utils requirement from 0.7 to 0.8
Updates the requirements on [crossbeam-utils](https://github.com/crossbeam-rs/crossbeam) to permit the latest version.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href="https://github.com/crossbeam-rs/crossbeam/blob/master/CHANGELOG.md">crossbeam-utils's changelog</a>.</em></p>
<blockquote>
<h1>Version 0.8.0</h1>
<ul>
<li>Bump the minimum supported Rust version to 1.36.</li>
<li>Bump <code>crossbeam-channel</code> to <code>0.5</code>.</li>
<li>Bump <code>crossbeam-deque</code> to <code>0.8</code>.</li>
<li>Bump <code>crossbeam-epoch</code> to <code>0.9</code>.</li>
<li>Bump <code>crossbeam-queue</code> to <code>0.3</code>.</li>
<li>Bump <code>crossbeam-utils</code> to <code>0.8</code>.</li>
</ul>
<h1>Version 0.7.3</h1>
<ul>
<li>Fix breakage with nightly feature due to <a href="https://github-redirect.dependabot.com/rust-lang/rust/issues/65214">rust-lang/rust#65214</a>.</li>
<li>Bump <code>crossbeam-channel</code> to <code>0.4</code>.</li>
<li>Bump <code>crossbeam-epoch</code> to <code>0.8</code>.</li>
<li>Bump <code>crossbeam-queue</code> to <code>0.2</code>.</li>
<li>Bump <code>crossbeam-utils</code> to <code>0.7</code>.</li>
</ul>
<h1>Version 0.7.2</h1>
<ul>
<li>Bump <code>crossbeam-channel</code> to <code>0.3.9</code>.</li>
<li>Bump <code>crossbeam-epoch</code> to <code>0.7.2</code>.</li>
<li>Bump <code>crossbeam-utils</code> to <code>0.6.6</code>.</li>
</ul>
<h1>Version 0.7.1</h1>
<ul>
<li>Bump <code>crossbeam-utils</code> to <code>0.6.5</code>.</li>
</ul>
<h1>Version 0.7.0</h1>
<ul>
<li>Remove <code>ArcCell</code>, <code>MsQueue</code>, and <code>TreiberStack</code>.</li>
<li>Change the interface of <code>ShardedLock</code> to match <code>RwLock</code>.</li>
<li>Add <code>SegQueue::len()</code>.</li>
<li>Rename <code>SegQueue::try_pop()</code> to <code>SegQueue::pop()</code>.</li>
<li>Change the return type of <code>SegQueue::pop()</code> to <code>Result</code>.</li>
<li>Introduce <code>ArrayQueue</code>.</li>
<li>Update dependencies.</li>
</ul>
<h1>Version 0.6.0</h1>
<ul>
<li>Update dependencies.</li>
</ul>
<h1>Version 0.5.0</h1>
<ul>
<li>Update <code>crossbeam-channel</code> to 0.3.</li>
<li>Update <code>crossbeam-utils</code> to 0.6.</li>
<li>Add <code>AtomicCell</code>, <code>SharedLock</code>, and <code>WaitGroup</code>.</li>
</ul>
<h1>Version 0.4.1</h1>
<ul>
<li>Fix a double-free bug in <code>MsQueue</code> and <code>SegQueue</code>.</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="d9dfc9e1ff"><code>d9dfc9e</code></a> Merge <a href="https://github-redirect.dependabot.com/crossbeam-rs/crossbeam/issues/581">#581</a></li>
<li><a href="2a3d84b8db"><code>2a3d84b</code></a> Prepare for the next release</li>
<li><a href="5ea6e705ea"><code>5ea6e70</code></a> Merge <a href="https://github-redirect.dependabot.com/crossbeam-rs/crossbeam/issues/580">#580</a></li>
<li><a href="b363b3dcd7"><code>b363b3d</code></a> Fix UB in destroy_array test</li>
<li><a href="e08b21cc08"><code>e08b21c</code></a> Merge <a href="https://github-redirect.dependabot.com/crossbeam-rs/crossbeam/issues/577">#577</a></li>
<li><a href="6217abf4a9"><code>6217abf</code></a> Ignore clippy::match_like_matches_macro lint</li>
<li><a href="9cbfae7380"><code>9cbfae7</code></a> Update cfg-if to 1</li>
<li><a href="24447495c9"><code>2444749</code></a> Merge <a href="https://github-redirect.dependabot.com/crossbeam-rs/crossbeam/issues/574">#574</a></li>
<li><a href="619f7db2df"><code>619f7db</code></a> Fixed a few typos</li>
<li><a href="7cc8377263"><code>7cc8377</code></a> Better plot.py with Python 3 & remove hard code (<a href="https://github-redirect.dependabot.com/crossbeam-rs/crossbeam/issues/569">#569</a>)</li>
<li>Additional commits viewable in <a href="https://github.com/crossbeam-rs/crossbeam/compare/crossbeam-utils-0.7.0...crossbeam-utils-0.8.0">compare view</a></li>
</ul>
</details>
<br />
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting ``@dependabot` rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- ``@dependabot` rebase` will rebase this PR
- ``@dependabot` recreate` will recreate this PR, overwriting any edits that have been made to it
- ``@dependabot` merge` will merge this PR after your CI passes on it
- ``@dependabot` squash and merge` will squash and merge this PR after your CI passes on it
- ``@dependabot` cancel merge` will cancel a previously requested merge and block automerging
- ``@dependabot` reopen` will reopen this PR if it is closed
- ``@dependabot` close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- ``@dependabot` ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- ``@dependabot` ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- ``@dependabot` ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
- ``@dependabot` use these labels` will set the current labels as the default for future PRs for this repo and language
- ``@dependabot` use these reviewers` will set the current reviewers as the default for future PRs for this repo and language
- ``@dependabot` use these assignees` will set the current assignees as the default for future PRs for this repo and language
- ``@dependabot` use this milestone` will set the current milestone as the default for future PRs for this repo and language
- ``@dependabot` badge me` will comment on this PR with code to add a "Dependabot enabled" badge to your readme
Additionally, you can set the following in your Dependabot [dashboard](https://app.dependabot.com):
- Update frequency (including time of day and day of week)
- Pull request limits (per update run and/or open at any time)
- Automerge options (never/patch/minor, and dev/runtime dependencies)
- Out-of-range updates (receive only lockfile updates, if desired)
- Security updates (receive only security updates, if desired)
</details>
This stops using `to_string` as a proxy for this now-provided precise API.
This reverts commit b71927224f and bupms the
dependency version in Cargo.toml.
Updates to path source walking.
This is a collection of loosely related changes to path source walking:
* Add more context to error messages.
* Allow `package.exclude` patterns to match directories. Previously, the walker would recurse into the directory, and skip every file. Instead, just skip the whole directory. This can be helpful if the directory is not readable, or otherwise want to avoid walking.
* Don't require `Cargo.toml` to be in root of a git repo in order to use git to guide the selection. I'm not sure I understand the original reasoning that (any) `Cargo.toml` had to reside next to the `.git` directory.
The last is a moderately risky change, since it's hard to predict how this might affect more complex project layouts or new interactions with `.gitignore` that didn't exist before. Also, I'm wondering if it should just ignore if it fails to open the repo instead of emitting an error?
Closes#1729Closes#6188Closes#8092
Re-implement proc-macro feature decoupling.
This is essentially a rewrite of #8003. Instead of adding proc-macro to the index, it uses a strategy of downloading all packages before doing feature resolution. Then the package can be inspected for the proc-macro field.
This is a fairly major change. A brief overview:
- `PackageSet` now has a `download_accessible` method which tries to download a minimal set of every accessible package. This isn't very smart right now, and errs on downloading too much. In most cases it should be the same (or nearly the same) as before. It downloads extra in the following cases:
- The check for `[target]` dependencies checks both host and target for every dependency. I could tighten that up a little so build dependencies only check for the host, but it would add some complexity and I wanted to get feedback first.
- Optional dependencies disabled by the new feature resolver will get downloaded.
- Removed the loop in computing unit dependencies where downloading used to reside.
- When downloading starts, it should now show a more accurate count of how many crates are to be downloaded. Previously the count would fluctuate while the graph is being built.
We don't need the complexity of most channels since this is not a
performance sensitive part of Cargo, nor is it likely to be so any time
soon. Coupled with recent bugs (#7840) we believe in `std::sync::mpsc`,
let's just not use that and use a custom queue type locally which should
be amenable to a blocking push soon too.
The `anyhow` crate interoperates with the `std::error::Error` trait
rather than a custom `Fail` trait, and this is the general trend of
error handling in Rust as well.
Note that this is mostly mechanical (sed) and intended to get the test
suite passing. As usual there's still more idiomatic cleanup that can
happen, but that's left to later commits.
Pulls in alexcrichton/curl-rust#304 which fixes a bug from the last curl
update in #7308. This bug was not introduced by the Cargo PR itself but
rather by updating the `curl` submodule in the `curl-sys` crate. Without
this bugfix all downloads of a crate will make a new connection to
crates.io, which drastically increases download time since setting up a
connection takes so long.
Extract Platform to a separate crate.
This moves the `Platform`, `Cfg`, `CfgExpr` types to a new crate named "cargo-platform". The intent here is to give users of `cargo_metadata` a way of parsing and inspecting cargo's platform values.
Along the way, I rewrote the error handling to remove `failure`, and to slightly improve the output.
I'm having doubts whether or not this is a good idea. As you can see from the `examples/matches.rs` example, it is nontrivial to use this (which also misses cargo's config values and environment variables). I don't know if anyone will actually use this. If this doesn't seem to have value, I would suggest closing it.
I've also included a sample script, `publish.py`, for publishing cargo itself. I suspect it will need tweaking, but I figure it would be a start and open for feedback.
Don't require the `serde` feature of `url`
Ends up meaning that in full crate compiles that `url` doesn't wait for
`serde` to finish, which in turn enables crates like `git2` to start
sooner!
Experiment: Create timing report.
This is just an experiment, so I'm not sure if we'll want to merge it.
This adds an HTML report which gets saved to disk when the build is finished. It is primarily geared for identifying slow dependencies, and for visualizing how pipelining affects the build.
Here's an example: https://ehuss.github.io/cargo-timing.html
You can mouse over the blocks to highlight the reverse-dependencies that are released when a unit finishes. `syn` is a really good example.
It does a few other things, like displaying a message after each unit is finished. See the docs for more information.
This home's release remove support for the old `.multirust`
directory. Also it fixes rustup_home and cargo_home implementation
when corresponding environment variables are absolute paths.
Instead of treating Windows differently, this just always uses canonical paths
on all platforms. This fixes a problem where symlinks were not treated
correctly on all platforms.
Switching rm_rf to the remove_dir_all crate because deleting symbolic links on
Windows is difficult.
Tighten requirements for git2 crates
Bring in a few updates, used to update libgit2 and fix a Windows issue
as well as updating the `url` dependencies.
Closes#7173
This commit fixes a test in Cargo to work around a seeming regression in
behavior in libgit2 around HTTP authentication. The expected flow for
HTTP authentication with git is that git sends an HTTP request and
receives an "unauthorized" response. It then sends another request with
authorization information and that's what we're testing is received in
the our test.
Previously libgit2 would issue a new HTTP connection if the previous one
was closed, but it looks like changes in libgit2 now require that the
same HTTP connection is used for the initial request and the subsequent
request with authorization information. This broke our test since it's
not using an HTTP compliant server at all and is just some handwritten
TCP reads/writes. The fix here is to basically stay with handwritten TCP
reads/writes but tweak how it happens so it's all on the same HTTP/TCP
connection to match what libgit2 is expecting.
Some extra assertions have also been added to try to prevent deadlocks
from happening in the future and instead make the test fail fast if this
situation comes up again.
These tests take a good amount of time to run locally and they're also
causing a lot of dependencies to get pulled into rust-lang/rust, so
let's have a separate crate that we just test on our own CI
Test the Resolver against the varisat Library
Resolution can be reduced to the SAT problem. So this is an alternative implementation of the resolver that uses a SAT library for the hard work. This is intended to be easy to read, as compared to the real resolver, and run as part of the test sweet to make sure the real resolver works as expected. Part of #6120.
Some notes on performance:
The initial version did not support public & private deps:
~64 loc, `O(nln(n))` vars, `O(nln(n) + n*d)` clauses, 0.5x slower on `prop_passes_validation`
The final version:
~163 loc, `O(dn^2`) vars, `O(dn^3)` clauses, 1.5x slower on `prop_passes_validation`
That comparison makes me feel better about spending months trying to get public & private deps to be fast enough for stabilization.
Parse less JSON on null builds
This commit fixes a performance pathology in Cargo today. Whenever Cargo
generates a lock file (which happens on all invocations of `cargo build`
for example) Cargo will parse the crates.io index to learn about
dependencies. Currently, however, when it parses a crate it parses the
JSON blob for every single version of the crate. With a lock file,
however, or with incremental builds only one of these lines of JSON is
relevant. Measured today Cargo building Cargo parses 3700 JSON
dependencies in the registry.
This commit implements an optimization that brings down the number of
parsed JSON lines in the registry to precisely the right number
necessary to build a project. For example Cargo has 150 crates in its
lock file, so now it only parses 150 JSON lines (a 20x reduction from
3700). This in turn can greatly improve Cargo's null build time. Cargo
building Cargo dropped from 120ms to 60ms on a Linux machine and 400ms
to 200ms on a Mac.
The commit internally has a lot more details about how this is done but
the general idea is to have a cache which is optimized for Cargo to read
which is maintained automatically by Cargo.
Closes#6866
This commit fixes a performance pathology in Cargo today. Whenever Cargo
generates a lock file (which happens on all invocations of `cargo build`
for example) Cargo will parse the crates.io index to learn about
dependencies. Currently, however, when it parses a crate it parses the
JSON blob for every single version of the crate. With a lock file,
however, or with incremental builds only one of these lines of JSON is
relevant. Measured today Cargo building Cargo parses 3700 JSON
dependencies in the registry.
This commit implements an optimization that brings down the number of
parsed JSON lines in the registry to precisely the right number
necessary to build a project. For example Cargo has 150 crates in its
lock file, so now it only parses 150 JSON lines (a 20x reduction from
3700). This in turn can greatly improve Cargo's null build time. Cargo
building Cargo dropped from 120ms to 60ms on a Linux machine and 400ms
to 200ms on a Mac.
The commit internally has a lot more details about how this is done but
the general idea is to have a cache which is optimized for Cargo to read
which is maintained automatically by Cargo.
Closes#6866
This is intended to fix#6747 where multiple Cargos invoked with the
same jobserver would all have their own token but not actually run
concurrently due to file locking. Instead the fix is that whenever Cargo
blocks for a file lock with a configured global jobserver, a token is
released just before we block and then reacquired afterwards. This way
we should ensure that we're not hogging a cpu/token unnecessarily
without doing any work!
Closes#6747
It can be difficult to figure out what's wrong when a user reports that
`cargo fix` fails. There's often a large list of warnings, and it can
be hard to figure out which one caused a compile error.
This makes the deny(warnings) in the testsuite conditional on a new
"deny-warnings" feature, that is then enabled in CI.
Ideally I could use the (reasonably well established) CI env var (like
we do for proptests), but I don't know how to get the attribute to be
defined in terms of an env var.
We're already pulling in zlib for other dependencies like curl/libgit2
so there's not really much use in duplicating the compression code with
miniz, so let's instruct `flate2` to use libz as well to compress and
decompress chunks.
This commit switches Cargo to using HTTP/2 by default. This is
controlled via the `http.multiplexing` configuration variable and has
been messaged out for testing [1] (although got very few responses).
There's been surprisingly little fallout from parallel downloads, so
let's see how this goes!
[1]: https://internals.rust-lang.org/t/testing-cargos-parallel-downloads/8466
Fix timeouts firing while tarballs are extracted
This commit fixes#6125 by ensuring that while we're extracting tarballs
or doing other synchronous work like grabbing file locks we're not
letting the timeout timers of each HTTP transfer keep ticking. This is
curl's default behavior (which we don't want in this scenario). Instead
the timeout logic is inlined directly and we manually account for the
synchronous work happening not counting towards timeout limits.
Closes#6125
This commit fixes#6125 by ensuring that while we're extracting tarballs
or doing other synchronous work like grabbing file locks we're not
letting the timeout timers of each HTTP transfer keep ticking. This is
curl's default behavior (which we don't want in this scenario). Instead
the timeout logic is inlined directly and we manually account for the
synchronous work happening not counting towards timeout limits.
Closes#6125
use proptest to fuzz the resolver
This has been a long time goal. This uses proptest to generate random registry indexes and throws them at the resolver.
It would be simple to generate a registry by,
1. make a list of name and version number each picked at random
2. for each pick a list of dependencies by making a list of name and version requirements at random.
Unfortunately, it would be extremely unlikely to generate any interesting cases, as the chance that the random name you depend on was also generated as the name of a crate is vanishingly small. So this implementation works very hard to ensure that it only generates valid dependency requirements.
This is still a WIP as it has many problems:
- [x] The current strategy is very convoluted. It is hard to see that it is correct, and harder to see how it can be expanded. Thanks to @centril for working with me on IRC to get this far. Do you have advice for improving it?
- [X] It is slow as molasses when run without release. I looked with a profilere and we seem to spend 2/3 of the time in `to_url`. Maybe we can special case `example.com` for test, like we do for `crates.io` or something? Edit: Done. `lazy_static` did its magic.
- [x] `proptest` does not yet work with `minimal-versions`, a taste of my own medicine.
- [x] I have not verified that, if I remove the fixes for other test that this regenerates them.
The current strategy does not:
- [x] generate interesting version numbers, it just dose 1.0.0, 2.0.0 ...
- [x] guarantee that the version requirements are possible to meet by the crate named.
- [ ] generate features.
- [ ] generate dev-dependencies.
- [x] build deep dependency trees, it seems to prefer to generate crates with 0 or 1 dependents so that on average the tree is 1 or 2 layers deep.
And last but not least, there are no interesting properties being tested. Like:
- [ ] If resolution was successful, then all the transitive requirements are met.
- [x] If resolution was successful, then unpublishing a version of a crate that was not selected should not change that.
- [x] If resolution was unsuccessful, then it should stay unsuccessful even if any version of a crate is unpublished.
- [ ] @maurer suggested testing for consistency. Same registry, same cargo version, same lockfile, every time.
- [ ] @maurer suggested a pareto optimality property (if all else stays the same, but new package versions are released, we don't get a new lockfile where every version is <= the old one, and at least one is < the old one)
This commit tweaks how configuration is loaded for `cargo install`, ensuring
that we only load configuration from `$HOME` instead of the current working
directory. This should make installations a little more consistent in that they
probably shouldn't cover project-local configuration but should respect global
configuration!
Closes#6025
This is actually a super tricky problem. We don't really have the capacity for
more than one line of update-able information in Cargo right now, so we need to
squeeze a lot of information into one line of output for Cargo. The main
constraints this tries to satisfy are:
* At all times it should be clear what's happening. Cargo shouldn't just hang
with no output when downloading a crate for a long time, a counter ideally
needs to be decreasing while the download progresses.
* If a progress bar is shown, it shouldn't jump around. This ends up just being
a surprising user experience for most. Progress bars should only ever
increase, but they may increase at different speeds.
* Cargo has, currently, at most one line of output (as mentioned above) to pack
information into. We haven't delved into fancier terminal features that
involve multiple lines of update-able output.
* When downloading crates as part of `cargo build` (the norm) we don't actually
know ahead of time how many crates are being downloaded. We rely on the
calculation of unit dependencies to naturally feed into downloading more
crates.
* Furthermore, once we decide to download a crate, we don't actually know how
big it is! We have to wait for the server to tell us how big it is.
There doesn't really seem to be a great solution that satisfies all of these
constraints unfortunately. As a result this commit implements a relatively
conservative solution which should hopefully get us most of the way there. There
isn't actually a progress bar but rather Cargo prints that it's got N crates
left to download, and if it takes awhile it prints out that there are M bytes
remaining.
Unfortunately the progress is pretty choppy and jerky, not providing a smooth
UI. This appears to largely be because Cargo will synchronously extract
tarballs, which for large crates can cause a noticeable pause. Cargo's not
really prepared internally to perform this work on helper threads, but ideally
if it could do so it would improve the output quite a bit! (making it much
smoother and also able to account for the time tarball extraction takes).
This commit implements parallel downloads using `libcurl` powered by
`libnghttp2` over HTTP/2. Using all of the previous refactorings this actually
implements usage of `Multi` to download crates in parallel. This achieves some
large wins locally, taking download times from 30s to 2s in the best case.
The standard output of Cargo is also changed as a result of this commit. It's
no longer useful for Cargo to print "Downloading ..." for each crate really as
they all start instantaneously. Instead Cargo now no longer prints `Downloading`
by default (unless attached to a pipe) and instead only has one progress bar for
all downloads. Currently this progress bar is discrete and based on the total
number of downloads, no longer specifying how much of one particular download
has happened. This provides a less granular view into what Cargo is doing but
it's hoped that it looks reasonable from an outside perspective as there's
still a progress bar indicating what's happening.
BUG fuzzing found a bug in the resolver, we need a complete set of conflicts to do backjumping
As mentioned in https://github.com/rust-lang/cargo/pull/5921#issuecomment-418890269, the new proptest found a live bug! This PR so far tracs my attempt to minimize the problematic input.
The problem turned out to be that we where backjumping on incomplete set of conflicts.
run some tests with minimal-versions on CI
In #5757 we discovered that sum test don't pass with minimal-versions, and so only added CI for `cargo check`. This PR is to see if that is still needed, and if it is then which test rely on upstream bugfix.
Improve verbose console and log for finding git repo in package check
Third attempt to resolve#5823 by improving logging and tests. This exposes the issue to testing, via verbose console output and is dependent on alexcrichton/git2-rs#341 as just released in git2 0.7.5 crate. Thus tests *should* now pass on all platforms, incl. windows, but I also intend to bump the minimal git2 release dependency (in a subsequently added commit).
cc: @Eh2406 thanks for your fix and help!
Fully capture rustc and rustdoc output when -Zcompile-progress is passed
Fixes#5764 and #5695.
On Windows, we will parse the ANSI escape code into console commands via my `fwdansi` package, based on @ishitatsuyuki's idea in https://github.com/rust-lang/cargo/issues/5695#issuecomment-406300234. Outside of Windows the content is forwarded as-is.
This commit updates the `cargo fix` implementation to iteratively apply fixes
from the compiler instead of only once. Currently the compiler can sometimes
emit overlapping suggestions, such as in the case of transitioning
::foo::<::Bar>();
to ...
crate::foo::<crate::Bar>();
and `rustfix` rightfully can't handle overlapping suggestions as there's no
clear way of how to disambiguate the fixes. To fix this problem Cargo will now
run `rustc` and `rustfix` multiple times, attempting to reach a steady state
where no fixes failed to apply.
Naturally this is a pretty tricky thing to do and we want to be sure that Cargo
doesn't loop forever, for example. A number of safeguards are in place to
prevent Cargo from going off into the weeds when fixing files, notably avoiding
to reattempt fixes if no successful fixes ended up being applied.
Closes#5813Closesrust-lang/rust#52754
This commit imports the `cargo fix` subcommand in rust-lang-nursery/rustfix
directly into Cargo as a subcommand. This should allow us to ease our
distribution story of `cargo fix` as we prepare for the upcoming 2018 edition
release.
It's been attempted here to make the code as idiomatic as possible for Cargo's
own codebase. Additionally all tests from cargo-fix were imported into Cargo's
test suite as well. After this lands and is published in nightly the `cargo-fix`
command in rust-lang-nursery/rustfix will likely be removed.
cc rust-lang/rust#52272
