Previously, we've calculated the set of crate types to learn about by
recursively walking the graph of units. However, to actually know
dependencies of a unit exactly, we must know target specific info, and
we don't know it at this moment (in fact, we are trying calculating it).
Note that crate-type calculation is already lazy, we don't have to calc
all crate-types upfront. So, let's just scrape this info once for
well-known crate types, and fill whatever is left lazily.
One historical annoyance I've always had with Cargo that I've found surprising
is that in some situations when you bump version numbers you'll have to end up
running `cargo update` later on to get everything to build. You get pretty wonky
error messages in this case as well saying a package doesn't exist when it
clearly does at a particular location!
I've had difficulty historically nailing down a test case for this but it looks
like we ironically already had one in our test suite and I also jury-rigged up
one from a case I ran into in the wild today.
Faster resolver: clean code and the `backtrack_stack`
This is a small extension to #5168 and is inspired by https://github.com/rust-lang/cargo/pull/4834#issuecomment-363518370
After #5168 these work (and don't on cargo from nightly.):
- `safe_core = "=0.22.4"`
- `safe_vault = "=0.13.2"`
But these don't work (and do on cargo from this PR.)
- `crust = "=0.24.0"`
- `elastic = "=0.3.0"`
- `elastic = "=0.4.0"`
- `elastic = "=0.5.0"`
- `safe_vault = "=0.14.0"`
It took some work to figure out why they are not working, and make a test case.
This PR remove use of `conflicting_activations` before it is extended with the conflicting from next.
https://github.com/rust-lang/cargo/pull/5187#issuecomment-373830919
However the `find_candidate(` is still needed so it now gets the conflicting from next before being called.
It often happens that the candidate whose child will fail leading to it's failure, will have older siblings that have already set up `backtrack_frame`s. The candidate knows that it's failure can not be saved by its siblings, but sometimes we activate the child anyway for the error messages. Unfortunately the child does not know that is uncles can't save it, so it backtracks to one of them. Leading to a combinatorial loop.
The solution is to clear the `backtrack_stack` if we are activating just for the error messages.
Edit original end of this message, no longer accurate.
#5168 means that when we find a permanent problem we will never **activate** its parent again. In practise there afften is a lot of work and `backtrack_frame`s between the problem and reactivating its parent. This PR removes `backtrack_frame`s where its parent and the problem are present. This means that when we find a permanent problem we will never **backtrack** into it again.
An alternative is to scan all cashed problems while backtracking, but this seemed more efficient.
Fix a regression with parsing multivalue options
By default, clap interprets
```
cargo run --bin foo bar baz
```
as
```
cargo run --bin foo --bin bar --bin baz
```
This behavior is different from docopt and does not play nicely with
positional arguments at all. Luckily, clap has a flag to get the
behavior we want, it just not the default! It will become the default in
the next version of clap, but, until that time, we should be careful
when using the combination of `.long`, `.value_name` and
`.multiple(true)`, and don't forget to specify `.number_of_values(1)` as
well.
@alexcrichton I'd love to merge this fix before updating cargo at rust-lang/rust :)
By default, clap interprets
```
cargo run --bin foo bar baz
```
as
```
cargo run --bin foo --bin bar --bin baz
```
This behavior is different from docopt and does not play nicely with
positional arguments at all. Luckily, clap has a flag to get the
behavior we want, it just not the default! It will become the default in
the next version of clap, but, until that time, we should be careful
when using the combination of `.long`, `.value_name` and
`.multiple(true)`, and don't forget to specify `.number_of_values(1)` as
well.
Don't abort resolution on transitive updates
This commit is directed at fixing #4127, allowing the resolver to automatically
perform transitive updates when required. A few use casese and tagged links are
hanging off #4127 itself, but the crux of the issue happens when you either add
a dependency or update a version requirement in `Cargo.toml` which conflicts
with something listed in your `Cargo.lock`. In this case Cargo would previously
provide an obscure "cannot resolve" error whereas this commit updates Cargo to
automatically perform a conservative re-resolution of the dependency graph.
It's hoped that this commit will help reduce the number of "unresolvable"
dependency graphs we've seen in the wild and otherwise make Cargo a little more
ergonomic to use as well. More details can be found in the source's comments!
Closes#4127
This commit is directed at fixing #4127, allowing the resolver to automatically
perform transitive updates when required. A few use casese and tagged links are
hanging off #4127 itself, but the crux of the issue happens when you either add
a dependency or update a version requirement in `Cargo.toml` which conflicts
with something listed in your `Cargo.lock`. In this case Cargo would previously
provide an obscure "cannot resolve" error whereas this commit updates Cargo to
automatically perform a conservative re-resolution of the dependency graph.
It's hoped that this commit will help reduce the number of "unresolvable"
dependency graphs we've seen in the wild and otherwise make Cargo a little more
ergonomic to use as well. More details can be found in the source's comments!
Closes#4127Closes#5182
Don't rewrite dep-info files if they don't change
Similar to how we treat lock files, read the contents, compare, and if they're
the same don't actually write the file.
Closes#5172
Copy `.pdb` files to `target` directory
`.pdb` files are for windows debug info (unlike on linux, debug info is
in a separate file). Windows executable actually hard-code paths to
`.pdb` files, so debugging mvsc rust programs works even without this
patch. However, if you want to distribute the executable to other
machines, you'd better distribute both `foo.exe` and `foo.pdb`, because
absolute paths won't work on another machine. Having same-named .pdb
file alongside the binary would work though.
closes#4960
Depending on the OS, there might be an additional artifacts for
debuginfo (dSYM folder for macs, .pbd file for windows). Given that we
can't disable `.pdb` for windows[1], let's just ignore all filenames!
[1]: https://github.com/rust-lang/rust/pull/28505)
`.pdb` files are for windows debug info (unlike on linux, debug info is
in a separate file). Windows executable actually hard-code paths to
`.pdb` files, so debugging mvsc rust programs works even without this
patch. However, if you want to distribute the executable to other
machines, you'd better distribute both `foo.exe` and `foo.pdb`, because
absolute paths won't work on another machine. Having same-named .pdb
file alongside the binary would work though.
closes#4960
Faster resolver: Cache past conflicting_activations, prevent doing the same work repeatedly.
This work is inspired by @alexcrichton's [comment](https://github.com/rust-lang/cargo/issues/4066#issuecomment-303912744) that a slow resolver can be caused by all versions of a dependency being yanked. Witch stuck in my brain as I did not understand why it would happen. If a dependency has no candidates then it will be the most constrained and will trigger backtracking in the next tick. Eventually I found a reproducible test case. If the bad dependency is deep in the tree of dependencies then we activate and backtrack `O(versions^depth)` times. Even tho it is fast to identify the problem that is a lot of work.
**The set up:**
1. Every time we backtrack cache the (dep, `conflicting_activations`).
2. Build on the work in #5000, Fail to activate if any of its dependencies will just backtrack to this frame. I.E. for each dependency check if any of its cached `conflicting_activations` are already all activated. If so we can just skip to the next candidate. We also add that bad `conflicting_activations` to our set of `conflicting_activations`, so that we can...
**The pay off:**
If we fail to find any candidates that we can activate in lite of 2, then we cannot be activated in this context, add our (dep, `conflicting_activations`) to the cache so that next time our parent will not bother trying us.
I hear you saying "but the error messages, what about the error messages?" So if we are at the end `!has_another` then we disable this optimization. After we mark our dep as being not activatable then we activate anyway. It won't resolve but it will have the same error message as before this PR. If we have been activated for the error messages then skip straight to the last candidate, as that is the only backtrack that will end with the user.
I added a test in the vain of #4834. With the old code the time to run was `O(BRANCHING_FACTOR ^ DEPTH)` and took ~3min with DEPTH = 10; BRANCHING_FACTOR = 5; with the new code it runs almost instantly with 200 and 100.
Clap
Reopening of #5129
So, looks like all tests are 🍏 on my machine!
I definitely want to refactor it some more, and also manually checked that we haven't regressed any help messages, but all the major parts are in place already.
Error message for package "depends on itself" lists the packages in the cycle.
I got a cycle while trying to build someone else's code and cargo's error message didn't point me in the right direction, just mentioned there was a cycle. I thought we could be a bit more helpful.
Don't know what you think of {:#?} as the display but it seemed minimal code so I thought I'd start with that. I could compress the output to one package per line if preferred.
Don't require dev-dependencies when not needed in certain cases
Specifically, when running `cargo install` and add a flag for this behaviour in `cargo build`.
This closes#4988.
Drop ignored tests
r? @alexcrichton
These tests are ignored, so its better to remove them. `run` does not supports `--bins` argument, so I've left a single test that checks specifically for this.
cc https://github.com/rust-lang/cargo/pull/3901
Try a lot harder to recover corrupt git repos
We've received a lot of intermittent bug reports historically about corrupt git
repositories. These inevitably happens as Cargo is ctrl-c'd or for whatever
other reason, and to provide a better user experience Cargo strives to
automatically handle these situations by blowing away the old checkout for a new
update.
This commit adds a new test which attempts to pathologically corrupt a git
database and checkout in an attempt to expose bugs in Cargo. Sure enough there
were some more locations that we needed to handle gracefully for corrupt git
checkouts. Notable inclusions were:
* The `fetch` operation in libgit2 would fail due to corrupt references. This
starts by adding an explicit whitelist for classes of errors coming out of
`fetch` to auto-retry by blowing away the repository. We need to be super
careful here as network errors commonly come out of this function and we don't
want to too aggressively re-clone.
* After a `fetch` succeeded a repository could fail to actual resolve a
git reference to the actual revision we want. This indicated that we indeed
needed to blow everything away and re-clone entirely again.
* When creating a checkout from a database the `reset` operation might fail due
to a corrupt local database of the checkout itself. If this happens we needed
to just blow it away and try again.
There's likely more lurking situations where we need to re-clone but I figure we
can discover those over time.
We've received a lot of intermittent bug reports historically about corrupt git
repositories. These inevitably happens as Cargo is ctrl-c'd or for whatever
other reason, and to provide a better user experience Cargo strives to
automatically handle these situations by blowing away the old checkout for a new
update.
This commit adds a new test which attempts to pathologically corrupt a git
database and checkout in an attempt to expose bugs in Cargo. Sure enough there
were some more locations that we needed to handle gracefully for corrupt git
checkouts. Notable inclusions were:
* The `fetch` operation in libgit2 would fail due to corrupt references. This
starts by adding an explicit whitelist for classes of errors coming out of
`fetch` to auto-retry by blowing away the repository. We need to be super
careful here as network errors commonly come out of this function and we don't
want to too aggressively re-clone.
* After a `fetch` succeeded a repository could fail to actual resolve a
git reference to the actual revision we want. This indicated that we indeed
needed to blow everything away and re-clone entirely again.
* When creating a checkout from a database the `reset` operation might fail due
to a corrupt local database of the checkout itself. If this happens we needed
to just blow it away and try again.
There's likely more lurking situations where we need to re-clone but I figure we
can discover those over time.
backtrack if can not activate
This is a fix for #4347
Unfortunately this too regressed error messages for the case that you specified a dependency feature that does not exist.
@alexcrichton advice on improving the message?
This hasn't been updated in awhile and in general we've been barely using it.
This drops the outdated dependency and vendors a small amount of the
functionality that it provided. I think eventually we'll want to transition away
from this method of assertions but I wanted to get this piece in to avoid too
much churn in one commit.
needs a test where we have an activation_error the then try activate something that dose not work and backtrack to where we had the activation_error then:
- Hit fast backtracking that go past the crate with the missing features
- Or give a bad error message that does not mention the activation_error.
The test will pass, but there is code that is not yet justified by tests
Previously we had logic to explicitly skip lock files but there's actually a
good case to read these from crates.io (#2263) so let's do so!
Closes#2263
* remove Workspace::current_manifest
* remove incorrect logging
* move empty check to Packages::into_package_id_specs
* add test case mentioned in issue
Currently Cargo doesn't publish lock files in crates.io crates but we'll
eventually be doing so, so this changes Cargo to recognize `Cargo.lock` when
it's published to crates.io as use it as the basis for resolution during `cargo
install`.
cc #2263
Some Clippy suggestions
This is just some suggestions from Clippy and intellij rust.
Clippy still has a lot to say, but this is some of the simple things.
Only allow one of each links attribute in resolver
This is a start on fixing the `links` part of #4902. It needs code that adds the links attribute to the index. But, I wanted to get feedback.
Display git errors during authentication.
Certain git errors during authentication were being converted to internal errors which meant that they are only seen if you pass `--verbose`. This may not be obvious, and many of these messages are helpful for diagnosing git errors. This change makes these errors always be displayed.
Fixes#5035.
Note: Some of the git errors are currently unhelpful. Once Cargo has updated git2-rs to include alexcrichton/git2-rs#298, these errors will improve. (@alexcrichton, I can make a PR to update Cargo for the changes in git2 if you'd like).
I'm uncertain if this is a good solution, since the error messages in some cases are a little verbose (such as showing `class=...`). Here is a sample of what some of the messages look like:
<details><summary>Error Message Examples</summary>
<p>
Example of the git message shown below the "attempted yadda yadda" message.
Scenario | Message
---------|--------
No ssh-agent, multiple usernames | `error authenticating: ; class=Ssh (23)`
| | †`error authenticating: no auth sock variable; class=Ssh (23)`
No ssh-agent, one username | `an unknown git error occurred`
| | †`error authenticating: no auth sock variable; class=Ssh (23)`
Incorrect ssh-agent setup | `error authenticating: failed connecting with agent; class=Ssh (23)`
ssh-agent no keys, one username | `an unknown git error occurred`
| | †`failed to acquire username/password from local configuration`
ssh-agent no keys, multiple usernames | `error authenticating: ; class=Ssh (23)`
| | †`no authentication available`
auth success, bad path | `fatal: '/path/to/repo/' does not appear to be a git repository; class=Ssh (23); code=Eof (-20)`
| | ‡`ERROR: Repository not found.; class=Ssh (23); code=Eof (-20)`
bad username | `an unknown git error occurred`
| | †`failed to acquire username/password from local configuration`
| | ‡`error authenticating: Username/PublicKey combination invalid; class=Ssh (23)`
† - Messages once git2-rs is updated.
‡ - Github message
</p>
</details>
