This is intended to fix#6747 where multiple Cargos invoked with the
same jobserver would all have their own token but not actually run
concurrently due to file locking. Instead the fix is that whenever Cargo
blocks for a file lock with a configured global jobserver, a token is
released just before we block and then reacquired afterwards. This way
we should ensure that we're not hogging a cpu/token unnecessarily
without doing any work!
Closes#6747
It can be difficult to figure out what's wrong when a user reports that
`cargo fix` fails. There's often a large list of warnings, and it can
be hard to figure out which one caused a compile error.
This makes the deny(warnings) in the testsuite conditional on a new
"deny-warnings" feature, that is then enabled in CI.
Ideally I could use the (reasonably well established) CI env var (like
we do for proptests), but I don't know how to get the attribute to be
defined in terms of an env var.
We're already pulling in zlib for other dependencies like curl/libgit2
so there's not really much use in duplicating the compression code with
miniz, so let's instruct `flate2` to use libz as well to compress and
decompress chunks.
This commit switches Cargo to using HTTP/2 by default. This is
controlled via the `http.multiplexing` configuration variable and has
been messaged out for testing [1] (although got very few responses).
There's been surprisingly little fallout from parallel downloads, so
let's see how this goes!
[1]: https://internals.rust-lang.org/t/testing-cargos-parallel-downloads/8466
Fix timeouts firing while tarballs are extracted
This commit fixes#6125 by ensuring that while we're extracting tarballs
or doing other synchronous work like grabbing file locks we're not
letting the timeout timers of each HTTP transfer keep ticking. This is
curl's default behavior (which we don't want in this scenario). Instead
the timeout logic is inlined directly and we manually account for the
synchronous work happening not counting towards timeout limits.
Closes#6125
This commit fixes#6125 by ensuring that while we're extracting tarballs
or doing other synchronous work like grabbing file locks we're not
letting the timeout timers of each HTTP transfer keep ticking. This is
curl's default behavior (which we don't want in this scenario). Instead
the timeout logic is inlined directly and we manually account for the
synchronous work happening not counting towards timeout limits.
Closes#6125
use proptest to fuzz the resolver
This has been a long time goal. This uses proptest to generate random registry indexes and throws them at the resolver.
It would be simple to generate a registry by,
1. make a list of name and version number each picked at random
2. for each pick a list of dependencies by making a list of name and version requirements at random.
Unfortunately, it would be extremely unlikely to generate any interesting cases, as the chance that the random name you depend on was also generated as the name of a crate is vanishingly small. So this implementation works very hard to ensure that it only generates valid dependency requirements.
This is still a WIP as it has many problems:
- [x] The current strategy is very convoluted. It is hard to see that it is correct, and harder to see how it can be expanded. Thanks to @centril for working with me on IRC to get this far. Do you have advice for improving it?
- [X] It is slow as molasses when run without release. I looked with a profilere and we seem to spend 2/3 of the time in `to_url`. Maybe we can special case `example.com` for test, like we do for `crates.io` or something? Edit: Done. `lazy_static` did its magic.
- [x] `proptest` does not yet work with `minimal-versions`, a taste of my own medicine.
- [x] I have not verified that, if I remove the fixes for other test that this regenerates them.
The current strategy does not:
- [x] generate interesting version numbers, it just dose 1.0.0, 2.0.0 ...
- [x] guarantee that the version requirements are possible to meet by the crate named.
- [ ] generate features.
- [ ] generate dev-dependencies.
- [x] build deep dependency trees, it seems to prefer to generate crates with 0 or 1 dependents so that on average the tree is 1 or 2 layers deep.
And last but not least, there are no interesting properties being tested. Like:
- [ ] If resolution was successful, then all the transitive requirements are met.
- [x] If resolution was successful, then unpublishing a version of a crate that was not selected should not change that.
- [x] If resolution was unsuccessful, then it should stay unsuccessful even if any version of a crate is unpublished.
- [ ] @maurer suggested testing for consistency. Same registry, same cargo version, same lockfile, every time.
- [ ] @maurer suggested a pareto optimality property (if all else stays the same, but new package versions are released, we don't get a new lockfile where every version is <= the old one, and at least one is < the old one)
This commit tweaks how configuration is loaded for `cargo install`, ensuring
that we only load configuration from `$HOME` instead of the current working
directory. This should make installations a little more consistent in that they
probably shouldn't cover project-local configuration but should respect global
configuration!
Closes#6025
This is actually a super tricky problem. We don't really have the capacity for
more than one line of update-able information in Cargo right now, so we need to
squeeze a lot of information into one line of output for Cargo. The main
constraints this tries to satisfy are:
* At all times it should be clear what's happening. Cargo shouldn't just hang
with no output when downloading a crate for a long time, a counter ideally
needs to be decreasing while the download progresses.
* If a progress bar is shown, it shouldn't jump around. This ends up just being
a surprising user experience for most. Progress bars should only ever
increase, but they may increase at different speeds.
* Cargo has, currently, at most one line of output (as mentioned above) to pack
information into. We haven't delved into fancier terminal features that
involve multiple lines of update-able output.
* When downloading crates as part of `cargo build` (the norm) we don't actually
know ahead of time how many crates are being downloaded. We rely on the
calculation of unit dependencies to naturally feed into downloading more
crates.
* Furthermore, once we decide to download a crate, we don't actually know how
big it is! We have to wait for the server to tell us how big it is.
There doesn't really seem to be a great solution that satisfies all of these
constraints unfortunately. As a result this commit implements a relatively
conservative solution which should hopefully get us most of the way there. There
isn't actually a progress bar but rather Cargo prints that it's got N crates
left to download, and if it takes awhile it prints out that there are M bytes
remaining.
Unfortunately the progress is pretty choppy and jerky, not providing a smooth
UI. This appears to largely be because Cargo will synchronously extract
tarballs, which for large crates can cause a noticeable pause. Cargo's not
really prepared internally to perform this work on helper threads, but ideally
if it could do so it would improve the output quite a bit! (making it much
smoother and also able to account for the time tarball extraction takes).
This commit implements parallel downloads using `libcurl` powered by
`libnghttp2` over HTTP/2. Using all of the previous refactorings this actually
implements usage of `Multi` to download crates in parallel. This achieves some
large wins locally, taking download times from 30s to 2s in the best case.
The standard output of Cargo is also changed as a result of this commit. It's
no longer useful for Cargo to print "Downloading ..." for each crate really as
they all start instantaneously. Instead Cargo now no longer prints `Downloading`
by default (unless attached to a pipe) and instead only has one progress bar for
all downloads. Currently this progress bar is discrete and based on the total
number of downloads, no longer specifying how much of one particular download
has happened. This provides a less granular view into what Cargo is doing but
it's hoped that it looks reasonable from an outside perspective as there's
still a progress bar indicating what's happening.
BUG fuzzing found a bug in the resolver, we need a complete set of conflicts to do backjumping
As mentioned in https://github.com/rust-lang/cargo/pull/5921#issuecomment-418890269, the new proptest found a live bug! This PR so far tracs my attempt to minimize the problematic input.
The problem turned out to be that we where backjumping on incomplete set of conflicts.
