Don't require the `serde` feature of `url`
Ends up meaning that in full crate compiles that `url` doesn't wait for
`serde` to finish, which in turn enables crates like `git2` to start
sooner!
Experiment: Create timing report.
This is just an experiment, so I'm not sure if we'll want to merge it.
This adds an HTML report which gets saved to disk when the build is finished. It is primarily geared for identifying slow dependencies, and for visualizing how pipelining affects the build.
Here's an example: https://ehuss.github.io/cargo-timing.html
You can mouse over the blocks to highlight the reverse-dependencies that are released when a unit finishes. `syn` is a really good example.
It does a few other things, like displaying a message after each unit is finished. See the docs for more information.
This home's release remove support for the old `.multirust`
directory. Also it fixes rustup_home and cargo_home implementation
when corresponding environment variables are absolute paths.
Instead of treating Windows differently, this just always uses canonical paths
on all platforms. This fixes a problem where symlinks were not treated
correctly on all platforms.
Switching rm_rf to the remove_dir_all crate because deleting symbolic links on
Windows is difficult.
Tighten requirements for git2 crates
Bring in a few updates, used to update libgit2 and fix a Windows issue
as well as updating the `url` dependencies.
Closes#7173
This commit fixes a test in Cargo to work around a seeming regression in
behavior in libgit2 around HTTP authentication. The expected flow for
HTTP authentication with git is that git sends an HTTP request and
receives an "unauthorized" response. It then sends another request with
authorization information and that's what we're testing is received in
the our test.
Previously libgit2 would issue a new HTTP connection if the previous one
was closed, but it looks like changes in libgit2 now require that the
same HTTP connection is used for the initial request and the subsequent
request with authorization information. This broke our test since it's
not using an HTTP compliant server at all and is just some handwritten
TCP reads/writes. The fix here is to basically stay with handwritten TCP
reads/writes but tweak how it happens so it's all on the same HTTP/TCP
connection to match what libgit2 is expecting.
Some extra assertions have also been added to try to prevent deadlocks
from happening in the future and instead make the test fail fast if this
situation comes up again.
These tests take a good amount of time to run locally and they're also
causing a lot of dependencies to get pulled into rust-lang/rust, so
let's have a separate crate that we just test on our own CI
Test the Resolver against the varisat Library
Resolution can be reduced to the SAT problem. So this is an alternative implementation of the resolver that uses a SAT library for the hard work. This is intended to be easy to read, as compared to the real resolver, and run as part of the test sweet to make sure the real resolver works as expected. Part of #6120.
Some notes on performance:
The initial version did not support public & private deps:
~64 loc, `O(nln(n))` vars, `O(nln(n) + n*d)` clauses, 0.5x slower on `prop_passes_validation`
The final version:
~163 loc, `O(dn^2`) vars, `O(dn^3)` clauses, 1.5x slower on `prop_passes_validation`
That comparison makes me feel better about spending months trying to get public & private deps to be fast enough for stabilization.
Parse less JSON on null builds
This commit fixes a performance pathology in Cargo today. Whenever Cargo
generates a lock file (which happens on all invocations of `cargo build`
for example) Cargo will parse the crates.io index to learn about
dependencies. Currently, however, when it parses a crate it parses the
JSON blob for every single version of the crate. With a lock file,
however, or with incremental builds only one of these lines of JSON is
relevant. Measured today Cargo building Cargo parses 3700 JSON
dependencies in the registry.
This commit implements an optimization that brings down the number of
parsed JSON lines in the registry to precisely the right number
necessary to build a project. For example Cargo has 150 crates in its
lock file, so now it only parses 150 JSON lines (a 20x reduction from
3700). This in turn can greatly improve Cargo's null build time. Cargo
building Cargo dropped from 120ms to 60ms on a Linux machine and 400ms
to 200ms on a Mac.
The commit internally has a lot more details about how this is done but
the general idea is to have a cache which is optimized for Cargo to read
which is maintained automatically by Cargo.
Closes#6866
This commit fixes a performance pathology in Cargo today. Whenever Cargo
generates a lock file (which happens on all invocations of `cargo build`
for example) Cargo will parse the crates.io index to learn about
dependencies. Currently, however, when it parses a crate it parses the
JSON blob for every single version of the crate. With a lock file,
however, or with incremental builds only one of these lines of JSON is
relevant. Measured today Cargo building Cargo parses 3700 JSON
dependencies in the registry.
This commit implements an optimization that brings down the number of
parsed JSON lines in the registry to precisely the right number
necessary to build a project. For example Cargo has 150 crates in its
lock file, so now it only parses 150 JSON lines (a 20x reduction from
3700). This in turn can greatly improve Cargo's null build time. Cargo
building Cargo dropped from 120ms to 60ms on a Linux machine and 400ms
to 200ms on a Mac.
The commit internally has a lot more details about how this is done but
the general idea is to have a cache which is optimized for Cargo to read
which is maintained automatically by Cargo.
Closes#6866
This is intended to fix#6747 where multiple Cargos invoked with the
same jobserver would all have their own token but not actually run
concurrently due to file locking. Instead the fix is that whenever Cargo
blocks for a file lock with a configured global jobserver, a token is
released just before we block and then reacquired afterwards. This way
we should ensure that we're not hogging a cpu/token unnecessarily
without doing any work!
Closes#6747
It can be difficult to figure out what's wrong when a user reports that
`cargo fix` fails. There's often a large list of warnings, and it can
be hard to figure out which one caused a compile error.
This makes the deny(warnings) in the testsuite conditional on a new
"deny-warnings" feature, that is then enabled in CI.
Ideally I could use the (reasonably well established) CI env var (like
we do for proptests), but I don't know how to get the attribute to be
defined in terms of an env var.
We're already pulling in zlib for other dependencies like curl/libgit2
so there's not really much use in duplicating the compression code with
miniz, so let's instruct `flate2` to use libz as well to compress and
decompress chunks.
This commit switches Cargo to using HTTP/2 by default. This is
controlled via the `http.multiplexing` configuration variable and has
been messaged out for testing [1] (although got very few responses).
There's been surprisingly little fallout from parallel downloads, so
let's see how this goes!
[1]: https://internals.rust-lang.org/t/testing-cargos-parallel-downloads/8466
Fix timeouts firing while tarballs are extracted
This commit fixes#6125 by ensuring that while we're extracting tarballs
or doing other synchronous work like grabbing file locks we're not
letting the timeout timers of each HTTP transfer keep ticking. This is
curl's default behavior (which we don't want in this scenario). Instead
the timeout logic is inlined directly and we manually account for the
synchronous work happening not counting towards timeout limits.
Closes#6125
This commit fixes#6125 by ensuring that while we're extracting tarballs
or doing other synchronous work like grabbing file locks we're not
letting the timeout timers of each HTTP transfer keep ticking. This is
curl's default behavior (which we don't want in this scenario). Instead
the timeout logic is inlined directly and we manually account for the
synchronous work happening not counting towards timeout limits.
Closes#6125
use proptest to fuzz the resolver
This has been a long time goal. This uses proptest to generate random registry indexes and throws them at the resolver.
It would be simple to generate a registry by,
1. make a list of name and version number each picked at random
2. for each pick a list of dependencies by making a list of name and version requirements at random.
Unfortunately, it would be extremely unlikely to generate any interesting cases, as the chance that the random name you depend on was also generated as the name of a crate is vanishingly small. So this implementation works very hard to ensure that it only generates valid dependency requirements.
This is still a WIP as it has many problems:
- [x] The current strategy is very convoluted. It is hard to see that it is correct, and harder to see how it can be expanded. Thanks to @centril for working with me on IRC to get this far. Do you have advice for improving it?
- [X] It is slow as molasses when run without release. I looked with a profilere and we seem to spend 2/3 of the time in `to_url`. Maybe we can special case `example.com` for test, like we do for `crates.io` or something? Edit: Done. `lazy_static` did its magic.
- [x] `proptest` does not yet work with `minimal-versions`, a taste of my own medicine.
- [x] I have not verified that, if I remove the fixes for other test that this regenerates them.
The current strategy does not:
- [x] generate interesting version numbers, it just dose 1.0.0, 2.0.0 ...
- [x] guarantee that the version requirements are possible to meet by the crate named.
- [ ] generate features.
- [ ] generate dev-dependencies.
- [x] build deep dependency trees, it seems to prefer to generate crates with 0 or 1 dependents so that on average the tree is 1 or 2 layers deep.
And last but not least, there are no interesting properties being tested. Like:
- [ ] If resolution was successful, then all the transitive requirements are met.
- [x] If resolution was successful, then unpublishing a version of a crate that was not selected should not change that.
- [x] If resolution was unsuccessful, then it should stay unsuccessful even if any version of a crate is unpublished.
- [ ] @maurer suggested testing for consistency. Same registry, same cargo version, same lockfile, every time.
- [ ] @maurer suggested a pareto optimality property (if all else stays the same, but new package versions are released, we don't get a new lockfile where every version is <= the old one, and at least one is < the old one)
This commit tweaks how configuration is loaded for `cargo install`, ensuring
that we only load configuration from `$HOME` instead of the current working
directory. This should make installations a little more consistent in that they
probably shouldn't cover project-local configuration but should respect global
configuration!
Closes#6025
This is actually a super tricky problem. We don't really have the capacity for
more than one line of update-able information in Cargo right now, so we need to
squeeze a lot of information into one line of output for Cargo. The main
constraints this tries to satisfy are:
* At all times it should be clear what's happening. Cargo shouldn't just hang
with no output when downloading a crate for a long time, a counter ideally
needs to be decreasing while the download progresses.
* If a progress bar is shown, it shouldn't jump around. This ends up just being
a surprising user experience for most. Progress bars should only ever
increase, but they may increase at different speeds.
* Cargo has, currently, at most one line of output (as mentioned above) to pack
information into. We haven't delved into fancier terminal features that
involve multiple lines of update-able output.
* When downloading crates as part of `cargo build` (the norm) we don't actually
know ahead of time how many crates are being downloaded. We rely on the
calculation of unit dependencies to naturally feed into downloading more
crates.
* Furthermore, once we decide to download a crate, we don't actually know how
big it is! We have to wait for the server to tell us how big it is.
There doesn't really seem to be a great solution that satisfies all of these
constraints unfortunately. As a result this commit implements a relatively
conservative solution which should hopefully get us most of the way there. There
isn't actually a progress bar but rather Cargo prints that it's got N crates
left to download, and if it takes awhile it prints out that there are M bytes
remaining.
Unfortunately the progress is pretty choppy and jerky, not providing a smooth
UI. This appears to largely be because Cargo will synchronously extract
tarballs, which for large crates can cause a noticeable pause. Cargo's not
really prepared internally to perform this work on helper threads, but ideally
if it could do so it would improve the output quite a bit! (making it much
smoother and also able to account for the time tarball extraction takes).
This commit implements parallel downloads using `libcurl` powered by
`libnghttp2` over HTTP/2. Using all of the previous refactorings this actually
implements usage of `Multi` to download crates in parallel. This achieves some
large wins locally, taking download times from 30s to 2s in the best case.
The standard output of Cargo is also changed as a result of this commit. It's
no longer useful for Cargo to print "Downloading ..." for each crate really as
they all start instantaneously. Instead Cargo now no longer prints `Downloading`
by default (unless attached to a pipe) and instead only has one progress bar for
all downloads. Currently this progress bar is discrete and based on the total
number of downloads, no longer specifying how much of one particular download
has happened. This provides a less granular view into what Cargo is doing but
it's hoped that it looks reasonable from an outside perspective as there's
still a progress bar indicating what's happening.
BUG fuzzing found a bug in the resolver, we need a complete set of conflicts to do backjumping
As mentioned in https://github.com/rust-lang/cargo/pull/5921#issuecomment-418890269, the new proptest found a live bug! This PR so far tracs my attempt to minimize the problematic input.
The problem turned out to be that we where backjumping on incomplete set of conflicts.
run some tests with minimal-versions on CI
In #5757 we discovered that sum test don't pass with minimal-versions, and so only added CI for `cargo check`. This PR is to see if that is still needed, and if it is then which test rely on upstream bugfix.
Improve verbose console and log for finding git repo in package check
Third attempt to resolve#5823 by improving logging and tests. This exposes the issue to testing, via verbose console output and is dependent on alexcrichton/git2-rs#341 as just released in git2 0.7.5 crate. Thus tests *should* now pass on all platforms, incl. windows, but I also intend to bump the minimal git2 release dependency (in a subsequently added commit).
cc: @Eh2406 thanks for your fix and help!
Fully capture rustc and rustdoc output when -Zcompile-progress is passed
Fixes#5764 and #5695.
On Windows, we will parse the ANSI escape code into console commands via my `fwdansi` package, based on @ishitatsuyuki's idea in https://github.com/rust-lang/cargo/issues/5695#issuecomment-406300234. Outside of Windows the content is forwarded as-is.
This commit updates the `cargo fix` implementation to iteratively apply fixes
from the compiler instead of only once. Currently the compiler can sometimes
emit overlapping suggestions, such as in the case of transitioning
::foo::<::Bar>();
to ...
crate::foo::<crate::Bar>();
and `rustfix` rightfully can't handle overlapping suggestions as there's no
clear way of how to disambiguate the fixes. To fix this problem Cargo will now
run `rustc` and `rustfix` multiple times, attempting to reach a steady state
where no fixes failed to apply.
Naturally this is a pretty tricky thing to do and we want to be sure that Cargo
doesn't loop forever, for example. A number of safeguards are in place to
prevent Cargo from going off into the weeds when fixing files, notably avoiding
to reattempt fixes if no successful fixes ended up being applied.
Closes#5813Closesrust-lang/rust#52754
This commit imports the `cargo fix` subcommand in rust-lang-nursery/rustfix
directly into Cargo as a subcommand. This should allow us to ease our
distribution story of `cargo fix` as we prepare for the upcoming 2018 edition
release.
It's been attempted here to make the code as idiomatic as possible for Cargo's
own codebase. Additionally all tests from cargo-fix were imported into Cargo's
test suite as well. After this lands and is published in nightly the `cargo-fix`
command in rust-lang-nursery/rustfix will likely be removed.
cc rust-lang/rust#52272
Right now the rust-lang/rust integration is compiling Cargo twice on dist
builds, once for Cargo and once for the RLS. This is due to a dependency of
Cargo being recompiled with different features when used from the RLS or not.
For now paper over this problem with a synthetic dependency to prevent Cargo
from being compiled twice.
Clap
Reopening of #5129
So, looks like all tests are 🍏 on my machine!
I definitely want to refactor it some more, and also manually checked that we haven't regressed any help messages, but all the major parts are in place already.
This hasn't been updated in awhile and in general we've been barely using it.
This drops the outdated dependency and vendors a small amount of the
functionality that it provided. I think eventually we'll want to transition away
from this method of assertions but I wanted to get this piece in to avoid too
much churn in one commit.
There are some important fixes in jobserver >=0.1.8. With earlier
versions, it's possible for cargo to panic with a "failed to acquire
jobserver token" error, which can be very hard to track down.
Requiring the latest version of jobserver makes sure that no such error
can make it into downstream distributions.
Cargo.toml: Replace '/' with 'OR' in 'license'
Catch up with our recommendations from 7dee65fe (#4898), which deprecated `/` in favor of vanilla SPDX license expressions.
I've gone with the disjunctive `OR`, because the README has:
> Cargo is primarily distributed under the terms of both the MIT license and the Apache License (Version 2.0).
Catch up with our recommendations from 7dee65fe (src/doc/manifest: Pin
'license' to SPDX 2.1 expressions and the 2.4 list, 2018-01-04,
#4898), which deprecated '/' in favor of vanilla SPDX license
expressions.
I've gone with the disjunctive OR, because the README has:
> Cargo is primarily distributed under the terms of both the MIT
> license and the Apache License (Version 2.0).
This commit updates the handling of git checkouts from the database to use
hardlinks if possible, speeding up this operation for large repositories
significantly.
As a refresher, Cargo caches git repositories in a few locations to speed up
local usage of git repositories. Cargo has a "database" folder which is a bare
checkout of any git repository Cargo has cached historically. This database
folder contains effectively a bunch of databases for remote repos that are
updated periodically.
When actually building a crate Cargo will clone this database into a different
location, the checkouts folder. Each rev we build (ever) is cached in the
checkouts folder. This means that once a checkout directory is created it's
frozen for all of time.
This latter step is what this commit is optimizing. When checking out the
database onto the local filesystem at a particular revision. Previously we were
instructing libgit2 to fall back to a "git aware" transport which was
exceedingly slow on some systems for filesystem-to-filesystem transfers. This
optimization (we just forgot to turn it on in libgit2) is a longstanding one and
should speed this up significantly!
Closes#4604
This commit is the initial steps to migrate Cargo's error handling from the
`error-chain` crate to the `failure` crate. This is intended to be a low-cost
(in terms of diff) transition where possible so it's note "purely idiomatic
`failure` crate" just yet.
The `error-chain` dependency is dropped in this commit and Cargo now canonically
uses the `Error` type from the `failure` crate. The main last remnant of
`error-chain` is a custom local extension trait to use `chain_err` instead of
`with_context`. I'll try to follow up with a commit that renames this later but
I wanted to make sure everything worked first! (and `chain_err` is used
practically everywhere).
Some minor tweaks happened in the tests as I touched up a few error messages
here and there but overall the UI of Cargo should be exactly the same before and
after this commit.
Although the checks are desirable, they cause warnings in the rust build
(due to workspaces) which could cause needless concern. The checks
aren't too important, so just disable them.
There's now a lock file upstream in rust-lang/rust so the one here isn't
actually used, and otherwise this crate is used as a dependency so the lock file
isn't respected anyway!
Use `same-file` to avoid unnecessary hard links
This is targeted at removing the need for a workaround in rust-lang/rust#39518,
allowing the main rust build system to move back to hard links which should be
much more efficient.
This is targeted at removing the need for a workaround in rust-lang/rust#39518,
allowing the main rust build system to move back to hard links which should be
much more efficient.
Add gitignore-like pattern matching logic to `list_files()` and throw
warnings for paths getting different inclusion/exclusion results from
the old and the new methods.
Migration Tracking: <https://github.com/rust-lang/cargo/issues/4268>
The API of `termcolor` fits what the system gives us much more nicely and should
be well battle-tested from ripgrep. Additionally we don't really need huge
terminfo parsers, that wasn't every really the intention of the color support
here.
Removing some allocations arounds the stored hashes by having nested hash maps
instead of tuple keys. Also remove an intermediate array when parsing
dependencies through a custom implementation of `Deserialize`. While this
doesn't make this code path blazingly fast it definitely knocks it down in the
profiles below other higher-value targets.
This commit adds a GNU make jobserver implementation to Cargo, both as a client
of existing jobservers and also a creator of new jobservers. The jobserver is
actually just an IPC semaphore which manifests itself as a pipe with N bytes
of tokens on Unix and a literal IPC semaphore on Windows. The rough protocol
is then if you want to run a job you read acquire the semaphore (read a byte on
Unix or wait on the semaphore on Windows) and then you release it when you're
done.
All the hairy details of the jobserver implementation are housed in the
`jobserver` crate on crates.io instead of Cargo. This should hopefully make it
much easier for the compiler to also share a jobserver implementation
eventually.
The main tricky bit here is that on Unix and Windows acquiring a jobserver token
will block the calling thread. We need to either way for a running job to exit
or to acquire a new token when we want to spawn a new job. To handle this the
current implementation spawns a helper thread that does the blocking and sends a
message back to Cargo when it receives a token. It's a little trickier with
shutting down this thread gracefully as well but more details can be found in
the `jobserver` crate.
Unfortunately crates are unlikely to see an immediate benefit of this once
implemented. Most crates are run with a manual `make -jN` and this overrides the
jobserver in the environment, creating a new jobserver in the sub-make. If the
`-jN` argument is removed, however, then `make` will share Cargo's jobserver and
properly limit parallelism.
Closes#1744
Convert CargoResult, CargoError into an implementation provided by error-chain. The previous is_human machinery is mostly removed; now errors are displayed unless of the Internal kind, verbose mode will print all errors.
Add year to project template variables
This adds the current year as a `year` variable for project templates. Some license files / headers include the year, so this should make it easier to include those in a template.
