2014-07-16 00:51:49 +00:00
|
|
|
[project]
|
|
|
|
name = "cargo"
|
2016-03-03 19:12:10 +00:00
|
|
|
version = "0.10.0"
|
2014-07-16 00:51:49 +00:00
|
|
|
authors = ["Yehuda Katz <wycats@gmail.com>",
|
2014-11-13 23:08:17 +00:00
|
|
|
"Carl Lerche <me@carllerche.com>",
|
|
|
|
"Alex Crichton <alex@alexcrichton.com>"]
|
2015-07-24 21:37:03 +00:00
|
|
|
license = "MIT/Apache-2.0"
|
|
|
|
homepage = "https://crates.io"
|
|
|
|
repository = "https://github.com/rust-lang/cargo"
|
2016-03-03 21:09:52 +00:00
|
|
|
documentation = "http://doc.crates.io/cargo"
|
2015-07-24 21:37:03 +00:00
|
|
|
description = """
|
|
|
|
Cargo, a package manager for Rust.
|
|
|
|
"""
|
2014-07-16 00:51:49 +00:00
|
|
|
|
2014-08-16 20:00:46 +00:00
|
|
|
[lib]
|
2014-07-16 00:51:49 +00:00
|
|
|
name = "cargo"
|
|
|
|
path = "src/cargo/lib.rs"
|
|
|
|
|
2014-12-19 03:01:37 +00:00
|
|
|
[dependencies]
|
2015-06-15 05:13:08 +00:00
|
|
|
advapi32-sys = "0.1"
|
Refactor Cargo's backend, again!
This commit started out identifying a relatively simple bug in Cargo. A recent
change made it such that the resolution graph included all target-specific
dependencies, relying on the structure of the backend to filter out those which
don't need to get built. This was unfortunately not accounted for in the portion
of the backend that schedules work, mistakenly causing spurious rebuilds if
different runs of the graph pulled in new crates. For example if `cargo build`
didn't build any target-specific dependencies but then later `cargo test` did
(e.g. a dev-dep pulled in a target-specific dep unconditionally) then it would
cause a rebuild of the entire graph.
This class of bug is certainly not the first in a long and storied history of
the backend having multiple points where dependencies are calculated and those
often don't quite agree with one another. The purpose of this rewrite is
twofold:
1. The `Stage` enum in the backend for scheduling work and ensuring that maximum
parallelism is achieved is removed entirely. There is already a function on
`Context` which expresses the dependency between targets (`dep_targets`)
which takes a much finer grain of dependencies into account as well as
already having all the logic for what-depends-on-what. This duplication has
caused numerous problems in the past, an unifying these two will truly grant
maximum parallelism while ensuring that everyone agrees on what their
dependencies are.
2. A large number of locations in the backend have grown to take a (Package,
Target, Profile, Kind) tuple, or some subset of this tuple. In general this
represents a "unit of work" and is much easier to pass around as one
variable, so a `Unit` was introduced which references all of these variables.
Almost the entire backend was altered to take a `Unit` instead of these
variables specifically, typically providing all of the contextual information
necessary for an operation.
A crucial part of this change is the inclusion of `Kind` in a `Unit` to ensure
that everyone *also* agrees on what architecture they're compiling everything
for. There have been many bugs in the past where one part of the backend
determined that a package was built for one architecture and then another part
thought it was built for another. With the inclusion of `Kind` in dependency
management this is handled in a much cleaner fashion as it's only calculated in
one location.
Some other miscellaneous changes made were:
* The `Platform` enumeration has finally been removed. This has been entirely
subsumed by `Kind`.
* The hokey logic for "build this crate once" even though it may be depended on
by both the host/target kinds has been removed. This is now handled in a much
nicer fashion where if there's no target then Kind::Target is just never used,
and multiple requests for a package are just naturally deduplicated.
* There's no longer any need to build up the "requirements" for a package in
terms of what platforms it's compiled for, this now just naturally falls out
of the dependency graph.
* If a build script is overridden then its entire tree of dependencies are not
compiled, not just the build script itself.
* The `threadpool` dependency has been replaced with one on `crossbeam`. The
method of calculating dependencies has quite a few non-static lifetimes and
the scoped threads of `crossbeam` are now used instead of a thread pool.
* Once any thread fails to execute a command work is no longer scheduled unlike
before where some extra pending work may continue to start.
* Many functions used early on, such as `compile` and `build_map` have been
massively simplified by farming out dependency management to
`Context::dep_targets`.
* There is now a new profile to represent running a build script. This is used
to inject dependencies as well as represent that a library depends on running
a build script, not just building it.
This change has currently been tested against cross-compiling Servo to Android
and passes the test suite (which has quite a few corner cases for build scripts
and such), so I'm pretty confident that this refactoring won't have at least too
many regressions!
2015-10-02 06:48:47 +00:00
|
|
|
crates-io = { path = "src/crates-io", version = "0.1" }
|
2016-01-20 17:07:47 +00:00
|
|
|
crossbeam = "0.2"
|
2015-04-26 21:42:55 +00:00
|
|
|
curl = "0.2"
|
|
|
|
docopt = "0.6"
|
2015-03-26 18:17:44 +00:00
|
|
|
env_logger = "0.3"
|
2015-06-15 05:13:08 +00:00
|
|
|
filetime = "0.1"
|
2015-04-26 21:42:55 +00:00
|
|
|
flate2 = "0.2"
|
Fix running Cargo concurrently
Cargo has historically had no protections against running it concurrently. This
is pretty unfortunate, however, as it essentially just means that you can only
run one instance of Cargo at a time **globally on a system**.
An "easy solution" to this would be the use of file locks, except they need to
be applied judiciously. It'd be a pretty bad experience to just lock the entire
system globally for Cargo (although it would work), but otherwise Cargo must be
principled how it accesses the filesystem to ensure that locks are properly
held. This commit intends to solve all of these problems.
A new utility module is added to cargo, `util::flock`, which contains two types:
* `FileLock` - a locked version of a `File`. This RAII guard will unlock the
lock on `Drop` and I/O can be performed through this object. The actual
underlying `Path` can be read from this object as well.
* `Filesystem` - an unlocked representation of a `Path`. There is no "safe"
method to access the underlying path without locking a file on the filesystem
first.
Built on the [fs2] library, these locks use the `flock` system call on Unix and
`LockFileEx` on Windows. Although file locking on Unix is [documented as not so
great][unix-bad], but largely only because of NFS, these are just advisory, and
there's no byte-range locking. These issues don't necessarily plague Cargo,
however, so we should try to leverage them. On both Windows and Unix the file
locks are released when the underlying OS handle is closed, which means that
if the process dies the locks are released.
Cargo has a number of global resources which it now needs to lock, and the
strategy is done in a fairly straightforward way:
* Each registry's index contains one lock (a dotfile in the index). Updating the
index requires a read/write lock while reading the index requires a shared
lock. This should allow each process to ensure a registry update happens while
not blocking out others for an unnecessarily long time. Additionally any
number of processes can read the index.
* When downloading crates, each downloaded crate is individually locked. A lock
for the downloaded crate implies a lock on the output directory as well.
Because downloaded crates are immutable, once the downloaded directory exists
the lock is no longer needed as it won't be modified, so it can be released.
This granularity of locking allows multiple Cargo instances to download
dependencies in parallel.
* Git repositories have separate locks for the database and for the project
checkout. The datbase and checkout are locked for read/write access when an
update is performed, and the lock of the checkout is held for the entire
lifetime of the git source. This is done to ensure that any other Cargo
processes must wait while we use the git repository. Unfortunately there's
just not that much parallelism here.
* Binaries managed by `cargo install` are locked by the local metadata file that
Cargo manages. This is relatively straightforward.
* The actual artifact output directory is just globally locked for the entire
build. It's hypothesized that running Cargo concurrently in *one directory* is
less of a feature needed rather than running multiple instances of Cargo
globally (for now at least). It would be possible to have finer grained
locking here, but that can likely be deferred to a future PR.
So with all of this infrastructure in place, Cargo is now ready to grab some
locks and ensure that you can call it concurrently anywhere at any time and
everything always works out as one might expect.
One interesting question, however, is what does Cargo do on contention? On one
hand Cargo could immediately abort, but this would lead to a pretty poor UI as
any Cargo process on the system could kick out any other. Instead this PR takes
a more nuanced approach.
* First, all locks are attempted to be acquired (a "try lock"). If this
succeeds, we're done.
* Next, Cargo prints a message to the console that it's going to block waiting
for a lock. This is done because it's indeterminate how long Cargo will wait
for the lock to become available, and most long-lasting operations in Cargo
have a message printed for them.
* Finally, a blocking acquisition of the lock is issued and we wait for it to
become available.
So all in all this should help Cargo fix any future concurrency bugs with file
locking in a principled fashion while also allowing concurrent Cargo processes
to proceed reasonably across the system.
[fs2]: https://github.com/danburkert/fs2-rs
[unix-bad]: http://0pointer.de/blog/projects/locking.html
Closes #354
2016-03-12 17:58:53 +00:00
|
|
|
fs2 = "0.2"
|
2016-02-26 21:47:56 +00:00
|
|
|
git2 = "0.4"
|
|
|
|
git2-curl = "0.4"
|
2015-04-26 21:42:55 +00:00
|
|
|
glob = "0.2"
|
2016-01-20 17:07:47 +00:00
|
|
|
kernel32-sys = "0.2"
|
2015-12-20 22:58:55 +00:00
|
|
|
libc = "0.2"
|
2016-02-26 21:47:56 +00:00
|
|
|
libgit2-sys = "0.4"
|
2015-04-26 21:42:55 +00:00
|
|
|
log = "0.3"
|
2015-06-15 05:13:08 +00:00
|
|
|
num_cpus = "0.2"
|
2015-04-26 21:42:55 +00:00
|
|
|
regex = "0.1"
|
|
|
|
rustc-serialize = "0.3"
|
2016-02-01 20:12:17 +00:00
|
|
|
semver = "0.2.2"
|
2016-01-20 17:07:47 +00:00
|
|
|
tar = "0.4"
|
2016-02-01 01:03:14 +00:00
|
|
|
term = "0.4.4"
|
2015-04-26 21:42:55 +00:00
|
|
|
time = "0.1"
|
|
|
|
toml = "0.1"
|
|
|
|
url = "0.2"
|
2015-07-22 21:38:35 +00:00
|
|
|
winapi = "0.2"
|
2015-01-20 16:01:52 +00:00
|
|
|
|
2015-03-09 19:07:35 +00:00
|
|
|
[dev-dependencies]
|
|
|
|
tempdir = "0.3"
|
2015-07-24 21:37:03 +00:00
|
|
|
hamcrest = "0.1"
|
2015-05-13 04:22:14 +00:00
|
|
|
bufstream = "0.1"
|
2015-06-13 00:37:19 +00:00
|
|
|
filetime = "0.1"
|
2014-09-27 04:14:46 +00:00
|
|
|
|
2014-07-16 00:51:49 +00:00
|
|
|
[[bin]]
|
|
|
|
name = "cargo"
|
|
|
|
test = false
|
2014-08-14 21:54:56 +00:00
|
|
|
doc = false
|
2014-07-16 00:51:49 +00:00
|
|
|
|
|
|
|
[[test]]
|
|
|
|
name = "tests"
|
2014-10-17 15:17:17 +00:00
|
|
|
|
|
|
|
[[test]]
|
|
|
|
name = "resolve"
|