The primary targets here are openssl and openssl-sys crates 0.9,
bringing support for OpenSSL 1.1.0. This requires updating the curl
and git2 related dependencies as well.
A small change is required in cargo itself for the new Hasher API.
Results from the hasher are simply unwrapped for now, matching the
Windows behavior that already panics on error.
This commit alters Cargo's behavior when the `-vv` option is passed (two verbose
flags) to stream output of all build scripts to the console. Cargo makes not
attempt to prevent interleaving or indicate *which* build script is producing
output, rather it simply forwards all output to one to the console.
Cargo still acts as a middle-man, capturing the output, to parse build script
output and interpret the results. The parsing is still deferred to completion
but the stream output happens while the build script is running.
On Unix this is implemented via `select` and on Windows this is implemented via
IOCP.
Closes#1106
Update TOML parser to pick up a bugfix
Cargo has previously accepted invalid TOML as valid, but this bugfix should fix
the problem. In order to prevent breaking all crates immediately toml-rs has a
compatibility mode which emulates the bug that was fixed. Cargo will issue a
warning if this compatibility is required to parse a crate.
Cargo has previously accepted invalid TOML as valid, but this bugfix should fix
the problem. In order to prevent breaking all crates immediately toml-rs has a
compatibility mode which emulates the bug that was fixed. Cargo will issue a
warning if this compatibility is required to parse a crate.
Compiling everything in one binary was getting annoying as it just took forever
to build, instead shard it all up so we can build just particular test suites at
a time.
Dearest Reviewer,
This branch resolves#1602 which relates to retrying network
issues automatically. There is a new utility helper for retrying
any network call.
There is a new config called net.retry value in the .cargo.config
file. The default value is 2. The documentation has also been
updated to reflect the new value.
Thanks
Becker
and each cargo-install instance creates and uses its own build directory. This
allows running several cargo-install instances in parallel.
If we fail to create a temporary directory for whatever reason fallback to
creating and using a target-install directory in the current directory.
closes#2606
Cargo has historically had no protections against running it concurrently. This
is pretty unfortunate, however, as it essentially just means that you can only
run one instance of Cargo at a time **globally on a system**.
An "easy solution" to this would be the use of file locks, except they need to
be applied judiciously. It'd be a pretty bad experience to just lock the entire
system globally for Cargo (although it would work), but otherwise Cargo must be
principled how it accesses the filesystem to ensure that locks are properly
held. This commit intends to solve all of these problems.
A new utility module is added to cargo, `util::flock`, which contains two types:
* `FileLock` - a locked version of a `File`. This RAII guard will unlock the
lock on `Drop` and I/O can be performed through this object. The actual
underlying `Path` can be read from this object as well.
* `Filesystem` - an unlocked representation of a `Path`. There is no "safe"
method to access the underlying path without locking a file on the filesystem
first.
Built on the [fs2] library, these locks use the `flock` system call on Unix and
`LockFileEx` on Windows. Although file locking on Unix is [documented as not so
great][unix-bad], but largely only because of NFS, these are just advisory, and
there's no byte-range locking. These issues don't necessarily plague Cargo,
however, so we should try to leverage them. On both Windows and Unix the file
locks are released when the underlying OS handle is closed, which means that
if the process dies the locks are released.
Cargo has a number of global resources which it now needs to lock, and the
strategy is done in a fairly straightforward way:
* Each registry's index contains one lock (a dotfile in the index). Updating the
index requires a read/write lock while reading the index requires a shared
lock. This should allow each process to ensure a registry update happens while
not blocking out others for an unnecessarily long time. Additionally any
number of processes can read the index.
* When downloading crates, each downloaded crate is individually locked. A lock
for the downloaded crate implies a lock on the output directory as well.
Because downloaded crates are immutable, once the downloaded directory exists
the lock is no longer needed as it won't be modified, so it can be released.
This granularity of locking allows multiple Cargo instances to download
dependencies in parallel.
* Git repositories have separate locks for the database and for the project
checkout. The datbase and checkout are locked for read/write access when an
update is performed, and the lock of the checkout is held for the entire
lifetime of the git source. This is done to ensure that any other Cargo
processes must wait while we use the git repository. Unfortunately there's
just not that much parallelism here.
* Binaries managed by `cargo install` are locked by the local metadata file that
Cargo manages. This is relatively straightforward.
* The actual artifact output directory is just globally locked for the entire
build. It's hypothesized that running Cargo concurrently in *one directory* is
less of a feature needed rather than running multiple instances of Cargo
globally (for now at least). It would be possible to have finer grained
locking here, but that can likely be deferred to a future PR.
So with all of this infrastructure in place, Cargo is now ready to grab some
locks and ensure that you can call it concurrently anywhere at any time and
everything always works out as one might expect.
One interesting question, however, is what does Cargo do on contention? On one
hand Cargo could immediately abort, but this would lead to a pretty poor UI as
any Cargo process on the system could kick out any other. Instead this PR takes
a more nuanced approach.
* First, all locks are attempted to be acquired (a "try lock"). If this
succeeds, we're done.
* Next, Cargo prints a message to the console that it's going to block waiting
for a lock. This is done because it's indeterminate how long Cargo will wait
for the lock to become available, and most long-lasting operations in Cargo
have a message printed for them.
* Finally, a blocking acquisition of the lock is issued and we wait for it to
become available.
So all in all this should help Cargo fix any future concurrency bugs with file
locking in a principled fashion while also allowing concurrent Cargo processes
to proceed reasonably across the system.
[fs2]: https://github.com/danburkert/fs2-rs
[unix-bad]: http://0pointer.de/blog/projects/locking.htmlCloses#354
This crate was recently updated to the next release of libgit2, and I've noticed
historically that a noop `cargo build` was slow in the git2-rs repository.
Curious to see if the new libgit2 version helped speed things up at all, I
tested it out.
Before this commit, a noop `cargo build` produced 599108 syscalls. After this
commit, a noop build produced 86925 syscalls, an 85% reduction in the number of
syscalls! Needless to say it's much faster.
Before v0.4, term used to return `Ok(true)` if something succeeded,
`Ok(false)` if the operation was unsupported, and `Err(io::Error)` if
there was an IO error. Now, it returns `Ok(())` if the operation
succeeds and `Err(term::Error)` if the operation fails. If the operation
is unsupported, it returns `Err(term::Error::NotSupported)`. This means
that, if `op` is unsupported, `try!(term.op())` will now return an error
instead of silently failing (well, return false but that's effectively
silent).
Fixes#2338
This is undoing part of commit dd34296Fix#2338
(Unfortunately I do not have a suggestion for how to make a unit test
for this problem; doing so would require putting in a bit more effort
than I have time for at the moment.)
* Updates git2-rs back to 0.3 now that the distribution issue on OSX has been
fixed.
* Updates libgit2-sys to using the `cmake` crate so building with VS 2015 can
work.
* Update pkg-config to totally disable it on MSVC (basically guaranteed to never
work)
This commit started out identifying a relatively simple bug in Cargo. A recent
change made it such that the resolution graph included all target-specific
dependencies, relying on the structure of the backend to filter out those which
don't need to get built. This was unfortunately not accounted for in the portion
of the backend that schedules work, mistakenly causing spurious rebuilds if
different runs of the graph pulled in new crates. For example if `cargo build`
didn't build any target-specific dependencies but then later `cargo test` did
(e.g. a dev-dep pulled in a target-specific dep unconditionally) then it would
cause a rebuild of the entire graph.
This class of bug is certainly not the first in a long and storied history of
the backend having multiple points where dependencies are calculated and those
often don't quite agree with one another. The purpose of this rewrite is
twofold:
1. The `Stage` enum in the backend for scheduling work and ensuring that maximum
parallelism is achieved is removed entirely. There is already a function on
`Context` which expresses the dependency between targets (`dep_targets`)
which takes a much finer grain of dependencies into account as well as
already having all the logic for what-depends-on-what. This duplication has
caused numerous problems in the past, an unifying these two will truly grant
maximum parallelism while ensuring that everyone agrees on what their
dependencies are.
2. A large number of locations in the backend have grown to take a (Package,
Target, Profile, Kind) tuple, or some subset of this tuple. In general this
represents a "unit of work" and is much easier to pass around as one
variable, so a `Unit` was introduced which references all of these variables.
Almost the entire backend was altered to take a `Unit` instead of these
variables specifically, typically providing all of the contextual information
necessary for an operation.
A crucial part of this change is the inclusion of `Kind` in a `Unit` to ensure
that everyone *also* agrees on what architecture they're compiling everything
for. There have been many bugs in the past where one part of the backend
determined that a package was built for one architecture and then another part
thought it was built for another. With the inclusion of `Kind` in dependency
management this is handled in a much cleaner fashion as it's only calculated in
one location.
Some other miscellaneous changes made were:
* The `Platform` enumeration has finally been removed. This has been entirely
subsumed by `Kind`.
* The hokey logic for "build this crate once" even though it may be depended on
by both the host/target kinds has been removed. This is now handled in a much
nicer fashion where if there's no target then Kind::Target is just never used,
and multiple requests for a package are just naturally deduplicated.
* There's no longer any need to build up the "requirements" for a package in
terms of what platforms it's compiled for, this now just naturally falls out
of the dependency graph.
* If a build script is overridden then its entire tree of dependencies are not
compiled, not just the build script itself.
* The `threadpool` dependency has been replaced with one on `crossbeam`. The
method of calculating dependencies has quite a few non-static lifetimes and
the scoped threads of `crossbeam` are now used instead of a thread pool.
* Once any thread fails to execute a command work is no longer scheduled unlike
before where some extra pending work may continue to start.
* Many functions used early on, such as `compile` and `build_map` have been
massively simplified by farming out dependency management to
`Context::dep_targets`.
* There is now a new profile to represent running a build script. This is used
to inject dependencies as well as represent that a library depends on running
a build script, not just building it.
This change has currently been tested against cross-compiling Servo to Android
and passes the test suite (which has quite a few corner cases for build scripts
and such), so I'm pretty confident that this refactoring won't have at least too
many regressions!
Unfortunately this flew past on the bots because we didn't built on one of the
10.7 bots, but it looks like libgit2 doesn't currently build on OSX 10.7 so
we'll have to wait until that's fixed to update.
This commit tweaks some metadata here and there to publish Cargo on crates.io.
* License fields are added to Cargo.tomls
* `registry` was renamed to `crates-io`
* API docs for the `cargo` crate are now generated via `make doc`
Cargo will soon support testing the cross compilation capabilities between 32/64
bit Windows, meaning that there's not "one true value" for any of these
environment variables that Cargo is setting (e.g. where to find `cl`, `link`,
etc). Instead, all dependencies have been updated to probe the system (in the
same manner as the compiler) for the tools that they're using.
All of the logic is housed in the `gcc` crate which now exposes a function to
probe the system for a particular tool. The updated crates here then use the
result of this probe to run the various build scripts. This all boils down to
being able to build MSVC targets inside an MSYS shell instead of requiring use
of a MSVC shell (which doesn't allow for simultaneous host builds and cross
builds).
* Download all snapshots and such from static.rust-lang.org (drive-by fix)
* Update dependencies to work and build on MSVC
* ssh2-sys uses `nmake` to build libssh2
* libgit2-sys uses `cmake` to build libgit2
* curl-sys uses `nmake` to build libcurl
* libz-sys uses `nmake` to build zlib
* miniz-sys uses `gcc-rs` to drive `cl.exe` manually
* term updated to pick up windows-specific deps on MSVC
* Updated .travis.install.deps.sh to install MSVC target libraries on Windows,
so the compiler can be used to target MSVC. A 64-bit target is now used by
default as well.
* Updated the configure script and Makefile to set appropriate environment
variables to build Cargo from a MSYS shell with a MSVC target. This is similar
to how Rust bootstraps on MSVC as well.
With this commit Cargo can successfully bootstrap itself on MSVC (use the MSVC
product to build itself again). The tests currently do not pass because
unwinding is not implemented, but soon!
This commit moves Cargo onto 100% stable Rust now that the necessary fs features
have been stabilized. The tests were updated to understand when they're running
with a non-nightly compiler and disable all plugin-related tests. The only major
feature here is that the calls to `fs::set_file_times` were reimplemented
manually in the tests that Cargo has.
Cargo still requires a *nightly compiler* due to the stable features not having
made their way into the stable channel yet, but it will soon work on all of the
stable, beta, and nightly compilers once 1.1 is released!
This commit adds support to allow specifying a custom output directory to Cargo.
First, the `build.target-dir` configuration key is checked, and failing that the
`CARGO_TARGET_DIR` environment variable is checked, and failing that the root
package's directory joined with the directory name "target" is used.
There are a few caveats to switching target directories, however:
* If the target directory is in the current source tree, and the folder name is
not called "target", then Cargo may walk the output directory when determining
whether a tree is fresh.
* If the target directory is not called "target", then Cargo may look inside it
currently for `Cargo.toml` files to learn about local packages.
* Concurrent usage of Cargo will still result in badness (#354), and this is now
exascerbated because many Cargo projects can share the same output directory.
* The top-level crate is not cached for future compilations, so if a crate is
built into directory `foo` and then that crate is later used as a dependency,
it will be recompiled.
The naming limitations can be overcome in time, but for now it greatly
simplifies the crawling routines and shouldn't have much of a negative impact
other than some Cargo runtimes (which can in turn be negated by following the
"target" name convention).
Closes#482
Along the way, this also removes the ability to specify `[[lib]]` targets while
getting warnings. While we're at it remove nearly all unstable features that
Cargo is depending on as well!
Due to libgit2 not supporting HTTP proxies, the custom transport API of the
library must be used to reimplement the HTTP transport with proxy support. The
git2-curl crate implements this transport on top of the curl-rust crate. By
using libcurl we gain all sorts of proxy support for free.
This transport is not used by default, however, as it is not well battle tested
and the architecture is not currently ideal (download the entire repo into
memory on a clone). Only when an HTTP proxy is present is the new transport
used.
The other drawback of git2-curl is that it does not currently support
authentication. If a private git repository is cloned or authentication is
required then it will generate an error instead of correctly asking for
credentials.
Closes#636
Whenever a build script or a plugin depends on a native dynamic library, the
dylib search path will be required to include the path to the generated dynamic
library. Currently Cargo does not augment the search path, causing load errors
or execution errors whenever builds are attempted in this case.
This commit remedies the situation by ensuring that dynamic libraries show up in
dylib search paths. This commit is necessary to fully migrate the remaining
tests from the old build command infrastructure to build scripts.
This commit includes a laundry list of updates and tweaks to reflect the current
API of the registry:
* `registry.host` has been renamed to `registry.index`
* New top-level manifest keys are now accepted:
* `homepage` - url
* `documentation` - url
* `repository` - url
* `description` - a markdown-less blurb
* `license` - string (verified by the registry on upload)
* `keywords` - string array
* `readme` - string pointing at a file
* Authors are now uploaded to the registry
* The upload format to the registry has changed to a body json payload
* Unpacking tarballs respects the executable bit for scripts and such.
* Downloading now follows redirects to go to S3.
* The download URL for a package has changed slightly.
* Verify path dependencies have a version listed when being uploaded
* Rename `upload` to `publish`
* Rename `ops::cargo_upload` to `ops::registry`
* Add a new `registry` package for interoperating with the registry
* Add the ability to modify owners via `cargo owner`
* Add a `readme` key to the manifest, and upload its contents to the registry.
* Add the ability to yank crates and their versions
* When packaging a library, verify that it builds from the packaged source by
unpacking the tarball and simulate running `cargo build` inside of it.
This commit extends the support in cargo's resolver to start resolving packages
with multiple versions as well as package requirements. This is a crucial step
forward when impelmenting the cargo registry as multiple versions will be
uploaded to the registry quite quickly!
This implements a fairly naive solution which should at least help cargo get out
the gates initially. This impelments a depth-first-search of the pacakage
dependency graph with a few sorting heuristics along the way to help out
resolution as it goes along.
Resolution errors will likely improve over time, but this commit does make an
effort to try to get some good error messages right off the bat.
It's looking more likely like plugins will not make it into the stable channel
of Rust, so this commits removes Cargo's personal dependence on the two
plugin-based pieces of functionality it was using:
1. Uses of the `regex!` macro now go through `Regex::new`.
2. Uses of the `docopt!` macro now go through `deriving(Decodable)` instead.
# cargo upload
The cargo-upload command will take the local package and upload it to the
specified registry. The local package is uploaded as a tarball compressed with
gzip under maximum compression. Most of this is done by just delegating to
`cargo package` The host to upload to is specified, in order of priority, by a
command line `--host` flag, the `registry.host` config key, and then the default
registry. The default registry is still `example.com`
The registry itself is still a work in progress, but the general plumbing for a
command such as this would look like:
1. Ensure the local package has been compressed into an archive.
2. Fetch the relevant registry and login token from config files.
3. Ensure all dependencies for a package are listed as coming from the same
registry.
4. Upload the archive to the registry with the login token.
5. The registry will verify the package is under 2MB (configurable).
6. The registry will upload the archive to S3, calculating a checksum in the
process.
7. The registry will add an entry to the registry's index (a git repository).
The entry will include the name of the package, the version uploaded, the
checksum of the upload, and then the list of dependencies (name/version req)
8. The local `cargo upload` command will succeed.
# cargo login
Uploading requires a token from the api server, and this token follows the same
config chain for the host except that there is no fallback. To implement login,
the `cargo login` command is used. With 0 arguments, the command will request
that a site be visited for a login token, and with an argument it will set the
argument as the new login token.
The `util::config` module was modified to allow writing configuration as well as
reading it. The support is a little lacking in that comments are blown away, but
the support is there at least.
# RegistrySource
An implementation of `RegistrySource` has been created (deleting the old
`DummyRegistrySource`). This implementation only needs a URL to be constructed,
and it is assumed that the URL is running an instance of the cargo registry.
## RegistrySource::update
Currently this will unconditionally update the registry's index (a git
repository). Tuning is necessary to prevent updating the index each time (more
coming soon).
## RegistrySource::query
This is called in the resolve phase of cargo. This function is given a
dependency to query for, and the source will simply look into the index to see
if any package with the name is present. If found, the package's index file will
be loaded and parsed into a list of summaries.
The main optimization of this function is to not require the entire registry to
ever be resident in memory. Instead, only necessary packages are loaded into
memory and parsed.
## RegistrySource::download
This is also called during the resolve phase of cargo, but only when a package
has been selected to be built (actually resolved). This phase of the source will
actually download and unpack the tarball for the package.
Currently a configuration file is located in the root of a registry's index
describing the root url to download packages from.
This function is optimized for two different metrics:
1. If a tarball is downloaded, it is not downloaded again. It is assumed that
once a tarball is successfully downloaded it will never change.
2. If the unpacking destination has a `.cargo-ok` file, it is assumed that the
unpacking has already occurred and does not need to happen again.
With these in place, a rebuild should take almost no time at all.
## RegistrySource::get
This function is simply implemented in terms of a PathSource's `get` function by
creating a `PathSource` for all unpacked tarballs as part of the `download`
stage.
## Filesystem layout
There are a few new directories as part of the `.cargo` home folder:
* `.cargo/registry/index/$hostname-$hash` - This is the directory containing the
actual index of the registry. `$hostname` comes from its url, and `$hash` is
the hash of the entire url.
* `.cargo/registry/cache/$hostname-$hash/$pkg-$vers.tar.gz` - This is a
directory used to cache the downloads of packages from the registry.
* `.cargo/registry/src/$hostname-$hash/$pkg-$vers` - This is the location of the
unpacked packages. They will be compiled from this location.
# New Dependencies
Cargo has picked up a new dependency on the `curl-rust` package in order to send
HTTP requests to the registry as well as send HTTP requests to download
tarballs.
This command will assemble the current package into a tarball ready for
uploading to the cargo registry. Currently no further verification is done
beyond packaging the local repository into a tarball, but in the future this
could execute other operations such as api stability tools.
This commit removes all distributed executables except for `cargo`. All
builtin subcommands are implemented through library calls, and the fallback
methods are retained to maintain extensability through new subcommands.
Closes#393
In general relying on external programs is dicey and tricky as they're very
different across systems in both how they're used as well as what versions
you'll find. Instead of binding to the least common denominator of CLI, we can
code against an exact version of libgit2.
This introduces a build-time dependency on cmake which libgit2 requires to build
itself, which is unfortunate, but thankfully it's only a build time dep. The
build process for libgit2 also automatically detects as many system libraries as
possible to use (if available), falling back to bundled versions if not
available. I have currently not figured how to control this, so the link-config
package is used to build libgit2 which requires that pkg-config be installed to
build cargo as well.
Closes#138
The hammer library currently has some shortcomings such as the inability to
document individual options. Additionally our handling with hammer of extra
arguments is dodgy at best currently.
This commit moves the repository to BurntSushi's docopt.rs library which seems
to more feature-complete at this time. Additionally, docopt has the great
benefit of a "document once, use everywhere" documentation strategy.
This migration solves two primary issues:
* Comprehensive and useful CLI documentation
* Gracefully handling flavorful combinations of arguments in odd combinations
Closes#218