Ensure that the dynamic linker search path contains the location of the output
directories for these dependencies when compiling with plugins.
Closes#597
There are some competing concerns when it comes to the output of compiling
dependencies:
* Not capturing anything leads to getting drowned in unrelated output
* Capturing requires coloration be compromised because of the way windows
terminal colors are implemented.
* Path dependencies are often developed in tandem with the rest of a package,
and capturing their output is not always desired.
To address these concerns, cargo previously captured output of dependent
compilations and then re-printed it to the screen if an error occurred. This
patch modifies the behavior to as follows:
* No output is captured. This preserves any coloration rustc provides.
* All dependencies are compiled with `-Awarnings`. This should suppress any
extraneous output from the compiler and it is considered a bug otherwise if
the compiler prints a warnings when `-Awarnings` is specified.
* All *path* dependencies (`path="..."`, overrides, etc) are *not* compiled with
`-Awarnings`. The reason for this is that you are always in control of these
packages and probably want to see warnings anyway.
Closes#490Closes#496
Right now the win64 snapshot builders are failing to produce a snapshot, and
I've managed to track it down to a path length issue. Windows paths have a
maximum of 260 characters, and the characters add up pretty fast for a path
like:
c:\bot\slave\cargo-nightly-win-64
\build\target\x86_64-w64-mingw32
\cargo-integration-test\home\.cargo
\git\checkouts\meta-dep-<hash>\
\$SHA\.git\...
The normal builders aren't failing I presume because `cargo-nightly-win-64` is
longer than `cargo-win64-64` (we must be *right up* against the limit). I've
confirmed that this shortening fixes the tests on the bots.
This command is used to download all dependencies of a package ahead of time to
ensure that no more network communication will be necessary as part of a build.
cc #358
# cargo upload
The cargo-upload command will take the local package and upload it to the
specified registry. The local package is uploaded as a tarball compressed with
gzip under maximum compression. Most of this is done by just delegating to
`cargo package` The host to upload to is specified, in order of priority, by a
command line `--host` flag, the `registry.host` config key, and then the default
registry. The default registry is still `example.com`
The registry itself is still a work in progress, but the general plumbing for a
command such as this would look like:
1. Ensure the local package has been compressed into an archive.
2. Fetch the relevant registry and login token from config files.
3. Ensure all dependencies for a package are listed as coming from the same
registry.
4. Upload the archive to the registry with the login token.
5. The registry will verify the package is under 2MB (configurable).
6. The registry will upload the archive to S3, calculating a checksum in the
process.
7. The registry will add an entry to the registry's index (a git repository).
The entry will include the name of the package, the version uploaded, the
checksum of the upload, and then the list of dependencies (name/version req)
8. The local `cargo upload` command will succeed.
# cargo login
Uploading requires a token from the api server, and this token follows the same
config chain for the host except that there is no fallback. To implement login,
the `cargo login` command is used. With 0 arguments, the command will request
that a site be visited for a login token, and with an argument it will set the
argument as the new login token.
The `util::config` module was modified to allow writing configuration as well as
reading it. The support is a little lacking in that comments are blown away, but
the support is there at least.
# RegistrySource
An implementation of `RegistrySource` has been created (deleting the old
`DummyRegistrySource`). This implementation only needs a URL to be constructed,
and it is assumed that the URL is running an instance of the cargo registry.
## RegistrySource::update
Currently this will unconditionally update the registry's index (a git
repository). Tuning is necessary to prevent updating the index each time (more
coming soon).
## RegistrySource::query
This is called in the resolve phase of cargo. This function is given a
dependency to query for, and the source will simply look into the index to see
if any package with the name is present. If found, the package's index file will
be loaded and parsed into a list of summaries.
The main optimization of this function is to not require the entire registry to
ever be resident in memory. Instead, only necessary packages are loaded into
memory and parsed.
## RegistrySource::download
This is also called during the resolve phase of cargo, but only when a package
has been selected to be built (actually resolved). This phase of the source will
actually download and unpack the tarball for the package.
Currently a configuration file is located in the root of a registry's index
describing the root url to download packages from.
This function is optimized for two different metrics:
1. If a tarball is downloaded, it is not downloaded again. It is assumed that
once a tarball is successfully downloaded it will never change.
2. If the unpacking destination has a `.cargo-ok` file, it is assumed that the
unpacking has already occurred and does not need to happen again.
With these in place, a rebuild should take almost no time at all.
## RegistrySource::get
This function is simply implemented in terms of a PathSource's `get` function by
creating a `PathSource` for all unpacked tarballs as part of the `download`
stage.
## Filesystem layout
There are a few new directories as part of the `.cargo` home folder:
* `.cargo/registry/index/$hostname-$hash` - This is the directory containing the
actual index of the registry. `$hostname` comes from its url, and `$hash` is
the hash of the entire url.
* `.cargo/registry/cache/$hostname-$hash/$pkg-$vers.tar.gz` - This is a
directory used to cache the downloads of packages from the registry.
* `.cargo/registry/src/$hostname-$hash/$pkg-$vers` - This is the location of the
unpacked packages. They will be compiled from this location.
# New Dependencies
Cargo has picked up a new dependency on the `curl-rust` package in order to send
HTTP requests to the registry as well as send HTTP requests to download
tarballs.
The CLI has started to have differences on windows than on other systems, and
instead of coding against multiple CLI versions, instead just rewrite everything
to use libgit2.
Travis is showing intermittent failures in
`test_cargo_compile_path_deps::path_dep_build_cmd` and
`test_cargo_compile_git_deps::git_dep_build_cmd` (added in #561 and #563).
I haven't managed to reproduce these failures locally, but I suspect the tests
are timing-sensitive. This tries to guarantee that the timestamps during the
two operations can't be the same.
`list_files_git` assumes that the package is at the root of the git repository. This breaks when trying to list files for a package in a subdirectory of a repo, or in other cases, e.g. if the user's home directory is a git repo.
Previously this was calling `.is_file()` on a relative path. This would fail if the path was not relative to the current working directory, for example when listing files in a path dependency.
When cloning a remote repository, the default behavior of libgit2 is to only
update the local ref that's actually checked out, when we actually would prefer
to update all refs of the remote repo. This removes a call to clone() to a
init_bare() + fetch() which should update all refs accordingly
Closes#565
This primarily blows away all *submodules* as well, which sometimes can be quite
large and take some time to update. Instead, re-use an existing checkout, just
reset it to the right revision if possible.
Also, move the submodule update step to occur unconditionally to account for
corrupt submodule checkouts or interrupted downloads. This update step should be
much faster than `git submodule update` because we're using libgit2, so yay!
The previous logic for recompiling any dependency was almost entirely based on
the mtimes of the relevant input files. This isn't quite what's desired for git
and registry dependencies because the mtime could be fluctuating while the files
aren't changing. For example:
1. Project A checks out git repo C at revision C1
2. Project A builds, records mtimes of C
3. Project B checks out git repo C at revision C2
4. Project B builds, records new mtimes of C
5. Project A is rebuilt, rebuilding C b/c mtimes are different
In step 5 here C should not be rebuilt because the revision didn't actually
change.
This commit alters git/registry dependencies to completely bypass the --dep-info
emitted and only rely on the git/registry source to inform what the fingerprint
is. This is the revision/version, respectively, and should be all that's
necessary to track changes to the repo and trigger a rebuild.
This primarily blows away all *submodules* as well, which sometimes can be quite
large and take some time to update. Instead, re-use an existing checkout, just
reset it to the right revision if possible.
Also, move the submodule update step to occur unconditionally to account for
corrupt submodule checkouts or interrupted downloads. This update step should be
much faster than `git submodule update` because we're using libgit2, so yay!
The previous logic for recompiling any dependency was almost entirely based on
the mtimes of the relevant input files. This isn't quite what's desired for git
and registry dependencies because the mtime could be fluctuating while the files
aren't changing. For example:
1. Project A checks out git repo C at revision C1
2. Project A builds, records mtimes of C
3. Project B checks out git repo C at revision C2
4. Project B builds, records new mtimes of C
5. Project A is rebuilt, rebuilding C b/c mtimes are different
In step 5 here C should not be rebuilt because the revision didn't actually
change.
This commit alters git/registry dependencies to completely bypass the --dep-info
emitted and only rely on the git/registry source to inform what the fingerprint
is. This is the revision/version, respectively, and should be all that's
necessary to track changes to the repo and trigger a rebuild.
This reverts a commit that worked around an overly restrictive lifetime bound on the `Writer` impl for `Box<Writer>` in libstd. Now that this restriction has been lifted, the workaround can be reverted.
This commit updates git2-rs to get the implementation of the authentication
callback in libgit2. Additionally this specifies the callback for whenever we're
cloning into the database or updating submodules.
Currently cargo will *not* ask for user input, but rather require you to have
authentication configured in git through some other means. There are currently
two primary methods of doing so:
1. Any SSH key in the local ssh-agent will be used for authentication with SSH
repositories.
2. The `credential.helper` interface (as specified by gitcredential(7)) has been
implemented in git2-rs to allow for picking up of storage of passwords in the
local git cache or keychain.
If these two methods fail, then there will likely be an authentication failure.
Interactive prompts for authentication have not been implemented as there is no
method to currently enter your password into the terminal silently.
A consequence of this commit is that cargo now depends on libssh2. A package was
created to create a static copy of libssh2, and this is now linked into cargo by
default.
It turned out that just building libssh2 was quite a beast in and of itself on
windows. The primary stickler point is that on the current release, 1.4.3,
libssh2 requires openssl on windows. At this time I don't want to pick up a
dependency on openssl for windows, and it turned out that the unreleased 1.4.4
version has a new backend for windows not based on openssl, but rather windows's
cryptography API.
The current bundled version of libssh2 is 1.4.4 with some light modifications to
actually build on windows (wow that was hard). All in all, we're now statically
linking to libssh 1.4.4 (not a runtime dependency).
Closes#493
Overrides are only queried by name, and it's possible for multiple to show up,
and just pick the first one.
Closes#461
r? @wycats -- I'm not convinced this is correct
This adds a command-line --no-git option to disable this behavior, as well as
adding a global config section for `git = false`. While I was at it I write some
documentation for the configuration format that cargo uses.
This adds a command-line --no-git option to disable this behavior, as well as
adding a global config section for `git = false`. While I was at it I write some
documentation for the configuration format that cargo uses.
