This means that if a project has a file with a space in the name it will
properly have its freshness calculated as opposed to always having it as a
candidate to be rebuilt.
Closes#648
Now that we have selective testing, this no longer makes any sense and all
queries to the path layout need to be based on the package being queried for.
This removes the primary flag from the Context, and requires that the `layout`
method have a local Package available
cc servo/servo#3580
This means that if a project has a file with a space in the name it will
properly have its freshness calculated as opposed to always having it as a
candidate to be rebuilt.
Closes#648
Development dependencies can never be the root of a cycle because nothing
depends on a development dependency, so there's no need to track the start of a
cycle at the edge going out to a development dependency.
If a cycle is later detected, it will still be reported.
Currently whenever a dev-dep is brought in to the build process the entire
library is rebuilt, but this is just unnecessary recompilation because the
library *can't* depend on the dev-dep.
This commit refines the dependency graph so the lib stage only depends on
transitive dependencies (non-dev-deps), and a new stage for tests was added
which depends on the packages libraries *and* the dev-deps. This way only the
test are rebuilt when dev-deps change, not libraries.
If the host system had $HOME/.cargo/config which configured a user name/email
then this test would fail because those would be prioritized over $USER.
Closes#514
To maintain consistency with `cargo {build,test,bench,clean}` the `update`
subcommand now takes a specific package via the `-p` argument instead of as a
positional argument.
This functionality allows running tests and benchmarks on any upstream
dependencies in the dependency graph. This is most useful for path sources all
developed in tandem (see Servo for instance).
In terms of built artifacts, this will actually preserve as many artifacts as
possible. That means that if you test a low-level dependency with the high-level
artifacts already built, the high-level artifacts will not get removed. This
means that it's possible to accidentally have a low-level dependency to depend
on a higher level one just because it's lib is picked up via -L, but this is
generally a necessary evil to get testing to not rebuild packages too often.
Closes#483
This adds a new argument to `cargo clean` which will enable selectively cleaning
particular packages. The command only cleans the package specified, no other
(not the dependencies of the package).
cc #537
If the host system had $HOME/.cargo/config which configured a user name/email
then this test would fail because those would be prioritized over $USER.
Closes#514
This commit adds a flag, --precise, to cargo update. This flag is used to update
a dependency to precisely an exact revision (or branch) as part of an update
step. For git repositories the argument is some form of reference, while
registry packages this will be a version number.
The flag --precise forces a non-aggressive update and will fail if the
--aggresive flag is specified.
Closes#484
Ensure that the dynamic linker search path contains the location of the output
directories for these dependencies when compiling with plugins.
Closes#597
As described in #613, this commit switches the semantics of `cargo update foo`
to updating *only* `foo`, not any of its dependencies. A new flag,
`--aggressive` was added to restore the old behavior.
The behavior of attempting to only unlock `foo`, and then if resolve fails
unlock all dependencies of `foo` is unimplemented as it's not super relevant
right now when the majority of dependencies are git dependencies and resolution
cannot fail for version-related reasons.
Closes#613
Ensure that the dynamic linker search path contains the location of the output
directories for these dependencies when compiling with plugins.
Closes#597
As described in #613, this commit switches the semantics of `cargo update foo`
to updating *only* `foo`, not any of its dependencies. A new flag,
`--aggressive` was added to restore the old behavior.
The behavior of attempting to only unlock `foo`, and then if resolve fails
unlock all dependencies of `foo` is unimplemented as it's not super relevant
right now when the majority of dependencies are git dependencies and resolution
cannot fail for version-related reasons.
Closes#613
There are some competing concerns when it comes to the output of compiling
dependencies:
* Not capturing anything leads to getting drowned in unrelated output
* Capturing requires coloration be compromised because of the way windows
terminal colors are implemented.
* Path dependencies are often developed in tandem with the rest of a package,
and capturing their output is not always desired.
To address these concerns, cargo previously captured output of dependent
compilations and then re-printed it to the screen if an error occurred. This
patch modifies the behavior to as follows:
* No output is captured. This preserves any coloration rustc provides.
* All dependencies are compiled with `-Awarnings`. This should suppress any
extraneous output from the compiler and it is considered a bug otherwise if
the compiler prints a warnings when `-Awarnings` is specified.
* All *path* dependencies (`path="..."`, overrides, etc) are *not* compiled with
`-Awarnings`. The reason for this is that you are always in control of these
packages and probably want to see warnings anyway.
Closes#490Closes#496
Right now the win64 snapshot builders are failing to produce a snapshot, and
I've managed to track it down to a path length issue. Windows paths have a
maximum of 260 characters, and the characters add up pretty fast for a path
like:
c:\bot\slave\cargo-nightly-win-64
\build\target\x86_64-w64-mingw32
\cargo-integration-test\home\.cargo
\git\checkouts\meta-dep-<hash>\
\$SHA\.git\...
The normal builders aren't failing I presume because `cargo-nightly-win-64` is
longer than `cargo-win64-64` (we must be *right up* against the limit). I've
confirmed that this shortening fixes the tests on the bots.
This command is used to download all dependencies of a package ahead of time to
ensure that no more network communication will be necessary as part of a build.
cc #358
# cargo upload
The cargo-upload command will take the local package and upload it to the
specified registry. The local package is uploaded as a tarball compressed with
gzip under maximum compression. Most of this is done by just delegating to
`cargo package` The host to upload to is specified, in order of priority, by a
command line `--host` flag, the `registry.host` config key, and then the default
registry. The default registry is still `example.com`
The registry itself is still a work in progress, but the general plumbing for a
command such as this would look like:
1. Ensure the local package has been compressed into an archive.
2. Fetch the relevant registry and login token from config files.
3. Ensure all dependencies for a package are listed as coming from the same
registry.
4. Upload the archive to the registry with the login token.
5. The registry will verify the package is under 2MB (configurable).
6. The registry will upload the archive to S3, calculating a checksum in the
process.
7. The registry will add an entry to the registry's index (a git repository).
The entry will include the name of the package, the version uploaded, the
checksum of the upload, and then the list of dependencies (name/version req)
8. The local `cargo upload` command will succeed.
# cargo login
Uploading requires a token from the api server, and this token follows the same
config chain for the host except that there is no fallback. To implement login,
the `cargo login` command is used. With 0 arguments, the command will request
that a site be visited for a login token, and with an argument it will set the
argument as the new login token.
The `util::config` module was modified to allow writing configuration as well as
reading it. The support is a little lacking in that comments are blown away, but
the support is there at least.
# RegistrySource
An implementation of `RegistrySource` has been created (deleting the old
`DummyRegistrySource`). This implementation only needs a URL to be constructed,
and it is assumed that the URL is running an instance of the cargo registry.
## RegistrySource::update
Currently this will unconditionally update the registry's index (a git
repository). Tuning is necessary to prevent updating the index each time (more
coming soon).
## RegistrySource::query
This is called in the resolve phase of cargo. This function is given a
dependency to query for, and the source will simply look into the index to see
if any package with the name is present. If found, the package's index file will
be loaded and parsed into a list of summaries.
The main optimization of this function is to not require the entire registry to
ever be resident in memory. Instead, only necessary packages are loaded into
memory and parsed.
## RegistrySource::download
This is also called during the resolve phase of cargo, but only when a package
has been selected to be built (actually resolved). This phase of the source will
actually download and unpack the tarball for the package.
Currently a configuration file is located in the root of a registry's index
describing the root url to download packages from.
This function is optimized for two different metrics:
1. If a tarball is downloaded, it is not downloaded again. It is assumed that
once a tarball is successfully downloaded it will never change.
2. If the unpacking destination has a `.cargo-ok` file, it is assumed that the
unpacking has already occurred and does not need to happen again.
With these in place, a rebuild should take almost no time at all.
## RegistrySource::get
This function is simply implemented in terms of a PathSource's `get` function by
creating a `PathSource` for all unpacked tarballs as part of the `download`
stage.
## Filesystem layout
There are a few new directories as part of the `.cargo` home folder:
* `.cargo/registry/index/$hostname-$hash` - This is the directory containing the
actual index of the registry. `$hostname` comes from its url, and `$hash` is
the hash of the entire url.
* `.cargo/registry/cache/$hostname-$hash/$pkg-$vers.tar.gz` - This is a
directory used to cache the downloads of packages from the registry.
* `.cargo/registry/src/$hostname-$hash/$pkg-$vers` - This is the location of the
unpacked packages. They will be compiled from this location.
# New Dependencies
Cargo has picked up a new dependency on the `curl-rust` package in order to send
HTTP requests to the registry as well as send HTTP requests to download
tarballs.
The CLI has started to have differences on windows than on other systems, and
instead of coding against multiple CLI versions, instead just rewrite everything
to use libgit2.
Travis is showing intermittent failures in
`test_cargo_compile_path_deps::path_dep_build_cmd` and
`test_cargo_compile_git_deps::git_dep_build_cmd` (added in #561 and #563).
I haven't managed to reproduce these failures locally, but I suspect the tests
are timing-sensitive. This tries to guarantee that the timestamps during the
two operations can't be the same.
`list_files_git` assumes that the package is at the root of the git repository. This breaks when trying to list files for a package in a subdirectory of a repo, or in other cases, e.g. if the user's home directory is a git repo.
Previously this was calling `.is_file()` on a relative path. This would fail if the path was not relative to the current working directory, for example when listing files in a path dependency.
When cloning a remote repository, the default behavior of libgit2 is to only
update the local ref that's actually checked out, when we actually would prefer
to update all refs of the remote repo. This removes a call to clone() to a
init_bare() + fetch() which should update all refs accordingly
Closes#565
This primarily blows away all *submodules* as well, which sometimes can be quite
large and take some time to update. Instead, re-use an existing checkout, just
reset it to the right revision if possible.
Also, move the submodule update step to occur unconditionally to account for
corrupt submodule checkouts or interrupted downloads. This update step should be
much faster than `git submodule update` because we're using libgit2, so yay!
The previous logic for recompiling any dependency was almost entirely based on
the mtimes of the relevant input files. This isn't quite what's desired for git
and registry dependencies because the mtime could be fluctuating while the files
aren't changing. For example:
1. Project A checks out git repo C at revision C1
2. Project A builds, records mtimes of C
3. Project B checks out git repo C at revision C2
4. Project B builds, records new mtimes of C
5. Project A is rebuilt, rebuilding C b/c mtimes are different
In step 5 here C should not be rebuilt because the revision didn't actually
change.
This commit alters git/registry dependencies to completely bypass the --dep-info
emitted and only rely on the git/registry source to inform what the fingerprint
is. This is the revision/version, respectively, and should be all that's
necessary to track changes to the repo and trigger a rebuild.
This primarily blows away all *submodules* as well, which sometimes can be quite
large and take some time to update. Instead, re-use an existing checkout, just
reset it to the right revision if possible.
Also, move the submodule update step to occur unconditionally to account for
corrupt submodule checkouts or interrupted downloads. This update step should be
much faster than `git submodule update` because we're using libgit2, so yay!
The previous logic for recompiling any dependency was almost entirely based on
the mtimes of the relevant input files. This isn't quite what's desired for git
and registry dependencies because the mtime could be fluctuating while the files
aren't changing. For example:
1. Project A checks out git repo C at revision C1
2. Project A builds, records mtimes of C
3. Project B checks out git repo C at revision C2
4. Project B builds, records new mtimes of C
5. Project A is rebuilt, rebuilding C b/c mtimes are different
In step 5 here C should not be rebuilt because the revision didn't actually
change.
This commit alters git/registry dependencies to completely bypass the --dep-info
emitted and only rely on the git/registry source to inform what the fingerprint
is. This is the revision/version, respectively, and should be all that's
necessary to track changes to the repo and trigger a rebuild.
This reverts a commit that worked around an overly restrictive lifetime bound on the `Writer` impl for `Box<Writer>` in libstd. Now that this restriction has been lifted, the workaround can be reverted.
This commit updates git2-rs to get the implementation of the authentication
callback in libgit2. Additionally this specifies the callback for whenever we're
cloning into the database or updating submodules.
Currently cargo will *not* ask for user input, but rather require you to have
authentication configured in git through some other means. There are currently
two primary methods of doing so:
1. Any SSH key in the local ssh-agent will be used for authentication with SSH
repositories.
2. The `credential.helper` interface (as specified by gitcredential(7)) has been
implemented in git2-rs to allow for picking up of storage of passwords in the
local git cache or keychain.
If these two methods fail, then there will likely be an authentication failure.
Interactive prompts for authentication have not been implemented as there is no
method to currently enter your password into the terminal silently.
A consequence of this commit is that cargo now depends on libssh2. A package was
created to create a static copy of libssh2, and this is now linked into cargo by
default.
It turned out that just building libssh2 was quite a beast in and of itself on
windows. The primary stickler point is that on the current release, 1.4.3,
libssh2 requires openssl on windows. At this time I don't want to pick up a
dependency on openssl for windows, and it turned out that the unreleased 1.4.4
version has a new backend for windows not based on openssl, but rather windows's
cryptography API.
The current bundled version of libssh2 is 1.4.4 with some light modifications to
actually build on windows (wow that was hard). All in all, we're now statically
linking to libssh 1.4.4 (not a runtime dependency).
Closes#493
Overrides are only queried by name, and it's possible for multiple to show up,
and just pick the first one.
Closes#461
r? @wycats -- I'm not convinced this is correct
This adds a command-line --no-git option to disable this behavior, as well as
adding a global config section for `git = false`. While I was at it I write some
documentation for the configuration format that cargo uses.
This adds a command-line --no-git option to disable this behavior, as well as
adding a global config section for `git = false`. While I was at it I write some
documentation for the configuration format that cargo uses.
It's quite annoying if you update a dependency, but it takes you awhile to get
the dependency building. Previously the dependency graph would have to be
updated each time because the lockfile was only written *after* a successful
build.
Other tools like `cargo generate-lockfile` will already generate a lockfile at
any time, so just make it easier by moving it up in the compilation process.
This test has been flaky on the bots for quite some time now, and the cause has
now been discovered. The root cause of the failure is that the execve for the
`cargo --list` command was failing with ETXTBUSY. In querying the manpage, this
means:
Executable was open for writing by one or more processes.
This error can be explained by the following trace:
1. Thread A, running the `cargo --list` test, opens the destination executable
for writing because it's copying the current executable into a different
location.
2. Thread B, some other test, forks the process. The file descriptor of the
destination executable of thread A is now duplicated in this process.
3. Thread A closes all files and such, and then goes to fork/exec
`cargo --list`.
4. Thread B has not had time to close all its descriptors. so it still has the
executable open for writing, causing the `execve` of thread A to fail.
This commit just removes these tested portions of the test, only testing that
cargo probes PATH.
Relative paths are now considered relative to the directory containing the
`.cargo/config` file specifying the relative path.
I also merge ConfigValueValue into ConfigValue by moving the Path directly next
to the String that defined it so we can track which strings came from which
paths.
Closes#458
This is often useful for picking up things like headers files from `*-sys`
packages when they had to compile locally (or perhaps for pkg-config).
Closes#449
This is often useful for picking up things like headers files from `*-sys`
packages when they had to compile locally (or perhaps for pkg-config).
Closes#449
Relative paths are now considered relative to the directory containing the
`.cargo/config` file specifying the relative path.
I also merge ConfigValueValue into ConfigValue by moving the Path directly next
to the String that defined it so we can track which strings came from which
paths.
Closes#458
This test has been flaky on the bots for quite some time now, and the cause has
now been discovered. The root cause of the failure is that the execve for the
`cargo --list` command was failing with ETXTBUSY. In querying the manpage, this
means:
Executable was open for writing by one or more processes.
This error can be explained by the following trace:
1. Thread A, running the `cargo --list` test, opens the destination executable
for writing because it's copying the current executable into a different
location.
2. Thread B, some other test, forks the process. The file descriptor of the
destination executable of thread A is now duplicated in this process.
3. Thread A closes all files and such, and then goes to fork/exec
`cargo --list`.
4. Thread B has not had time to close all its descriptors. so it still has the
executable open for writing, causing the `execve` of thread A to fail.
This commit just removes these tested portions of the test, only testing that
cargo probes PATH.
It turned out most of the methods in libgit2 don't actually require a Signature
structure, they're all mostly optional. This commit updates to this version of
libgit2 where the arguments are all optional.
This generates errors for malformed semver versions during the decoding process
rather than later in the convertion to a package id. This also cuts down on the
large number of derived traits to only what's necessary.
Closes#54
This command will assemble the current package into a tarball ready for
uploading to the cargo registry. Currently no further verification is done
beyond packaging the local repository into a tarball, but in the future this
could execute other operations such as api stability tools.
This command will assemble the current package into a tarball ready for
uploading to the cargo registry. Currently no further verification is done
beyond packaging the local repository into a tarball, but in the future this
could execute other operations such as api stability tools.
This option is used to disable the --test flag to rustc for a test or benchmark
target in order to signal that the binary already knows how to run the testing
infrastructure.
The test/benchmark is still compiled and run as usual, and the exit code is
expected to reflect the result of the test/benchmark.
Closes#431
This commit removes all distributed executables except for `cargo`. All
builtin subcommands are implemented through library calls, and the fallback
methods are retained to maintain extensability through new subcommands.
Closes#393
This generates errors for malformed semver versions during the decoding process
rather than later in the convertion to a package id. This also cuts down on the
large number of derived traits to only what's necessary.
Closes#54
All subprocesses will now be invoked with CARGO_MANIFEST_DIR pointing at the
root of the source directory that they are working on (compiling). This commit
also reorganizes the version environment variables to use the new
infrastructure.
Closes#433
Previously, the Git code avoided cloning a local copy of the local
database for a repository if the repo already existed (and instead tried
to fetch). However, fetching does not sync and reinitialize submodules,
which would require more work.
To avoid this problem, and possibly other subtle updating issues in the
future, the code now wipes the local clone whenever it detects that the
HEAD of a particular branch has changed. This avoids subtle
inconsistencies between fresh clones and updates.
Note: This only affects local-to-local clones. It does not initiate any
new network activity. It may require some additional disk usage, but
only when a remote fetch that was already happening discovers new
commits on the checked-out branch.
Previously `cargo update foo` would only update the package `foo` if it were a
direct dependency of the local package. This meant that to update a transitive
dependency you would have to update the top-level dependency.
This commit adds the ability to update any dependency by name, regardless of
where it is in the dependency graph. This commit is a bit lossy in terms of
behavior. We are guaranteed that the set of immediate dependencies for any one
package have unique names, but not for the entire package graph. This means that
when you invoke `cargo update foo` it could possibly update two packages named
`foo`.
I believe this behavior to be acceptable for now and we can add a more stringent
update syntax later (something like `cargo update --namespace foo bar`). I
believe we'll always want this CLI usage, however.
The original choice for completely destroying $OUT_DIR was to focus on
repeatable builds to ensure that stale items in $OUT_DIR don't affect later
compilations. This commit moves this responsibility from cargo to the build
command itself.
Build commands are now responsible for cleaning out old artifacts and ensuring
that when invoked the state of $OUT_DIR is as it would be if it started with an
empty $OUT_DIR. This will allow for very long build commands based on `make` to
have a much faster incremental build time as they won't have to rebuild
everything when updated.
This can cause non-repeatable builds with build commands if stale artifacts are
never removed (but still used in the rust compilation step), but this will now
be considered a bug in the build command rather than for cargo itself.
Closes#382
This key is used to help determine the set of input files for a package. The
normal method (via walking or git ls-files) is filtered via all the patterns
provided in `exclude` of the project section.
The toml key is a string array corresponding to a set of glob patterns according
to the syntax of libglob (provided in the standard distribution). The set of
files normally found will be filtered by each pattern, and if any pattern
matches then the path is excluded.
This will later on be used for `cargo package` when generating a tarball to get
uploaded.
When testing the freshness of a build command, the fingerprint of an entire
package is generated. The current method of fingerprinting a path source is to
just stat() the entire tree and look at mtimes. This works fine when the build
command places *all* output in the build directory.
Some systems, like autotools, place *most* output in the build directory and
some output in the source directory, however (see spidermonkey). In this case
the coarse-grained "consider all files in a directory as input" is too painful
for the build command as the package will forever be rebuilt.
This commit adds support for filtering the list of files in a directory
considered to be part of a package by using `git ls-files`. This git-specific
logic can be extended to other VCSs when cargo grows support for them.
Closes#379
This commit changes the hash of Profile to only take into account
flags/variables that affect the actual output file itself (as opposed to its
location), and then changes cargo {test, build, doc} to all use the same
directory of output (in order to share deps).
This will cause a `cargo build` to remove all of the tests generated by `cargo
test`, but it speeds up the cycle of `cargo test` followed by a `cargo build` by
not needing to rebuild all dependencies.
Additionally, `cargo bench` now shares the same directory as
`cargo build --release` for the same reasons as above.
Closes#348
Previously an invocation of `cargo build` would generate a lockfile *without*
dev dependencies, but an invocation of `cargo test` would generate a lockfile
*with* dependencies. This causes odd problems and diffs with the lockfile.
This commit switches the resolve-to-be-a-lockfile to using all dependencies of a
package, while the resolve-to-be-compiled continues to use just the dependencies
needed for the current build.
Previously an invocation of `cargo build` would generate a lockfile *without*
dev dependencies, but an invocation of `cargo test` would generate a lockfile
*with* dependencies. This causes odd problems and diffs with the lockfile.
This commit switches the resolve-to-be-a-lockfile to using all dependencies of a
package, while the resolve-to-be-compiled continues to use just the dependencies
needed for the current build.
The syntax was originally chosen to perhaps allow multiple libraries in the future, but that is less and less likely to happen at this point. This commit deprecates the now-misleading `[[lib]]` in favor of `[lib]` to indicate that only one can be present.
This does not start allowing `[bin]` or `[example]` and such as I felt that it was too many ways to specify what's essentially the same thing. This also as a bonus allows a top-level `bin = []` to completely opt-out of binary inference (as well as for tests and examples).
Closes#327
We observed that after updating the git repo in a test, unless we waited
for 1s, we got intermittent cases where a subsequent update of the git
source would not pick up the changes.
We observed this both with the git CLI and the branch using libgit2.
There may be a better solution, but we did not find it.
This commit starts to accept `[lib]` as a single library per package. It also
allows opting-out of binaries/tests/examples with a top level `foo = []` rather
than taking an empty array to mean that inference should be enabled.
cc #327
Only print a failure if the command didn't have an exit status. Also, clarify
the failure message in the case that an exit was detected but it was not a
successful status.
Closes#350
This adds a `doctest = false` option to the `Cargo.toml` config for a target to
disable doc tests for the target. Notably `test = false` does not disable this
as it is separately toggleable.
Closes#353
These were previously just run by executing the raw binary, but this didn't
ensure that any of the necessary paths were in the dylib search path for the
host platform.
This commit enhances the return type of `compile_targets` to have information
about the result of compilation, along with the ability to spawn correctly
configured processes.
This primarily fixes `cargo test` and `cargo run` whenever dynamic dependencies
are involved, but it also fixes the two commands whenever there's a native
dynamic dependency involved as well.
Whenever `cargo test` is run and a testable library target is available, the doc
tests will be run. This can be opted out of with `test = false` as usual.
This is currently not super useful due to rust-lang/rust#16157, but I expect
that to be merged soon. In the meantime examples will need to `extern crate foo`
explicitly.
Closes#334
This removes the check in the compilation phase, but adds an error if you're
documenting a library and a binary with the same name (as the rustdoc output
would conflict).
Closes#318
Whenever `cargo test` is run and a testable library target is available, the doc
tests will be run. This can be opted out of with `test = false` as usual.
This is currently not super useful due to rust-lang/rust#16157, but I expect
that to be merged soon. In the meantime examples will need to `extern crate foo`
explicitly.
I customized my git repo template to default to the standard pre-commit
hook that checks for e.g. trailing whitespace and refuses to commit if
any trailing whitespace was found. This causes some of cargo's tests to
fail.
To be independent of the user's git template, create the repo for the
test without using any, by specifying an empty template.
This removes the check in the compilation phase, but adds an error if you're
documenting a library and a binary with the same name (as the rustdoc output
would conflict).
On windows, git will check out files with different line endings causing the
same serialized resolve to be resolved slightly differently. Instead of
comparing contents, this commit alters by testing whether the decoded resolve is
equivalent to the to-be-written resolve and only aborts writing if the two are
equal.
This commit refines the granularity of fingerprints from packages to targets,
building on the existing infrastructure. All invocations of rustc now include
the --dep-info flag which will emit makefile-like dependency information. This
information is then parsed later on to determine if all the files are fresh.
The purpose of this commit is to refine the set of files which indicate that a
target needs to be re-built. Before this commit if you modified a test it would
rebuild the entire library, but after this commit it only rebuilds the relevant
test.
Closes#214Closes#289
This commit refactors all related infrastructure for the cargo_rustc module to
move the granularity of fingerprints from packages to targets. This involved
heavily modifying the `JobQueue` structure to understand a finer-grained target
for dirtiness propagation and dependency management, and then dealing with
fallout in the main module.
The fingerprint module has been refactored to support a per-target fingerprint.
A fallout of this change is that each build step has its own fingerprint,
including the custom build command step. This will be implemented in a future
commit.
As fallout of this reorganization, we are now exploiting the maximal parallelism
within packages themselves whereas before we were only partially parallelizing.
No new features were added as part of this commit, so I just ensured that all
the tests ran.
The existing rustdoc behavior for working with an already-present directory is
good enough, so there's no need for cargo to help out by blowing things away.
This also makes the copy-back portion easier by just not having it.
Closes#284
I've changed the error message that appears when a dependency can't be located. With my changes, mismatches between Cargo.toml dependency names and actual dependency package names are (slightly) more apparent. This doesn't change the error that appears when a dependency has an invalid source.
Example:
```toml
# Cargo.toml
...
[dependencies.foo]
path = "foo"
```
```toml
# foo/Cargo.toml
[package]
name = "not_foo"
```
Previously:
```
$ cargo build
No package named Dependency { name: foo, namespace: file:/home/michael/cargo_test, req: *, transitive: true } found
```
Now:
```
$ cargo build
No package named `foo` found (required by `bar`).
Location searched: /some/folder/not_foo
Version required: *
```
In verbose mode I think it would be also nice to print a list of packages that Cargo *does* know about. Something like: "Known packages amongst dependencies: not_foo, etc"
The existing rustdoc behavior for working with an already-present directory is
good enough, so there's no need for cargo to help out by blowing things away.
This also makes the copy-back portion easier by just not having it.
Closes#284
When copying files over from the old root to the new root on a fresh
compilation, care must be taken to preserve the correct plugin/host version of
each dependency. The previous code copied back over at most one library, but
this commit fixes this behavior by copying over all targets necessary for
compilation.
* Make sure plugins link to plugin dependencies, not target dependencies.
Previously the --extern flag was being passed incorrectly but the dependency
was being picked up by -L anyway.
* Fix a type and actually put the host dep directory into LD_LIBRARY_PATH, not
the target directory. A test was added for this change.
The hammer library currently has some shortcomings such as the inability to
document individual options. Additionally our handling with hammer of extra
arguments is dodgy at best currently.
This commit moves the repository to BurntSushi's docopt.rs library which seems
to more feature-complete at this time. Additionally, docopt has the great
benefit of a "document once, use everywhere" documentation strategy.
This migration solves two primary issues:
* Comprehensive and useful CLI documentation
* Gracefully handling flavorful combinations of arguments in odd combinations
Closes#218
The fingerprinting code was erroneously using all sources from a manifest when
calculating the fingerprint which meant that sources not yet downloaded were
attempted to be fingerprinted.
The correct source to fingerprint is located in the SourceId field of the
resolved PackageId.
Closes#259
At runtime rustc will dlopen() plugins, and if plugins have dynamic dependencies
they're likely to be in target/deps, so we need to make sure that directory is
in the right search path.
This previously relied on the test itself printing failure, but this is not
always the case for test which, for example, segfault. Instead this ensures that
*something* is always printed when a test fails, normally a short blurb about
what executable failed to run.
This previously relied on the test itself printing failure, but this is not
always the case for test which, for example, segfault. Instead this ensures that
*something* is always printed when a test fails, normally a short blurb about
what executable failed to run.
Sometimes it's a pain to match the *exact* output of a process, so this adds a
placeholder where `[..]` indicates "any more input on this line". It can in
theory be used multiple times in a input string.
The filename of the test for a dylib wasn't being calcuated correctly, so when
freshness was copying data over it ended up copying the same file twice.
The test and bin executables weren't getting the correct --extern flags when
tested and built, so the names were conflicting. This passes --extern for the
local crate to ensure the right crate is picked up.
This command is used to create a new cargo repository at a destination that
previously does not exist. A separate command, cargo-init, will be implemented
to initialize an already-existing repository.
cc #21
This is currently rebased on #238
This command is used to create a new cargo repository at a destination that
previously does not exist. A separate command, cargo-init, will be implemented
to initialize an already-existing repository.
cc #21
This ends up serving the same purpose, but a critical change is that it
canonicalizes the relevant git url (if one is used) to ensure that the same
package from two slightly different locations is always built the same way.
Sadly I'm not quite sure how to add a test for this as it involves using remote
git urls which are unusable during tests.
In order to ensure there are no stale artifacts as part of a build, this commit
houses all output of native build commands in their own directories. Each
directory is on a per-package basis, and the output is preserved if the package
is fresh or discarded if it is not.
This does not remove the DEPS_DIR environment variable, it just wires it to the
same value as OUT_DIR.
And aftewards selectively move them back into place if they're fresh. This
prevents stale files from showing up from old builds.
This currently breaks anyone build `build=` scripts, the fix is coming in the
next commit.
Closes#205
TOML recently grew multiline string literals as well as literal strings with no
escaping, which can be used instead of the hokey escape_path() function.
This adds a new .cargo/config option which allows configuring the ar and linker
tools that rustc invokes. This should aid in any cross-compilation attempts.
It's not unreasonable to have unittests in a separate submodule of the
crate (being called `test` or `tests`), and having them in their own
file can be very sensible. Thus, the `src/test.rs` implicit default is
likely to trip up some perfectly reasonable use-cases. There's already
the `tests/...` default, so repairing a codebase after this removal is
just moving `src/test.rs` to `tests/whatever_name_you_want.rs`.
Closes#187.
It's not unreasonable to have unittests in a separate submodule of the
crate (being called `test` or `tests`), and having them in their own
file can be very sensible. Thus, the `src/test.rs` implicit default is
likely to trip up some perfectly reasonable use-cases. There's already
the `tests/...` default, so repairing a codebase after this removal is
just moving `src/test.rs` to `tests/whatever_name_you_want.rs`.
Closes#187.
This allows the dependency queue to properly handle packages with the same
name but from different sources.
A test was added which exercieses this functionality by depending on two
different revs of the same git repo.
This allows the dependency queue to properly handle packages with the same
name but from different sources.
A test was added which exercieses this functionality by depending on two
different revs of the same git repo.
This commit implements full support for plugins by answering the question of
whether any target needed as a plugin or needed as a target dependency. This
commit builds on the previous abstractions to enable parallel compilation
wherever possible.
Most notably, `resolve` now takes the root, so it can properly link the
root package with its dependencies (which is required to build the
--externs for the root package).
This commit adds support for passing --extern to dependencies. It allows
multiple copies of the same named dependency in the system, as long as
they come from different repos.
This touches up the filtering logic to ensure that upstream dependencies as well
as local dependencies are built with optimizations when `cargo build --release`
is used.
If a package had both bin and lib deps, `cargo test` was not building the `lib`
dependency when building the bins with `--test`. This commit adds an extra
"test" profile (not compiled with --test) for situations such as this which is
filtered out normally but kept around for the `cargo test` case.
This PR moves Cargo over to the new naming based on [RFC 35](https://github.com/rust-lang/rfcs/blob/master/complete/0035-remove-crate-id.md).
* You should no longer use `#[crate_name]` or `#[crate_id]` in any crate managed by Cargo.
* You no longer need a `[[lib]]` section in your `Cargo.toml` if you name your library `src/lib.rs`.
* You no longer need a `[[bin]]` section in your `Cargo.toml` if you name your library `src/main.rs`.
* You can include multiple `[[bins]]` in `src/bin/*.rs` and Cargo will treat them as if they were specified via `[[bin]]`.
This commit does not yet add support for `-C metadata` or `-C extra-file-name`, but that is coming soon.
Fingerprinting will fail at an `fs::stat()` call if there is a symlink in
a package's directory pointing to a non-existent file or directory.
This commit recovers from an `fs::stat(`) error on these bogus symlinks by
faking an mtime of 0, which should not affect the overall fingerprint.
Fix#135
Fingerprinting will fail at an fs::stat() call if there is a broken
symlink in a package's directory. This commit recovers from fs::stat()
errors on broken symlinks by treating them as having mtime 0, which
should not affect the overall fingerprint.
Fix#135
* The installation script was modified to recognize when its running on windows,
as well as tweaking how it downloads and installs snapshots. The goal here was
to make the script runnable on buildbot for mac/linux/windows with 32/64 bit
options on mac/linux.
* The installation script now install rustc to `rustc/bin` in the local
directory to have parallel builds on buildbot.
* The tests now store all their temporary state locally in the build directory
to enable parallel builds on buildbot.
* A shell test is ignored which assumed the presence of a TTY output.
* The installation script was modified to recognize when its running on windows,
as well as tweaking how it downloads and installs snapshots. The goal here was
to make the script runnable on buildbot for mac/linux/windows with 32/64 bit
options on mac/linux.
* The installation script now install rustc to `rustc/bin` in the local
directory to have parallel builds on buildbot.
* The tests now store all their temporary state locally in the build directory
to enable parallel builds on buildbot.
* A shell test is ignored which assumed the presence of a TTY output.
When compiling a package with a nested dependency, any modification to the outer
package would trigger a recompilation of the inner package. This commit alters
the fingerprint() method to take a PackageId to query about the location of a
package and only lookup the files relevant to that package.
The dependency structure of a PathSource is now everything rooted at the
original Cargo.toml minus all subdirectories which contain a Cargo.toml
Windows occasionally has \r\n while everywhere else has \n. Instead of worrying
about the difference, just replace all instances of \r with nothing and rely on
matching against \n.
* Add a convenience method bin() for generating the name of a binary. On windows
this remembers to append `.exe`.
* Stop executing relative paths to binaries and relying on PATH. This is
suffering from rust-lang/rust#15149 and failing to spawn processes on windows.
Additionally, this allows the tests to work with a pre-installed cargo becuase
the freshly built executables are precisely specified.
* A new function, escape_path(), was added for tests. When generated source
files with paths, this function needs to be called to properly escape the
\-character that appears in windows path names. Without this function we would
be generating invalid TOML and rust.
Apparently git checkouts have objects in the database with permissions 444 which
need to be changed to something with a write permission before removal.
This commit adds support for recognizing "fingerprints" of upstream
dependencies. When a dependency's fingerprint change, it must be rebuilt.
Currently the fingerprint unconditionally includes the version of the compiler
you're using as well as a specialized version depending on the type of source
you're compiling from:
- git sources return their fingerprint as the current SHA. This will
disregard any local changes.
- path sources return their fingerprint as the maximum mtime of any file found
at the location. This is a little too coarse and may rebuild packages too
often (due to sub-packages), but this should get the job done for now.
When executing `cargo compile`, dependencies are not rebuilt if their
fingerprint remained constant.
This commit enables support for custom precompilation commands to be triggered
before a package builds via rustc. The current interface is to have
`build = "foo"` in the `[project]` section of Cargo.toml and cargo will just
execute the exact command given.
At the moment, the rustc exec for the root project inherits the stdout
and stderr FDs (so that warnings and errors flow through), but
output from dependencies is only emitted if the compilation fails to
avoid warning noise from dependencies.
At the same time, we started adding a generic error handling concept to
Cargo.
The idea is that every Result gets converted to a CargoError, which
includes all the information that Cargo needs to print out a friendly
human error message and exit gracefully.
