cabe1cca0a
This didn't actually result in any code changes yet, for now this simply flips the edition flag for all of our crates and documentation. |
||
---|---|---|
.. | ||
benchsuite | ||
capture | ||
workspaces | ||
README.md |
Cargo Benchmarking
This directory contains some benchmarks for cargo itself. This uses Criterion for running benchmarks. It is recommended to read the Criterion book to get familiar with how to use it. A basic usage would be:
cd benches/benchsuite
cargo bench
The tests involve downloading the index and benchmarking against some
real-world and artificial workspaces located in the workspaces
directory.
Beware that the initial download can take a fairly long amount of time (10
minutes minimum on an extremely fast network) and require significant disk
space (around 4.5GB). The benchsuite will cache the index and downloaded
crates in the target/tmp/bench
directory, so subsequent runs should be
faster. You can (and probably should) specify individual benchmarks to run to
narrow it down to a more reasonable set, for example:
cargo bench -- resolve_ws/rust
This will only download what's necessary for the rust-lang/rust workspace (which is about 330MB) and run the benchmarks against it (which should take about a minute). To get a list of all the benchmarks, run:
cargo bench -- --list
Viewing reports
The benchmarks display some basic information on the command-line while they
run. A more complete HTML report can be found at
target/criterion/report/index.html
which contains links to all the
benchmarks and summaries. Check out the Criterion book for more information on
the extensive reporting capabilities.
Comparing implementations
Knowing the raw numbers can be useful, but what you're probably most interested in is checking if your changes help or hurt performance. To do that, you need to run the benchmarks multiple times.
First, run the benchmarks from the master branch of cargo without any changes. To make it easier to compare, Criterion supports naming the baseline so that you can iterate on your code and compare against it multiple times.
cargo bench -- --save-baseline master
Now you can switch to your branch with your changes. Re-run the benchmarks compared against the baseline:
cargo bench -- --baseline master
You can repeat the last command as you make changes to re-compare against the master baseline.
Without the baseline arguments, it will compare against the last run, which can be helpful for comparing incremental changes.
Capturing workspaces
The workspaces
directory contains several workspaces that
provide a variety of different workspaces intended to provide good exercises
for benchmarks. Some of these are shadow copies of real-world workspaces. This
is done with the tool in the capture
directory. The tool will
copy Cargo.lock
and all of the Cargo.toml
files of the workspace members.
It also adds an empty lib.rs
so Cargo won't error, and sanitizes the
Cargo.toml
to some degree, removing unwanted elements. Finally, it
compresses everything into a tgz
.
To run it, do:
cd benches/capture
cargo run -- /path/to/workspace/foo
The resolver benchmarks also support the CARGO_BENCH_WORKSPACES
environment
variable, which you can point to a Cargo workspace if you want to try
different workspaces. For example:
CARGO_BENCH_WORKSPACES=/path/to/some/workspace cargo bench
TODO
This is just a start for establishing a benchmarking suite for Cargo. There's a lot that can be added. Some ideas:
- Fix the benchmarks so that the resolver setup doesn't run every iteration.
- Benchmark this section of
code
which builds the unit graph. The performance there isn't great, and it would
be good to keep an eye on it. Unfortunately that would mean doing a bit of
work to make
generate_targets
publicly visible, and there is a bunch of setup code that may need to be duplicated. - Benchmark the fingerprinting code.
- Benchmark running the
cargo
executable. Running something likecargo build
orcargo check
with everything "Fresh" would be a good end-to-end exercise to measure the overall overhead of Cargo. - Benchmark pathological resolver scenarios. There might be some cases where
the resolver can spend a significant amount of time. It would be good to
identify if these exist, and create benchmarks for them. This may require
creating an artificial index, similar to the
resolver-tests
. This should also consider scenarios where the resolver ultimately fails. - Benchmark without
Cargo.lock
. I'm not sure if this is particularly valuable, since we are mostly concerned with incremental builds which will always have a lock file. - Benchmark just
resolve::resolve
without anything else. This can help focus on just the resolver.