knowledge/technology/applications/cli/hyperfine.md

obj	repo
application	https://github.com/sharkdp/hyperfine

hyperfine

A command-line benchmarking tool.

Usage

To run a benchmark, you can simply call hyperfine .... The argument(s) can be any shell command. For example:

hyperfine 'sleep 0.3'

Hyperfine will automatically determine the number of runs to perform for each command. By default, it will perform at least 10 benchmarking runs and measure for at least 3 seconds. To change this, you can use the -r/--runs option:

hyperfine --runs 5 'sleep 0.3'

If you want to compare the runtimes of different programs, you can pass multiple commands:

hyperfine 'hexdump file' 'xxd file'

Warmup runs and preparation commands

For programs that perform a lot of disk I/O, the benchmarking results can be heavily influenced by disk caches and whether they are cold or warm.

If you want to run the benchmark on a warm cache, you can use the -w/--warmup option to perform a certain number of program executions before the actual benchmark:

hyperfine --warmup 3 'grep -R TODO *'

Conversely, if you want to run the benchmark for a cold cache, you can use the -p/--prepare option to run a special command before each timing run. For example, to clear harddisk caches on Linux, you can run

sync; echo 3 | sudo tee /proc/sys/vm/drop_caches

To use this specific command with hyperfine, call sudo -v to temporarily gain sudo permissions and then call:

hyperfine --prepare 'sync; echo 3 | sudo tee /proc/sys/vm/drop_caches' 'grep -R TODO *'

Parameterized benchmarks

If you want to run a series of benchmarks where a single parameter is varied (say, the number of threads), you can use the -P/--parameter-scan option and call:

hyperfine --prepare 'make clean' --parameter-scan num_threads 1 12 'make -j {num_threads}'

This also works with decimal numbers. The -D/--parameter-step-size option can be used to control the step size:

hyperfine --parameter-scan delay 0.3 0.7 -D 0.2 'sleep {delay}'

This runs sleep 0.3, sleep 0.5 and sleep 0.7.

For non-numeric parameters, you can also supply a list of values with the -L/--parameter-list option:

hyperfine -L compiler gcc,clang '{compiler} -O2 main.cpp'

Options

Option	Description
-w, --warmup <NUM>	Perform NUM warmup runs before the actual benchmark. This can be used to fill (disk) caches for I/O-heavy programs.
-m, --min-runs <NUM>	Perform at least NUM runs for each command (default: 10).
-M, --max-runs <NUM>	Perform at most NUM runs for each command. By default, there is no limit.
-r, --runs <NUM>	Perform exactly NUM runs for each command. If this option is not specified, hyperfine automatically determines the number of runs.
-s, --setup <CMD>	Execute CMD before each set of timing runs. This is useful for compiling your software with the provided parameters, or to do any other work that should happen once before a series of benchmark runs, not every time as would happen with the --prepare option.
-p, --prepare <CMD>	Execute CMD before each timing run. This is useful for clearing disk caches, for example. The --prepare option can be specified once for all commands or multiple times, once for each command. In the latter case, each preparation command will be run prior to the corresponding benchmark command.
-c, --cleanup <CMD>	Execute CMD after the completion of all benchmarking runs for each individual command to be benchmarked. This is useful if the commands to be benchmarked produce artifacts that need to be cleaned up.
-P, --parameter-scan <VAR> <MIN> <MAX>	Perform benchmark runs for each value in the range MIN..MAX. Replaces the string '{VAR}' in each command by the current parameter value.
-D, --parameter-step-size <DELTA>	This argument requires --parameter-scan to be specified as well. Traverse the range MIN..MAX in steps of DELTA.
-L, --parameter-list <VAR> <VALUES>	Perform benchmark runs for each value in the comma-separated list VALUES. Replaces the string '{VAR}' in each command by the current parameter value.
-i, --ignore-failure	Ignore non-zero exit codes of the benchmarked programs.
--export-asciidoc <FILE>	Export the timing summary statistics as an AsciiDoc table to the given FILE.
--export-csv <FILE>	Export the timing summary statistics as CSV to the given FILE. If you need the timing results for each individual run, use the JSON export format. The output time unit is always seconds.
--export-json <FILE>	Export the timing summary statistics and timings of individual runs as JSON to the given FILE. The output time unit is always seconds
--export-markdown <FILE>	Export the timing summary statistics as a Markdown table to the given FILE.
--show-output	Print the stdout and stderr of the benchmark instead of suppressing it. This will increase the time it takes for benchmarks to run, so it should only be used for debugging purposes or when trying to benchmark output speed.
-n, --command-name <NAME>	Give a meaningful name to a command. This can be specified multiple times if several commands are benchmarked.
--output <WHERE>	Control where the output of the benchmark is redirected. Note that some programs like 'grep' detect when standard output is /dev/null and apply certain optimizations. To avoid that, consider using '--output=pipe'. <WHERE> can be: - null: Redirect output to /dev/null (the default). - pipe: Feed the output through a pipe before discarding it. - inherit: Don't redirect the output at all (same as '--show-output'). - <FILE>: Write the output to the given file.
--input <WHERE>	Control where the input of the benchmark comes from. <WHERE> can be: - null: Read from /dev/null (the default). - <FILE>: Read the input from the given file.