Merge pull request #1674 from Kobzol/runtime-doc

Add more runtime benchmark documentation

Author: Kobzol

collector/README.md

@@ -6,9 +6,12 @@ Hardware and software details of the machine that executes the CI details can be
 
 ## The benchmarks
 
-The individual compile time benchmarks are described in the `README` file in the
+Compile time benchmarks are described in the `README` file in the
 `collector/compile-benchmarks` directory.
 
+Runtime benchmarks are described in the `README` file in the
+`collector/runtime-benchmarks` directory.
+
 ## How to build
 
 Before doing anything else, you should build `collector` (for running the
@@ -65,7 +68,8 @@ marked with a '?' in the `compare` page.
 
 ### How to benchmark a change on your own machine
 
-The following command runs the benchmark suite using a local rustc:
+The following command runs the compile benchmark suite (which measures how long does it take to compile
+various crates with rustc) using a local rustc:
 ```
 ./target/release/collector bench_local <RUSTC>
 ```
@@ -167,6 +171,22 @@ something like this:
 where `$RUST_ORIGINAL` and `$RUST_MODIFIED` are paths (relative or absolute) to
 the relevant rustc executables.
 
+#### Runtime benchmarks
+There is also a runtime benchmark suite, which measures the performance of Rust programs compiled
+by a selected version of rustc. You can run it using the following command:
+```bash
+./target/release/collector bench_runtime_local <RUSTC>
+```
+
+### Benchmarking options
+
+The following options alter the behaviour of the `bench_runtime_local` subcommand.
+- `--no-isolate`: you can use this flag to make repeated local benchmarks faster. It will cause the
+  `collector` to reuse compiled artifacts of the runtime benchmark groups.
+
+The `bench_runtime_local` command also shares some options with the `bench_local` command, notably
+`--id`, `--db`, `--cargo`, `--include`, `--exclude` and `--iterations`. 
+
 ### How to view the measurements on your own machine
 
 Once the benchmarks have been run, build and start the website.

collector/runtime-benchmarks/README.md

@@ -11,6 +11,68 @@ Runtime benchmarks are divided into groups so that some benchmarks can use diffe
 dependency crates and also so that they are grouped together by a relevant area
 (e.g. hashmap benchmarks).
 
+## Benchmark descriptions
+> The runtime benchmark suite is currently experimental, so it is possible that some benchmarks will
+be heavily modified or removed, and new ones will be added. Once the suite will be more stable, the
+individual benchmarks will be described here.
+
+## How to add a new benchmark
+First you should decide whether you will create a new benchmark group or not. If you find a group that
+seems relevant to your benchmark (e.g. if you want to add a new benchmark that tests the performance
+of a hash map, the `hashmap` group is ideal for that), then
+[add the benchmark](#adding-a-benchmark-to-a-benchmark-group) to it directly. If not, you should create
+a new group.
+
+### Creating a new benchmark group
+You can create a new benchmark group either by copying an existing group or by creating a new binary
+crate in this directory and adding a dependency on the [`benchlib`](../benchlib) crate to it.
+
+By convention, if a group (its directory) is called `foo`, then the crate name should be `foo-bench`.
+This convention exists to enable creation of groups that have the same name as a dependency that they
+benchmark.
+
+Each group should call the `run_benchmark_group` function from `benchlib` in its `main` function, and
+define a set of benchmarks inside a closure passed to the function. This is an example of how that could
+look like:
+
+```rust
+use benchlib::benchmark::run_benchmark_group;
+
+fn main() {
+    // Initialize the benchmarking infrastructure
+    run_benchmark_group(|group| {
+        // Register a benchmark called bench_1
+        group.register_benchmark("bench_1", || {
+            // This closure should prepare data that will be needed for the benchmark (if any),
+            // and then return a closure that will actually be benchmarked/profiled.
+            let data = vec![0; 1024];
+            move || {
+                // Only this will be actually benchmarked
+                data.iter().sum::<u64>()
+            }
+        });
+    });
+}
+```
+
+### Adding a benchmark to a benchmark group
+Once you have selected a benchmark group, add a new benchmark to it by calling `group.register_benchmark(...)`.
+See above for the description of this function.
+
+Note that if your benchmark requires only immutable access to some input data, consider creating the
+data only once in `main`, and then referencing it in the benchmarked function. This will make the
+benchmark run faster if the data preparation is expensive. It could also in theory reduce noise/variance,
+because the data will exist on a stable address in memory and won't be (re)allocated before each benchmark
+iteration.
+
+> Currently, there is a trade-off to doing a lot of stuff in `main` - it will make the enumeration of
+> benchmarks slower, which can be annoying when doing many local benchmarks. See below for more information.
+
+### What benchmarks are we interested in?
+It is hard to say in general, but we tend to prefer benchmarks containing real-world code that does
+something useful, rather than microbenchmarks. Benchmarks also shouldn't be too short - a benchmark
+should take at least tens or hundreds of milliseconds.
+
 ## How are benchmarks executed
 The `collector` compiles each benchmark group and then invokes it with the `list` argument to list
 all benchmarks contained in the group.