From dfe5e31c5203648ecc312a34a752754099a90cd4 Mon Sep 17 00:00:00 2001 From: Austin Bonander Date: Wed, 1 Jul 2020 14:52:16 -0700 Subject: [PATCH] perf: document `sqlx-bench` and pool fairness benchmark --- sqlx-bench/README.md | 39 +++++++++++ .../1000_fair_pdf_small.svg | 64 +++++++++++++++++ .../1000_unfair_pdf_small.svg | 68 +++++++++++++++++++ .../100_fair_pdf_small.svg | 60 ++++++++++++++++ .../100_unfair_pdf_small.svg | 60 ++++++++++++++++ .../10_fair_pdf_small.svg | 56 +++++++++++++++ .../10_unfair_pdf_small.svg | 48 +++++++++++++ .../5000_fair_pdf_small.svg | 56 +++++++++++++++ .../5000_unfair_pdf_small.svg | 68 +++++++++++++++++++ .../500_fair_pdf_small.svg | 44 ++++++++++++ .../500_unfair_pdf_small.svg | 52 ++++++++++++++ .../50_fair_pdf_small.svg | 52 ++++++++++++++ .../50_unfair_pdf_small.svg | 52 ++++++++++++++ .../5_fair_pdf_small.svg | 60 ++++++++++++++++ .../5_unfair_pdf_small.svg | 60 ++++++++++++++++ .../2020-07-01-bench_pgpool_acquire/REPORT.md | 32 +++++++++ 16 files changed, 871 insertions(+) create mode 100644 sqlx-bench/README.md create mode 100644 sqlx-bench/results/2020-07-01-bench_pgpool_acquire/1000_fair_pdf_small.svg create mode 100644 sqlx-bench/results/2020-07-01-bench_pgpool_acquire/1000_unfair_pdf_small.svg create mode 100644 sqlx-bench/results/2020-07-01-bench_pgpool_acquire/100_fair_pdf_small.svg create mode 100644 sqlx-bench/results/2020-07-01-bench_pgpool_acquire/100_unfair_pdf_small.svg create mode 100644 sqlx-bench/results/2020-07-01-bench_pgpool_acquire/10_fair_pdf_small.svg create mode 100644 sqlx-bench/results/2020-07-01-bench_pgpool_acquire/10_unfair_pdf_small.svg create mode 100644 sqlx-bench/results/2020-07-01-bench_pgpool_acquire/5000_fair_pdf_small.svg create mode 100644 sqlx-bench/results/2020-07-01-bench_pgpool_acquire/5000_unfair_pdf_small.svg create mode 100644 sqlx-bench/results/2020-07-01-bench_pgpool_acquire/500_fair_pdf_small.svg create mode 100644 sqlx-bench/results/2020-07-01-bench_pgpool_acquire/500_unfair_pdf_small.svg create mode 100644 sqlx-bench/results/2020-07-01-bench_pgpool_acquire/50_fair_pdf_small.svg create mode 100644 sqlx-bench/results/2020-07-01-bench_pgpool_acquire/50_unfair_pdf_small.svg create mode 100644 sqlx-bench/results/2020-07-01-bench_pgpool_acquire/5_fair_pdf_small.svg create mode 100644 sqlx-bench/results/2020-07-01-bench_pgpool_acquire/5_unfair_pdf_small.svg create mode 100644 sqlx-bench/results/2020-07-01-bench_pgpool_acquire/REPORT.md diff --git a/sqlx-bench/README.md b/sqlx-bench/README.md new file mode 100644 index 00000000..da6a4ac0 --- /dev/null +++ b/sqlx-bench/README.md @@ -0,0 +1,39 @@ +SQLx Self-Benchmarks +==================== + +This Cargo project implements various benchmarks for SQLx using +[Criterion](https://crates.io/crates/criterion). + +### Available Benchmarks + +* Group `pg_pool`: benchmarks `sqlx::Pool` against a PostgreSQL server. + * `DATABASE_URL` must be set (or in `.env`) pointing to a PostgreSQL server. + It should preferably be running on the same machine as the benchmarks to reduce latency. + * The `postgres` feature must be enabled for this benchmark to run. + * Benchmarks: + * `bench_pgpool_acquire`: benchmarks `Pool::acquire()` when many concurrent tasks are also using + the pool, with or without the pool being fair. Concurrently to the benchmark iteration + function calling and blocking on `Pool::acquire()`, a varying number of background tasks are + also calling `acquire()` and holding the acquired connection for 500µs each before releasing + it back to the pool. The pool is created with `.min_size(50).max_size(50)` so we shouldn't + be measuring anything but the actual overhead of `Pool`'s bookeeping. + +### Running + +You must choose a runtime to execute the benchmarks on; the feature flags are the same as the `sqlx` crate: + +```bash +cargo bench --features runtime-tokio +cargo bench --features runtime-async-std +``` + +When complete, the benchmark results will be in `target/criterion/`. +Open `target/criterion/report/index.html` or pick one of the benchmark subfolders and open +`report/index.html` there to view the results. + +Benchmark Results +------- + +If you want to share the results here, please follow the format below. + +* [2020/07/01: `pg_pool` benchmark added to test pool fairness changes](results/2020-07-01-bench_pgpool_acquire/REPORT.md) diff --git a/sqlx-bench/results/2020-07-01-bench_pgpool_acquire/1000_fair_pdf_small.svg b/sqlx-bench/results/2020-07-01-bench_pgpool_acquire/1000_fair_pdf_small.svg new file mode 100644 index 00000000..6446c35b --- /dev/null +++ b/sqlx-bench/results/2020-07-01-bench_pgpool_acquire/1000_fair_pdf_small.svg @@ -0,0 +1,64 @@ + + +Density (a.u.) + + +Average Time (ms) + + + +0.05 + + + +0.1 + + + +0.15 + + + +0.2 + + + +0.25 + + + +0.3 + + + +0.35 + + + +0.4 + + + +0.45 + + + +0.5 + + + + +30 + + + +32 + + + +34 + + + + + diff --git a/sqlx-bench/results/2020-07-01-bench_pgpool_acquire/1000_unfair_pdf_small.svg b/sqlx-bench/results/2020-07-01-bench_pgpool_acquire/1000_unfair_pdf_small.svg new file mode 100644 index 00000000..309a62b2 --- /dev/null +++ b/sqlx-bench/results/2020-07-01-bench_pgpool_acquire/1000_unfair_pdf_small.svg @@ -0,0 +1,68 @@ + + +Density (a.u.) + + +Average Time (ms) + + + +5e-4 + + + +0.001 + + + +0.0015 + + + +0.002 + + + +0.0025 + + + +0.003 + + + +0.0035 + + + +0.004 + + + +0.0045 + + + +0.005 + + + +0.0055 + + + + +0 + + + +200 + + + +400 + + + + + diff --git a/sqlx-bench/results/2020-07-01-bench_pgpool_acquire/100_fair_pdf_small.svg b/sqlx-bench/results/2020-07-01-bench_pgpool_acquire/100_fair_pdf_small.svg new file mode 100644 index 00000000..65f2e620 --- /dev/null +++ b/sqlx-bench/results/2020-07-01-bench_pgpool_acquire/100_fair_pdf_small.svg @@ -0,0 +1,60 @@ + + +Density (a.u.) + + +Average Time (ms) + + + +0.1 + + + +0.2 + + + +0.3 + + + +0.4 + + + +0.5 + + + +0.6 + + + +0.7 + + + +0.8 + + + + +0 + + + +1 + + + +2 + + + +3 + + + + + diff --git a/sqlx-bench/results/2020-07-01-bench_pgpool_acquire/100_unfair_pdf_small.svg b/sqlx-bench/results/2020-07-01-bench_pgpool_acquire/100_unfair_pdf_small.svg new file mode 100644 index 00000000..db708916 --- /dev/null +++ b/sqlx-bench/results/2020-07-01-bench_pgpool_acquire/100_unfair_pdf_small.svg @@ -0,0 +1,60 @@ + + +Density (a.u.) + + +Average Time (ms) + + + +0.02 + + + +0.04 + + + +0.06 + + + +0.08 + + + +0.1 + + + +0.12 + + + +0.14 + + + +0.16 + + + +0.18 + + + + +0 + + + +5 + + + +10 + + + + + diff --git a/sqlx-bench/results/2020-07-01-bench_pgpool_acquire/10_fair_pdf_small.svg b/sqlx-bench/results/2020-07-01-bench_pgpool_acquire/10_fair_pdf_small.svg new file mode 100644 index 00000000..b887f78e --- /dev/null +++ b/sqlx-bench/results/2020-07-01-bench_pgpool_acquire/10_fair_pdf_small.svg @@ -0,0 +1,56 @@ + + +Density (a.u.) + + +Average Time (ns) + + + +0.01 + + + +0.02 + + + +0.03 + + + +0.04 + + + +0.05 + + + +0.06 + + + +0.07 + + + + +340 + + + +360 + + + +380 + + + +400 + + + + + diff --git a/sqlx-bench/results/2020-07-01-bench_pgpool_acquire/10_unfair_pdf_small.svg b/sqlx-bench/results/2020-07-01-bench_pgpool_acquire/10_unfair_pdf_small.svg new file mode 100644 index 00000000..884f443d --- /dev/null +++ b/sqlx-bench/results/2020-07-01-bench_pgpool_acquire/10_unfair_pdf_small.svg @@ -0,0 +1,48 @@ + + +Density (a.u.) + + +Average Time (ns) + + + +0.02 + + + +0.04 + + + +0.06 + + + +0.08 + + + +0.1 + + + + +340 + + + +350 + + + +360 + + + +370 + + + + + diff --git a/sqlx-bench/results/2020-07-01-bench_pgpool_acquire/5000_fair_pdf_small.svg b/sqlx-bench/results/2020-07-01-bench_pgpool_acquire/5000_fair_pdf_small.svg new file mode 100644 index 00000000..41bb9dd9 --- /dev/null +++ b/sqlx-bench/results/2020-07-01-bench_pgpool_acquire/5000_fair_pdf_small.svg @@ -0,0 +1,56 @@ + + +Density (a.u.) + + +Average Time (ms) + + + +0.01 + + + +0.02 + + + +0.03 + + + +0.04 + + + +0.05 + + + +0.06 + + + +0.07 + + + + +160 + + + +170 + + + +180 + + + +190 + + + + + diff --git a/sqlx-bench/results/2020-07-01-bench_pgpool_acquire/5000_unfair_pdf_small.svg b/sqlx-bench/results/2020-07-01-bench_pgpool_acquire/5000_unfair_pdf_small.svg new file mode 100644 index 00000000..51c712ae --- /dev/null +++ b/sqlx-bench/results/2020-07-01-bench_pgpool_acquire/5000_unfair_pdf_small.svg @@ -0,0 +1,68 @@ + + +Density (a.u.) + + +Average Time (s) + + + +0.05 + + + +0.1 + + + +0.15 + + + +0.2 + + + +0.25 + + + +0.3 + + + +0.35 + + + +0.4 + + + +0.45 + + + +0.5 + + + + +0 + + + +2 + + + +4 + + + +6 + + + + + diff --git a/sqlx-bench/results/2020-07-01-bench_pgpool_acquire/500_fair_pdf_small.svg b/sqlx-bench/results/2020-07-01-bench_pgpool_acquire/500_fair_pdf_small.svg new file mode 100644 index 00000000..249f59b4 --- /dev/null +++ b/sqlx-bench/results/2020-07-01-bench_pgpool_acquire/500_fair_pdf_small.svg @@ -0,0 +1,44 @@ + + +Density (a.u.) + + +Average Time (ms) + + + +0.2 + + + +0.4 + + + +0.6 + + + +0.8 + + + +1 + + + + +15 + + + +16 + + + +17 + + + + + diff --git a/sqlx-bench/results/2020-07-01-bench_pgpool_acquire/500_unfair_pdf_small.svg b/sqlx-bench/results/2020-07-01-bench_pgpool_acquire/500_unfair_pdf_small.svg new file mode 100644 index 00000000..b9387fc3 --- /dev/null +++ b/sqlx-bench/results/2020-07-01-bench_pgpool_acquire/500_unfair_pdf_small.svg @@ -0,0 +1,52 @@ + + +Density (a.u.) + + +Average Time (ms) + + + +0.002 + + + +0.004 + + + +0.006 + + + +0.008 + + + +0.01 + + + + +0 + + + +50 + + + +100 + + + +150 + + + +200 + + + + + diff --git a/sqlx-bench/results/2020-07-01-bench_pgpool_acquire/50_fair_pdf_small.svg b/sqlx-bench/results/2020-07-01-bench_pgpool_acquire/50_fair_pdf_small.svg new file mode 100644 index 00000000..c8958b9e --- /dev/null +++ b/sqlx-bench/results/2020-07-01-bench_pgpool_acquire/50_fair_pdf_small.svg @@ -0,0 +1,52 @@ + + +Density (a.u.) + + +Average Time (us) + + + +0.005 + + + +0.01 + + + +0.015 + + + +0.02 + + + +0.025 + + + + +0 + + + +50 + + + +100 + + + +150 + + + +200 + + + + + diff --git a/sqlx-bench/results/2020-07-01-bench_pgpool_acquire/50_unfair_pdf_small.svg b/sqlx-bench/results/2020-07-01-bench_pgpool_acquire/50_unfair_pdf_small.svg new file mode 100644 index 00000000..e16502cc --- /dev/null +++ b/sqlx-bench/results/2020-07-01-bench_pgpool_acquire/50_unfair_pdf_small.svg @@ -0,0 +1,52 @@ + + +Density (a.u.) + + +Average Time (us) + + + +0.005 + + + +0.01 + + + +0.015 + + + +0.02 + + + +0.025 + + + +0.03 + + + +0.035 + + + + +0 + + + +50 + + + +100 + + + + + diff --git a/sqlx-bench/results/2020-07-01-bench_pgpool_acquire/5_fair_pdf_small.svg b/sqlx-bench/results/2020-07-01-bench_pgpool_acquire/5_fair_pdf_small.svg new file mode 100644 index 00000000..3ebb0258 --- /dev/null +++ b/sqlx-bench/results/2020-07-01-bench_pgpool_acquire/5_fair_pdf_small.svg @@ -0,0 +1,60 @@ + + +Density (a.u.) + + +Average Time (ns) + + + +0.01 + + + +0.02 + + + +0.03 + + + +0.04 + + + +0.05 + + + +0.06 + + + +0.07 + + + + +340 + + + +360 + + + +380 + + + +400 + + + +420 + + + + + diff --git a/sqlx-bench/results/2020-07-01-bench_pgpool_acquire/5_unfair_pdf_small.svg b/sqlx-bench/results/2020-07-01-bench_pgpool_acquire/5_unfair_pdf_small.svg new file mode 100644 index 00000000..c33fee3e --- /dev/null +++ b/sqlx-bench/results/2020-07-01-bench_pgpool_acquire/5_unfair_pdf_small.svg @@ -0,0 +1,60 @@ + + +Density (a.u.) + + +Average Time (ns) + + + +0.01 + + + +0.02 + + + +0.03 + + + +0.04 + + + +0.05 + + + +0.06 + + + +0.07 + + + +0.08 + + + +0.09 + + + + +340 + + + +360 + + + +380 + + + + + diff --git a/sqlx-bench/results/2020-07-01-bench_pgpool_acquire/REPORT.md b/sqlx-bench/results/2020-07-01-bench_pgpool_acquire/REPORT.md new file mode 100644 index 00000000..0fb71b14 --- /dev/null +++ b/sqlx-bench/results/2020-07-01-bench_pgpool_acquire/REPORT.md @@ -0,0 +1,32 @@ +### 2020/07/01: `pg_pool` benchmark added to test pool fairness changes + +* Commit: 7c8ef602661c9cb62c8cf43aaf0f8faaf5b0aed5 +* Machine specs: + * (`lscpu`) Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz + * 64 GB DDR4 RAM + * (Distro / `uname -a`) Arch Linux / kernel `5.6.15-zen2-1-zen` + * (`rustc -V`) `rustc 1.43.0-nightly (834bc5650 2020-02-24)` +* Command: `cargo bench --features runtime-tokio,postgres -- --measurement-time 30` + +| Connections | Unfair Pool | Fair Pool| +|-------------|-------------|----------| +| 5 | ![](5_unfair_pdf_small.svg) | ![](5_fair_pdf_small.svg) | +| 10 | ![](10_unfair_pdf_small.svg) | ![](10_fair_pdf_small.svg) | +| 50 | ![](50_unfair_pdf_small.svg) | ![](50_fair_pdf_small.svg) | +| 100 | ![](100_unfair_pdf_small.svg) | ![](100_fair_pdf_small.svg) | +| 500 | ![](500_unfair_pdf_small.svg) | ![](500_fair_pdf_small.svg) | +| 1000 | ![](1000_unfair_pdf_small.svg) | ![](1000_fair_pdf_small.svg) | +| 5000 | ![](5000_unfair_pdf_small.svg) | ![](5000_fair_pdf_small.svg) | + + +When the pool is set to be fair it appears the average time for `acquire()` at high contention +(500+ tasks using 50 pool connections) actually goes down by up to an order of magnitude since tasks +in the waiter queue aren't getting preempted. + +This appears to be the solution to our problem of tasks timing out waiting for a connection at high +load. At very low contention (5-10 tasks using 50 pool connections), the average time for `acqure()` +increases by perhaps 10% but a difference between 80µs and 90µs is hardly noticeable in real +applications. + +A normal MPMC channel may care about that kind of overhead, though, which is why most implementations +aren't perfectly fair.