Skip to content

Benchmarks: make the gperf baseline a tuned map - #41

Merged
lemire merged 1 commit into
mainfrom
francisco/gperf-baseline-fairness
Nov 14, 2026
Merged

lemire merged 1 commit into
mainfrom
francisco/gperf-baseline-fairness

Conversation

@FranciscoThiesen

@FranciscoThiesen FranciscoThiesen commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #29.

#29 asked for the gperf baseline to be a map with tuned options. Checking it turned up two lanes that weren't measuring gperf at all:

  • JavaScript reserved words ran a std::unordered_map lookup after every gperf hit.
  • Counters was a std::unordered_set; there was no gperf table for it.

Changes

  • Every gperf input is a map now, like the other methods in the benchmark: %struct-type with an int value (the key's declaration index), plus %readonly-tables and %enum. The wrappers return entry->value.
  • New benchmarks/gperf/counters.gperf; the JS lane no longer touches std::unordered_map.
  • The string comparison is chosen per set, by measurement. %compare-lengths is used for MIME types, letters, Counters and Java FQCNs; %compare-strncmp is kept for the short-key sets and the S&P 500 / Nasdaq tickers, where the length table costs more than it saves.
  • All tables are regenerated with GNU gperf 3.3 by one script, benchmarks/gperf/regenerate.sh [path/to/gperf]. The letters, HTTP Headers 50 and FQCN inputs move into benchmarks/gperf/ with the others. gen_symbol_headers.py writes the ticker inputs in the same form.
  • Verification now checks the value gperf returns, and bench_tickers verifies its two gperf tables.
  • benchmarks/gperf/README.md documents the options and the measurements behind them.

Options measured and not used

  • %switch=1: slower on every set, up to 2.6x on hits.
  • -m 10000: shrank some tables but made no consistent difference to lookup time (it already saturates around -m 100).
  • %compare-lengths everywhere: big wins on long or similar keys (FQCN hits 12.2 → 4.3 ns) but it hurts short ones (S&P 100 misses 1.8 → 4.0 ns), hence the per-set choice.

Results

gperf lane, ns per lookup, main → this branch (Apple M3 Max, clang 22, best of two runs). The last column is make_perfect_map for reference.

Set hits misses mixed ours, hits
URL Protocols 5.43 → 5.56 2.04 → 2.18 6.13 → 6.10 0.93
S&P 100 Tickers 5.49 → 5.64 1.79 → 1.82 6.98 → 7.21 1.61
C++ Keywords 5.38 → 5.35 3.11 → 3.19 6.74 → 6.89 1.23
HTTP Headers 7.81 → 8.01 4.99 → 5.08 7.72 → 8.06 0.86
MIME Types 8.80 → 6.41 5.04 → 6.01 9.52 → 7.92 1.18
Letters a-z 0.73 → 0.78 1.35 → 1.36 2.92 → 2.90 0.53
HTTP Headers 50 8.50 → 8.28 4.64 → 4.75 7.87 → 8.00 1.77
JavaScript Reserved Words 22.27 → 7.20 4.07 → 3.36 18.06 → 7.59 1.13
Counters 13.89 → 7.71 9.64 → 3.96 17.62 → 8.45 1.61
Java FQCNs 12.24 → 4.31 7.78 → 5.33 11.41 → 5.65 1.96

The short-key sets are unchanged within noise, and so are the S&P 500 and Nasdaq gperf lanes in bench_tickers. The README headline gperf number (URL Protocols) is unaffected. MIME misses get slower with %compare-lengths while hits and mixed get faster; I kept it there because the mixed workload improves.

Not measured: x86-64 / glibc. The per-set choice depends on how the C library's memcmp compares with its strncmp, so it may come out differently there.

CI doesn't build the benchmarks, so I checked locally:

  • clang 22 builds bench_protocol and bench_tickers, and all 10 protocol sets and the 4 ticker checks verify;
  • GCC 16 compiles all 12 generated tables with -Wall -Wextra and no warnings, and every key (6,411 in all) returns its value. The full benchmarks don't build with Homebrew GCC on macOS, because its libstdc++ doesn't match the SDK headers that abseil includes; that's unrelated to this change.

Every gperf lane is now a map, like the other methods: %struct-type with an
int value (the declaration index), and the wrapper returns entry->value.

Two lanes did not measure gperf:
- JavaScript reserved words ran a std::unordered_map lookup after every gperf
  hit (22.3 -> 7.2 ns on hits, Apple M3 Max).
- Counters was a std::unordered_set, not gperf (13.9 -> 7.7 ns).

Options, measured per set (M3 Max, clang 22, min of two runs):
- %compare-lengths on MIME, letters, counters and FQCN, where it wins
  (FQCN hits 12.2 -> 4.3 ns, MIME 9.0 -> 6.4 ns). Kept %compare-strncmp on
  the short-key sets, where the length table costs more (S&P 100 misses
  1.8 -> 4.0 ns with it).
- %switch=1 was slower on every set (up to 2.6x on hits), and -m 10000 made
  no consistent difference, so neither is used.

All tables are regenerated with GNU gperf 3.3 by one script,
benchmarks/gperf/regenerate.sh. The letters, headers50 and FQCN inputs move
into benchmarks/gperf/ with the others. Verification now checks gperf's
values, and bench_tickers verifies its two gperf tables. The ticker lanes
and the short-key sets are unchanged within noise.
@lemire
lemire merged commit c65d776 into main Nov 14, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

benchmarks: gperf results maybe can be improved

2 participants