A benchmarking tool for Valkey, an in-memory data store. Measures performance across different commits and configurations, including TLS and cluster modes.
- Benchmarks Valkey server with various commands (SET, GET, RPUSH, etc.)
- Tests with different data sizes and pipeline configurations
- Supports TLS and cluster mode testing
- Handles automatic server setup and teardown
- Collects detailed performance metrics and generates reports
- Compares performance between different Valkey versions/commits
- Provides CPU pinning via
tasksetusing configuration file settings - Runs continuous benchmarking via GitHub Actions workflow
- Tracks commits and manages progress automatically
- Includes Grafana dashboards for visualizing performance metrics
- Search module performance testing with valkey search, adaptable to other modules
- Performance profiling with flamegraph generation via
perf - Regression detection comparing the latest benchmarked commits in PostgreSQL
- Git
- Python 3.6+
- Linux environment (for taskset CPU pinning)
- Build tools required by Valkey (gcc, make, etc.)
Use a virtual environment for development:
# Create and activate venv
python3 -m venv venv
. venv/bin/activate
# Install dependencies
pip install --require-hashes -r requirements.txt
# Install pip-tools for dependency management
pip install pip-toolsTo update dependencies, edit requirements.in and regenerate the lock file:
. venv/bin/activate
pip-compile --generate-hashes requirements.in -o requirements.txt- valkey-search module
perftool for profiling (optional:sudo yum install perforsudo apt-get install linux-tools-generic)bunzip2for dataset extraction (sudo yum install bzip2orsudo apt-get install bzip2)
valkey-perf-benchmark/
├── .github/
│ ├── workflows/ # GitHub Actions workflows
│ │ ├── valkey_benchmark.yml # Continuous core benchmarking (self-hosted runners)
│ │ ├── search_benchmark.yml # Continuous valkey-search module benchmarking
│ │ ├── benchmark_release_tags.yml # Benchmark specific Valkey release tags
│ │ ├── module-benchmark.yml # Module framework smoke test (CI)
│ │ ├── basic.yml # Basic benchmark validation
│ │ ├── cluster_tls.yml # Cluster and TLS specific tests
│ │ ├── tests.yml # Unit test suite (pytest)
│ │ ├── check_format.yml # Code formatting checks (black)
│ │ └── sync-dashboards.yml # Sync Grafana dashboards to the deployment
│ └── workflow-templates/
│ └── pr-benchmark-template.yml # Reusable PR-triggered benchmark template
├── configs/ # Benchmark configuration files
│ ├── benchmark-configs.json # Standard core benchmark configs
│ ├── benchmark-config-arm.json # Core configs tuned for arm hosts
│ ├── benchmark-configs-cluster-tls.json # Cluster + TLS core configs
│ ├── benchmark-config-tag-arm.json # Tag-index benchmark configs (arm)
│ ├── fts-benchmarks-arm.json # Full search test suite (16 groups, arm)
│ ├── fts-benchmarks-shortened-arm.json # Shortened search suite (4 groups, arm)
│ └── module-test-arm.json # Minimal module framework smoke test
├── dashboards/ # Grafana dashboards and AWS infrastructure
│ ├── grafana/ # Grafana dashboard definitions and Helm config
│ ├── kubernetes/ # Kubernetes manifests (ALB Ingress)
│ ├── infrastructure/ # CloudFormation templates
│ ├── scripts/ # Phase-based deployment scripts (00-06)
│ ├── schema.sql # PostgreSQL database schema
│ └── README.md # Deployment documentation
├── utils/ # Utility scripts (Python package)
│ ├── compare_benchmark_results.py # Result comparison, statistics, and graphs
│ ├── postgres_track_commits.py # Commit tracking and management (PostgreSQL)
│ ├── push_to_postgres.py # Push metrics to PostgreSQL
│ ├── detect_regression.py # Detect regressions between latest commits
│ ├── cpu_utils.py # CPU core range parsing and allocation
│ └── git_utils.py # Git ref resolution and commit fetching
├── benchmark.py # Main entry point (core and modules)
├── valkey_build.py # Builds Valkey server from source (ServerBuilder)
├── benchmark_build.py # Builds valkey-benchmark from unstable (BenchmarkBuilder)
├── module_build.py # Builds Valkey module .so files (ModuleBuilder)
├── valkey_server.py # Manages Valkey server instances (ServerLauncher)
├── valkey_benchmark.py # Client-side benchmark execution (ClientRunner) — core + modules
├── profiler.py # Generic performance profiler (flamegraphs)
├── cpu_monitor.py # CPU monitoring during tests
├── per_cpu_monitor.py # Per-CPU monitoring (scheduler issue detection)
├── metrics_sampler.py # 1 Hz per-second sampler (opt-in via per_second_sampling)
├── samplers/ # Pluggable sample sources the sampler selects between
│ ├── base.py # SamplerContext, SampleSource, shared CLI helper
│ ├── valkey_info.py # Raw INFO ALL fields
│ ├── latency_histogram.py # Raw LATENCY HISTOGRAM reply
│ ├── process_cpu.py # Raw /proc/stat and per-thread server CPU ticks (local only)
│ └── disk.py # Raw block device stat counters (local only)
├── process_metrics.py # Parses and formats benchmark results (MetricsProcessor)
├── tests/ # Test suite
│ ├── integration/ # Integration tests (+ README)
│ └── test_*.py # Unit tests (pytest)
├── scripts/ # Helper scripts
│ └── setup_datasets.py # Search dataset + query generator
├── datasets/ # Search test datasets and queries (auto-generated)
│ ├── search_terms.csv
│ └── proximity_phrases.csv
├── requirements.in # Direct dependencies (human-editable)
└── requirements.txt # Locked dependencies with hashes (auto-generated, includes test deps)
Each benchmark run clones a fresh copy of the Valkey repository for the target commit. When --valkey-path is omitted, the repository is cloned into valkey_<commit> and removed after the run to maintain build isolation and repeatability.
# Run server and client benchmarks together with default configuration
python benchmark.py
# Run only the client component against an existing server
python benchmark.py --mode client --target-ip 192.168.1.100
# Use a specific configuration file
python benchmark.py --config ./configs/my-custom-config.json
# Benchmark specific commit(s) or ref(s)
python benchmark.py --commits 1a2b3c4d
# Use a pre-existing Valkey dir
python benchmark.py --valkey-path /path/to/valkey
# Without --valkey-path a directory named valkey_<commit> is cloned and later removed
# Use a custom valkey-benchmark executable
# (when omitted, the latest valkey-benchmark is cloned and built from unstable)
python benchmark.py --valkey-benchmark-path /path/to/custom/valkey-benchmark
# Use a pre-running Valkey server (skips build / launch / cleanup)
python benchmark.py --valkey-path /path/to/valkey --use-running-server--mode accepts both (default — run server and client on the same host) or
client (run only the client against an existing server). There is no separate
server-only mode.
# Compare against a baseline ref
python benchmark.py --commits HEAD --baseline unstable
# Run multiple benchmark runs for statistical reliability
python benchmark.py --commits HEAD --runs 5The project includes a comparison tool for analyzing benchmark results with statistical analysis and graph generation.
# Basic comparison between two result files
python utils/compare_benchmark_results.py --baseline results/commit1/metrics.json --new results/commit2/metrics.json --output comparison.md
# Generate graphs along with comparison
python utils/compare_benchmark_results.py --baseline results/commit1/metrics.json --new results/commit2/metrics.json --output comparison.md --graphs --graph-dir graphs/
# Filter to show only RPS metrics
python utils/compare_benchmark_results.py --baseline results/commit1/metrics.json --new results/commit2/metrics.json --output comparison.md --metrics rps --graphs
# Filter to show only latency metrics
python utils/compare_benchmark_results.py --baseline results/commit1/metrics.json --new results/commit2/metrics.json --output comparison.md --metrics latency --graphs- Automatic Run Averaging: Groups and averages multiple benchmark runs with identical configurations
- Statistical Analysis: Calculates means, standard deviations, and Coefficient of Variation (CV) with sample standard deviation (n-1)
- Coefficient of Variation: Provides normalized variability metrics (CV = σ/μ × 100%) for scale-independent comparison across performance metrics
- Graph Generation: matplotlib-based visualization including:
- Consolidated comparison graphs for all metrics
- Variance line graphs showing individual run values with error bars
- RPS-focused filtering for integration purposes
- Metrics Filtering: Supports filtering by metric type (all, rps, latency)
- Standardized Output: Generates markdown reports with statistical information including CV
- Module Benchmark Support: Uses the module commit as the version identifier and reports the underlying core engine commit (see below)
When comparing module benchmarks (e.g. valkey-search), each result record carries both a module_commit (the module build under test) and a commit (the core Valkey engine it ran against). The comparison tool handles these as follows:
- Version identifier: If
module_commitis present it is used as the version shown in the report title, table headers, and graph legends. Otherwise the tool falls back tocommit, then to the run timestamp. - Core commit line: For module comparisons the report adds a
**Core commit:**line so the core engine version is visible alongside the module version:**Core commit:** <sha>when both sides ran against the same core commit**Core commit:** <baseline-sha> (baseline) → <new-sha> (new)when they differ
When multiple runs are available, the comparison tool displays comprehensive statistical information:
Metric Value (n=X, σ=Y, CV=Z%)
Where:
n: Number of runsσ: Standard deviationCV: Coefficient of Variation as a percentage
The Coefficient of Variation (CV) is useful for:
- Scale-independent comparison: Compares variability across metrics with different units (e.g., RPS vs latency)
- Performance consistency assessment: Lower CV indicates more consistent performance
- Benchmark reliability evaluation: High CV indicates unstable test conditions
- % Change: The relative difference between the two means (e.g.,
+3±2%). The±is the uncertainty range — given the noise in the runs, the true change likely falls within that range (so+3±2%means somewhere between +1% and +5%). - Significance indicators: Determined by whether the 95% confidence intervals of the baseline and new means overlap.
- ✅ — CIs do not overlap, change is in the favorable direction (higher RPS or lower latency)
- ❌ — CIs do not overlap, change is in the unfavorable direction
- ➖ — CIs overlap, the difference cannot be distinguished from run-to-run noise
- ❔ — One or both sides have only a single run (n ≤ 1), so no CI can be computed
A large % change may still be marked ➖: if variance is high — the indicator reflects statistical confidence, not the magnitude of the change.
- Consolidated Comparison Graphs: Single graphs showing all metrics with legend format
{commit}-P{pipeline}/IO{io_threads} - Variance Line Graphs: Individual run values with standard deviation visualization and error bars
# Use an already running Valkey server (client mode only with `--valkey-path`)
python benchmark.py --mode client --valkey-path /path/to/valkey --use-running-server
# Specify custom results directory
python benchmark.py --results-dir ./my-results
# Set logging level
python benchmark.py --log-level DEBUG
# Use a custom valkey-benchmark executable (useful for testing modified versions)
python benchmark.py --valkey-benchmark-path /path/to/custom/valkey-benchmark
# Run only a subset of test groups / scenarios (test_groups configs)
python benchmark.py --config configs/fts-benchmarks-arm.json --module search --groups 1,2 --scenarios a,b
# Run a single cluster mode from a config that lists both [false, true]
python benchmark.py --config configs/fts-benchmarks-arm.json --cluster-mode-filter true
# Skip profiling passes and config_sets for a quick single pass
python benchmark.py --skip-profiling --skip-config-set
# Provide the repository so comparison reports link to commits
python benchmark.py --commits HEAD --baseline unstable --repository valkey-io/valkeyWhen using --use-running-server or benchmarking remote servers, restarting the server between benchmark runs is the user's responsibility. Failure to restart between runs affects test results.
The --valkey-benchmark-path option specifies a custom path to the valkey-benchmark executable. This is useful when:
- Testing a modified version of
valkey-benchmark - Using a pre-built binary from a different location
- Benchmarking with a specific version of the tool
When not specified, the tool clones and builds the latest valkey-benchmark from the Valkey unstable branch, so the benchmark client stays independent of the server commit under test.
# Example: Use a custom benchmark tool
python benchmark.py --valkey-benchmark-path /usr/local/bin/valkey-benchmark
# Example: Use with custom Valkey path
python benchmark.py --valkey-path /custom/valkey --valkey-benchmark-path /custom/valkey/src/valkey-benchmarkCreate benchmark configurations in JSON format. Each object represents a single set of options and configurations are not automatically cross-multiplied. Example:
[
{
"requests": [10000000],
"keyspacelen": [10000000],
"data_sizes": [16, 64, 256],
"pipelines": [1, 10, 100],
"commands": ["SET", "GET"],
"cluster_mode": "yes",
"tls_mode": "yes",
"warmup": 10,
"io-threads": [1, 4, 8],
"server_cpu_range": "0-1",
"client_cpu_range": "2-3",
"custom-server-configs": {
"maxmemory": "4gb",
"timeout": "300"
},
"post_commands": ["INFO memory", "MEMORY STATS"]
}
]Internally, this basic format is compiled into the scenario model at load time: each combination of the list-valued fields (requests x keyspacelen x data_sizes x pipelines x clients x commands) becomes a generated test group containing a single scenario, and the runner executes only the test_groups path. This is purely an internal translation: benchmark invocations and metrics.json outputs are unchanged, and both the basic format and the test_groups format (see Module Config Structure) remain fully supported.
Add server configs the benchmark does not manage (e.g. memory limits, timeouts):
"custom-server-configs": {
"maxmemory": "4gb",
"timeout": "300",
"maxclients": "10000"
}Combine with a baseline .conf file:
"custom-server-config-file": "/etc/valkey/baseline.conf",
"custom-server-configs": {
"hz": "100",
"tcp-keepalive": "60"
}| Parameter | Description | Data Type | Multiple Values |
|---|---|---|---|
requests |
Number of requests to perform | Integer | Yes |
keyspacelen |
Key space size (number of distinct keys) | Integer | Yes |
data_sizes |
Size of data in bytes | Integer | Yes |
pipelines |
Number of commands to pipeline | Integer | Yes |
clients |
Number of concurrent client connections | Integer | Yes |
commands |
Valkey commands to benchmark | String | Yes |
cluster_mode |
Whether to enable cluster mode | String ("yes"/"no") | No |
tls_mode |
Whether to enable TLS | String ("yes"/"no") | No |
warmup |
Warmup time in seconds before benchmarking | Integer | No |
io-threads |
Number of I/O threads for server | Integer | Yes |
server_cpu_range |
CPU cores for server (e.g. "0-3", "0,2,4", or "144-191,48-95") | String | No |
client_cpu_range |
CPU cores for client (e.g. "4-7", "1,3,5", or "0-3,8-11") | String | No |
custom-server-configs |
Additional server configuration options the benchmark does not manage (e.g. {"maxmemory": "4gb", "timeout": "300"}). |
Object (key-value pairs) | No |
custom-server-config-file |
Path to a Valkey-format .conf file passed positionally to valkey-server. Used as a baseline configuration. |
String (path) | No |
build_args |
Extra NAME=value variables passed to make when the framework builds Valkey (e.g. ["BUILD_EXT_STORAGE=yes"]). BUILD_TLS is not allowed because it comes from tls_mode. Each name may appear once. |
List of strings | No |
Note on custom-server-configs: This field lets you pass extra configuration options to the Valkey server at startup (e.g. maxmemory, timeout, maxclients, tcp-keepalive, hz), and to override a benchmark-managed default such as maxmemory-policy.
Precedence: Configs are applied in this order on the valkey-server command line:
custom-server-config-file(positional, parsed first, lowest priority)- Benchmark-managed defaults (
--key valueflags) custom-server-configs(--key valueflags, highest priority)
A key you set in custom-server-configs wins over the benchmark default for that key: the framework skips its own default instead of appending it afterwards. For example, setting "maxmemory-policy": "noeviction" takes effect even though the default is allkeys-lru.
custom-server-configs is for settings the framework does not manage. A key the framework sets from another config field (port, tls_mode, modules, cluster_mode, cluster_nodes, cluster_config_dir) or needs for process management (daemonize, logfile, save, appendonly, protected-mode) is rejected at config validation with a message naming the reason. io-threads may be set through custom-server-configs, but not at the same time as the top-level io-threads field, which sweeps multiple values.
When warmup is provided for read commands, the benchmark performs three stages:
- A data injection pass using the corresponding write command with
--sequentialto seed the keyspace. - A warmup run of the read command (without
--sequential) for the specified duration. - The main benchmark run of the read command.
Supported commands:
"SET", "GET", "RPUSH", "LPUSH", "LPOP", "SADD", "SPOP", "HSET", "GET", "MGET", "LRANGE", "SPOP", "ZPOPMIN"
For module benchmarks (e.g. valkey-search) a scenario-based format is also supported.
Instead of a cross-product of commands x data_sizes x ..., each scenario is an
explicit test with its own command, dataset, clients, duration/requests,
and warmup. Scenarios are grouped under test_groups. Sub-flags of the command
can be expanded via options (e.g. NOCONTENT) into separate variants.
Two scenario types worth calling out:
type: "write"/type: "read"— a single benchmark process against the server. Reads may becluster_execution: "parallel"to fan out one client per cluster node.type: "mixed"— spawns concurrent write and read processes in the same test window. Each sub-scenario listed underwrites: [...]andreads: [...]becomes its ownvalkey-benchmarkprocess on its own CPU range and produces its own metric entry (test_phase: mixed_write/mixed_read). Options declared on a mixed scenario are applied to every read sub-scenario, not to a (non-existent) top-level command.
Datasets consumed by scenarios can be generated ahead of time from a
dataset_generation block in the same config, via scripts/setup_datasets.py.
Supported CSV transforms include wikipedia, inject, repeat, prefix_gen,
proximity_phrase, expansion, fuzzy, numeric_range, and tag_list. The
fuzzy transform pairs with a type: "fuzzy" query generator so a query term
matches variant 0 of its corresponding dataset row and target_distance-edit
variants (insert/delete/substitute) match under fuzzy search.
See configs/module-test-arm.json for a runnable example that includes a
mixed workload and per-cluster-execution options.
post_commands is a list of Valkey commands that runs after a benchmark run
finishes, including when that run failed. It can be used to record server-side
state, such as memory fragmentation, keyspace stats, index stats, INFO
sections, and so on.
It can be set on a scenario or at the config level. A scenario's own list wins; otherwise the config-level list applies.
A failing command is logged as a warning and never fails the benchmark.
In cluster mode the commands are sent to the single port given by port
rather than to every node in cluster_ports.
Each command's output is captured to post_commands.json in the results
directory. The file is a JSON array where each element records the
command and its result:
[
{
"command": "INFO memory",
"results": {
"used_memory": 1048576,
"used_memory_rss": 2097152,
"mem_fragmentation_ratio": 2.0
}
},
{
"command": "DBSIZE",
"results": 1000
}
]Benchmark results are stored in the results/ directory, organized by commit ID:
results/
└── <commit-id>/
├── logs.txt # Benchmark logs
├── metrics.json # Performance metrics in JSON format
└── valkey_log_cluster_disabled.log # Valkey server logs
Sample metrics.json
[
{
"timestamp": "2025-05-28T01:29:42+02:00",
"commit": "ff7135836b5d9ccaa19d5dbaf2a0b0325755c8b4",
"command": "SET",
"data_size": 16,
"pipeline": 10,
"clients": 10000000,
"requests": 10000000,
"rps": 556204.44,
"avg_latency_ms": 0.813,
"min_latency_ms": 0.28,
"p50_latency_ms": 0.775,
"p95_latency_ms": 1.159,
"p99_latency_ms": 1.407,
"max_latency_ms": 2.463,
"cluster_mode": false,
"tls": false
}
]The project includes several GitHub Actions workflows for automated testing and deployment:
valkey_benchmark.yml: Continuous benchmarking workflow that runs on self-hosted EC2 runners- Benchmarks new commits from the Valkey unstable branch
- Manages commit tracking via PostgreSQL
- Uploads results to S3 and pushes metrics to PostgreSQL
- Supports manual triggering with configurable commit limits
search_benchmark.yml: Continuous valkey-search module benchmarking- Runs on a schedule (every 4 hours) and on manual dispatch
- Determines core and module commits, builds valkey-server + valkey-search, and benchmarks
- Configurable module branch (default
main), repeat count, cluster mode, and profiling
benchmark_release_tags.yml: Benchmarks a specified list of Valkey release tagsmodule-benchmark.yml: Module framework smoke test on GitHub runners- Validates the module benchmarking framework with a minimal test
- Uses
configs/module-test-arm.json(small, quick) - Builds valkey-server + valkey-search and runs a quick smoke test
basic.yml: Basic benchmark validationcluster_tls.yml: Tests for cluster and TLS configurationstests.yml: Runs the unit test suite (pytest)check_format.yml: Code formatting validation (black)sync-dashboards.yml: Syncs Grafana dashboard JSON changes to the deployment
The system uses PostgreSQL to track benchmarking progress. Commits are stored in the benchmark_commits table with their status, configuration, and architecture.
in_progress: Workflow has selected the commit and is running the benchmarkcomplete: Full workflow completed - metrics are in PostgreSQL and available in dashboards
The system cleans up in_progress entries on the next workflow run. This ensures:
- Failed benchmark runs are retried
- Commits stuck in progress do not block future runs
- The tracking reflects completed work accurately
Each commit is tracked with the actual configuration content used for benchmarking. This provides:
- Comparison of results across different configurations by actual content
- Tracking of exact config parameters used for each benchmark
- Detection when config content changes even if file name stays the same
- Benchmarking the same commit with different configs
- Reproducibility by storing complete config data
The project includes a complete AWS infrastructure for visualizing benchmark results using Grafana:
- AWS EKS Fargate - Serverless Kubernetes (no EC2 nodes)
- Amazon RDS PostgreSQL - Stores benchmark metrics and Grafana configuration
- CloudFront CDN - Global content delivery with HTTPS
- Application Load Balancer - Secured for CloudFront-only access
See dashboards/README.md for complete deployment guide and architecture details.
utils/postgres_track_commits.py: Manages commit tracking, status updates, and cleanup operations using PostgreSQLutils/push_to_postgres.py: Pushes benchmark metrics to PostgreSQL with dynamic schema supportutils/compare_benchmark_results.py: Compares benchmark results across commits (statistics + graphs)utils/detect_regression.py: Detects performance regressions between the latest two benchmarked commits in PostgreSQLutils/cpu_utils.py: CPU core range parsing and per-node allocation for servers/clientsutils/git_utils.py: Resolves git refs and fetches commits (handles shallow clones)
configs/benchmark-configs.json: Standard benchmark configurationsconfigs/benchmark-configs-cluster-tls.json: Specialized configurations for cluster and TLS testing
For local development, simply run:
python benchmark.pyCode formatting is enforced by CI using black. To format locally:
pip install black==25.1.0
black .The project includes a comprehensive test suite:
- Unit tests: Cover core logic functions (parsing, validation, statistics, metrics processing)
- Integration tests: Validate benchmark workflows with mock components
Tests run without requiring a Valkey server or PostgreSQL.
# Install dependencies (includes test deps)
pip install --require-hashes -r requirements.txt
# Run all tests
python -m pytest tests/ -v
# Run only unit tests
python -m pytest tests/ -v --ignore=tests/integration/
# Run only integration tests
python -m pytest tests/integration/ -v
# Run tests excluding slow tests
python -m pytest tests/ -v -m "not slow"The integration tests (tests/integration/) validate benchmark workflows end-to-end using mock components — no Valkey server, database, or network required. See tests/integration/README.md for details.
Create new JSON configuration files in the configs/ directory following the existing format. Each configuration object represents a benchmark scenario.
The framework is module-agnostic: it never hard-codes anything about
valkey-search. All modules run through the same path — benchmark.py builds and
launches the server (loading the module .so), and ClientRunner in
valkey_benchmark.py executes the scenarios described by the config's
test_groups. --module is simply a label that routes results to
results/{module}_tests/. No Python code changes are required to add a module.
To benchmark a new module (JSON, Bloom, TimeSeries, a custom module, …) you supply three things and reuse everything else (server lifecycle, CPU pinning, profiling, metrics, PostgreSQL upload, PR workflow):
-
Build the module
.sowith its native build system, and note the path:cd ../valkey-json && make # or ./build.sh / cmake — whatever the module uses ls -lh .build-release/libjson.so
-
Author a config using the
test_groups/scenariosstructure (copy the shape ofconfigs/fts-benchmarks-arm.jsonor the smallerconfigs/module-test-arm.json). Point the server at your module via themodulesarray and describe the workload:setup_commands— your module's index/schema setup (search usesFT.CREATE; JSON might need none).command— any command string your module accepts. Placeholders (__rand_int__,__field:NAME__,{tag}) work regardless of module.dataset/dataset_generation— optional; add generated data if your workload needs it (the transforms inscripts/setup_datasets.pyare generic and extensible).
-
Run it through the same entry point search uses:
python benchmark.py \ --module json \ --module-path ../valkey-json/.build-release/libjson.so \ --valkey-path ../valkey \ --config configs/json-benchmarks.json \ --groups 1
Results land in results/json_tests/ and can be compared with
utils/compare_benchmark_results.py and pushed to PostgreSQL with
utils/push_to_postgres.py exactly like search results. The generic
infrastructure (profiler.py, cpu_monitor.py / per_cpu_monitor.py,
process_metrics.py) applies automatically.
To wire a module into continuous CI benchmarking, see Onboarding a New Module to Continuous Benchmarking.
The framework supports generic module testing through a unified ClientRunner.
Module tests use structured test_groups with scenarios:
{
"test_groups": [{
"group": 1,
"scenarios": [
{
"type": "write",
"cluster_execution": "single",
"setup_commands": ["FT.CREATE idx ..."],
"command": "HSET ...",
"dataset": "data.xml",
"clients": 1000,
"maxdocs": 50000
},
{
"type": "read",
"cluster_execution": "parallel",
"command": "FT.SEARCH idx __field:term__",
"dataset": "queries.csv",
"clients": 1000,
"duration": 60,
"warmup": 20,
"post_commands": ["INFO memory", "FT.INFO idx"]
}
]
}],
"cluster_mode": false,
"tls_mode": false
}Scenario fields: Each scenario names its workload with exactly one of test or command (except type: mixed scenarios, which use writes/reads sub-scenarios instead):
| Field | Description |
|---|---|
test |
Predefined valkey-benchmark test name, run as -t NAME. Used by compiled basic-format configs. |
command |
Arbitrary command line, run after --. Supports __rand_int__ placeholders. |
data_size |
Payload size in bytes, passed as -d N when present. |
keyspacelen |
Per-scenario key space size, passed as -r N. Falls back to the config-level keyspacelen[0]. |
warmup_inline |
Adds --warmup N to the main benchmark run. Distinct from warmup, which performs a separate warm-up run first. |
restart_before |
Restarts the managed server before the scenario runs (flushes the database instead when using a running server). |
populate_with |
Write workload that seeds the keyspace before the measured run. For a test scenario it is a predefined write name (e.g. SET), run as -t NAME. For a command scenario it is an arbitrary write command string (e.g. SET key:__rand_int__ __data__), run after --. On a type: mixed scenario a single word must be a predefined write name and runs as -t NAME, and a longer string runs as an arbitrary write command after --. The populate pass runs sequentially and shares the main run's seed, so the measured run hits the seeded keys. It produces no metrics row and no sampled rows. |
populate_clients |
Client count for the populate pass only. Requires populate_with. Defaults to the scenario's clients, or 1 when the scenario has none (a mixed parent carries its client counts on its children). |
populate_benchmark_args |
benchmark_args-shaped list applied to the populate pass only, validated identically. Requires populate_with. The populate pass does not inherit the scenario's benchmark_args, so a load-only flag such as --keysize belongs here and a measured-only flag such as --zipfian stays out of the load. |
populate_retries |
Additional populate attempts when the keyspace comes up short, default 0. Requires populate_with. At 0 the pass runs once and its key count is never checked. Above 0 the summed DBSIZE across the active ports is compared against min(requests, keyspacelen) after every attempt, and a short count (or a nonzero exit, which valkey-benchmark returns when the server rejects writes) sleeps two seconds and runs the pass again. Exhausting the attempts raises. |
benchmark_args |
List of strings passed straight through to valkey-benchmark itself, spliced into the argv before --csv and therefore always left of the -- separator. Each string is shlex-split, so "--zipfian 1.0" becomes two argv elements. Use it for client flags the framework does not model, including flags that only exist in a patched valkey-benchmark supplied via --valkey-benchmark-path. This targets the benchmark client, unlike custom-server-configs, which passes settings to valkey-server. On a type: mixed parent the value is inherited by every writes/reads child that does not set its own. Flags the framework emits itself (-h -p -c -P -d -n -r -t --duration --seed --sequential --threads --warmup --cluster --tls --csv, the TLS certificate flags and the dataset flags) are rejected at config validation; use the corresponding scenario field instead. |
benchmark_args |
List of strings passed straight through to valkey-benchmark itself, spliced into the argv before --csv and therefore always left of the -- separator. Each string is shlex-split, so "--zipfian 1.0" becomes two argv elements. Use it for client flags the framework does not model, including flags that only exist in a patched valkey-benchmark supplied via --valkey-benchmark-path. This targets the benchmark client, unlike custom-server-configs, which passes settings to valkey-server. On a type: mixed parent the value is inherited by every writes/reads child that does not set its own. Flags the framework emits itself (-h -p -c -P -d -n -r -t --duration --seed --sequential --threads --warmup --cluster --tls --csv, the TLS certificate flags and the dataset flags) are rejected at config validation; use the corresponding scenario field instead. |
Multi-node clusters: Config with cluster_mode array runs both single-node and distributed tests:
{
"cluster_mode": [false, true],
"cluster_nodes": 5,
"cluster_ports": [6379, 6380, 6381, 6382, 6383],
"cpu_allocation": {
"cores_per_server": 8,
"cores_per_client": 8,
"servers": ["0-7", "8-15", "16-23", "24-31", "32-39"],
"clients": ["40-47", "48-55", "56-63", "64-71", "72-79"]
},
"modules": [
{
"path": "../valkey-search/.build-release/libsearch.so",
"startup_args": ["--use-coordinator"]
}
]
}Module Loading: Supports loading multiple modules with per-module startup arguments:
{
"modules": [
{
"path": "../valkey-search/.build-release/libsearch.so",
"startup_args": ["--use-coordinator", "--timeout", "30"]
},
{
"path": "../valkey-json/.build-release/libjson.so",
"startup_args": []
}
]
}CPU Allocation: Two methods (mutually exclusive):
-
New (recommended):
cpu_allocationobject-
Automatic: Provide
cores_per_server+cores_per_client"cpu_allocation": { "cores_per_server": 8, "cores_per_client": 8 }
Framework calculates ranges: servers [0-7, 8-15, 16-23...], clients [40-47, 48-55...]
The client pool holds one range per server node, or one range per mixed client process when a
type: mixedscenario launches more processes than there are nodes. Eachwrites/readschild is its own process wanting a fullcores_per_clientslice, so a single-node config with one write and one read child gets two client ranges:cores_per_server8 withcores_per_client24 gives servers0-7and clients8-31and32-55. The first ranges stay the per-node ones, so cluster pinning is unchanged. -
Manual override: Provide
servers+clientsarrays"cpu_allocation": { "servers": ["0-7", "8-15", "16-23", "24-31", "32-39"], "clients": ["40-47", "48-55", "56-63", "64-71", "72-79"] }
Explicit ranges used as-is (cores_per_* ignored if present)
A core appearing in both arrays is rejected at config validation.
-
-
Old:
server_cpu_range+client_cpu_range(single-node only)
CME Parallel Execution: Scenarios can specify execution strategy:
"cluster_execution": "single"- One client with cluster routing (default)"cluster_execution": "parallel"- N clients (one per node), aggregated metrics- Optional:
"parallel_clients": 10- Custom client count
Filter modes: --cluster-mode-filter [false|true] runs specific mode only.
Key pattern: Use {tag} in HSET for cluster routing: HSET rd0-{tag}:__rand_int__
A module team (valkey-json, valkey-bloom, a custom module, …) can reuse this
framework's continuous benchmarking workflow without forking it. The reference
implementation is .github/workflows/search_benchmark.yml: it runs on a schedule
(and manual dispatch), benchmarks new module commits over time, stores results in
PostgreSQL, and surfaces them in Grafana. Model your module's workflow on it.
What the workflow does, step by step (adapt each to your module):
- Checks out the module repo, valkey core (
unstable), and this benchmark repo. - Builds valkey-server, the
valkey-benchmarkclient, and the module.so(search uses./build.sh→.build-release/libsearch.so; use your module's build system and.sopath). - Determines which module commits still need benchmarking via
utils/postgres_track_commits.py determineand marks themin_progress. - For each commit: checkout + build the module, then run
python benchmark.py --module <name> --module-path <.so> --valkey-path <core> --config <config> .... - Pushes results to PostgreSQL with
utils/push_to_postgres.pyand marks the commitscomplete.
Infrastructure it needs:
- A self-hosted runner (the search workflow uses
[self-hosted, valkey-search-arm]) sized for your benchmark, with build tools andperfinstalled. - PostgreSQL + AWS creds (RDS with IAM auth in the search setup) provided as repo secrets.
- A config using the
test_groups/scenariosstructure with your module'ssetup_commands,commandtemplates, and datasets — see Adapting the Framework for a New Module. - A Grafana dashboard (optional) — add a JSON under
dashboards/grafana/and it is picked up bysync-dashboards.yml.
Both utils/push_to_postgres.py and utils/postgres_track_commits.py take a
--table identifier that decides which metrics table your results go into —
this is how each module keeps its data separate. resolve_table_name() maps it:
--table value |
Resulting table |
|---|---|
core (default) |
benchmark_metrics |
tag |
benchmark_tags_metrics |
any other id (e.g. search, json) |
benchmark_metrics_{table} (e.g. benchmark_metrics_search) |
The identifier must match ^[a-z][a-z0-9_]{0,30}$. So a new module typically
passes its own name, e.g. --table json → results land in benchmark_metrics_json.
A separate --test-type flag (e.g. core, fts) is not used for table
naming — it is stored on each row as a tag for filtering/grouping in dashboards.
python utils/push_to_postgres.py \
--results-dir ./results/<core_sha>_<module_sha> \
--table json --test-type json \
--host <rds-endpoint> --port 5432 \
--database <db> --username <user> --password "$PGPASSWORD"Use the same --table value in postgres_track_commits.py (for commit
tracking) and push_to_postgres.py (for results) so tracking and metrics stay
aligned.
A module that only wants local/manual runs needs none of this CI infrastructure —
just build the .so and run benchmark.py as shown in
Running Module Tests Locally.
Both valkey core and module repositories can set up automated PR benchmarking using our unified workflow template.
-
Copy the template from this repo:
cp .github/workflow-templates/pr-benchmark-template.yml \ .github/workflows/benchmark-on-label.yml
-
Customize for your repository:
- Runner label: Update
runs-onwith your self-hosted runner label - For module repos: Update
MODULE_NAME,.sopath, and build commands - For core repo: Remove or comment out module-specific steps
- Runner label: Update
-
Trigger benchmarks by adding the
run-benchmarklabel to any PR
The template includes clear CUSTOMIZE markers for:
Module Repositories (valkey-search, valkey-json, etc.):
- Module build command (build.sh, make, cmake)
- Path to .so file (e.g.,
.build-release/libsearch.so) - Module name for
--moduleparameter - Benchmark config file path
Core Repository (valkey/valkey):
- Uses conditional steps based on
github.repository - Most sections work without modification
See .github/workflow-templates/pr-benchmark-template.yml for detailed inline documentation.
- Triggered by
run-benchmarklabel on PRs - Compares PR branch against base branch
- Posts results as PR comment
- Uploads artifacts for detailed analysis
- Automatic cleanup and label removal
--module-path requires a pre-built .so file (not source directory) since modules use different build systems (make, cmake, build.sh) and may need specific compilers.
Build module first:
cd valkey-search
make BUILD_TLS=yes # or ./build.sh, cmake, etc.
ls -lh .build-release/libsearch.soRun benchmarks:
python benchmark.py \
--module search \
--module-path ../valkey-search/.build-release/libsearch.so \
--valkey-path ../valkey \
--config configs/fts-benchmarks-arm.json \
--groups 1Framework manages server lifecycle automatically.
Note: Datasets are automatically generated on first run if missing. The initial run may take 30-60 minutes to download Wikipedia and generate datasets. Subsequent runs use cached datasets and start immediately.
Use benchmark.py with --module search:
# Run FTS test Group 1 (datasets auto-generated if missing)
python benchmark.py \
--module search \
--valkey-path /path/to/valkey \
--config configs/fts-benchmarks-arm.json \
--groups 1
# Run specific scenarios within groups
python benchmark.py \
--module search \
--valkey-path /path/to/valkey \
--config configs/fts-benchmarks-arm.json \
--scenarios a,b
# Filter by groups (works with any config using test_groups structure)
python benchmark.py \
--module search \
--valkey-path /path/to/valkey \
--config configs/fts-benchmarks-arm.json \
--groups 1,2Results saved to results/search_tests/ with optional flamegraphs if profiling enabled in config.
Note: --groups and --scenarios work with any configuration using the test_groups structure (core or module tests).
CPU pinning is configured in the config file:
{
"server_cpu_range": "0-7", // Pin server to cores 0-7 (when using --mode both)
"client_cpu_range": "8-15" // Pin benchmark client to cores 8-15
}With --mode both (recommended):
Framework manages server automatically with CPU pinning from config.
python benchmark.py \
--module search \
--module-path ../valkey-search/.build-release/libsearch.so \
--valkey-path ../valkey \
--config configs/fts-benchmarks-arm.json \
--groups 1With --use-running-server (manual server management):
Start server yourself with desired CPU pinning.
# Start server manually
taskset -c 0-7 /path/to/valkey-server --loadmodule libsearch.so ...
# Run benchmarks
python benchmark.py \
--module search \
--valkey-path /path/to/valkey \
--use-running-server \
--config configs/fts-benchmarks-arm.json \
--groups 1The framework uses a transform-based dataset generation system that supports multiple testing strategies:
Config Format:
"dataset_generation": {
"dataset_name.xml": {
"doc_count": 50000,
"fields": [
{
"name": "field0",
"size": 100,
"transforms": [
{"type": "wikipedia"},
{"type": "inject", "term": "MARKER_TERM", "percentage": 0.5}
]
}
]
}
}Supported Transforms:
wikipedia: Extract Wikipedia content (base text)inject: Add marker terms at specified percentagerepeat: Duplicate terms N times (for term_repetition tests)prefix_gen: Generate prefix variations (for prefix_explosion tests)proximity_phrase: Generate N-term phrases for proximity testing- Parameters:
term_count,combinations(1=best case, 100=worst case),repeats(copies per pattern) - Supports CSV output (no Wikipedia needed)
- Parameters:
expansion: Generate wildcard-expansion variants (term001_a, term001_aa, ...) grouped asexpansion_count×docs_per_expansioncopies ×term_countbase termsfuzzy: Generate misspelled variants for fuzzy-match tests- Parameters:
variant_count,docs_per_variant,term_count,min_word_length,max_word_length,target_distance - Variant 0 is the correctly-spelled base word (matches
type: "fuzzy"queries); variants ≥1 applytarget_distanceLevenshtein edits (insert/delete/substitute)
- Parameters:
numeric_range: Random numeric value in[min, max](for NUMERIC index tests)tag_list: Pipe-separated random tags from a supplied list (for TAG index tests)vector: Marks a vector field. Vector data lives in a structured.npyfile (not CSV); the presence of this transform routes generation to the structured NPY writer. Used for KNN / HNSW vector-search tests, including hybrid configs that combine a vector field with text/numeric/tag fields.
Compact Format (for field explosion):
"field_explosion_50k.xml": {
"doc_count": 50000,
"generate_fields": {
"count": 50,
"size": 1000,
"transforms": [{"type": "wikipedia"}]
}
}Query Generation:
"query_generation": {
"proximity_5term_queries.csv": {
"type": "proximity_phrase",
"doc_count": 100,
"term_count": 5
}
}- Auto-generates query CSVs matching ingestion datasets
- Supports type-based generation (extensible for future query types)
Supported query types:
proximity_phrase— multi-column phrase terms matching aproximity_phrasetransformprefix/suffix— substring queries extracted from an existing source CSVexpansion— zero-padded base terms (e.g.term001) matching anexpansiontransformfuzzy— deterministic base words matching variant 0 of afuzzytransform, so every query has known matches in the dataset (parameters:min_word_length,max_word_length)tag_only— rotates through a suppliedtagslist to produce category filters for composed TAG queriesvector— generates a structured.npyof(search_term, query_vector)pairs (L2-normalized random vectors, filename-seeded for reproducibility) plus a companion CSV of the search terms, for KNN / hybrid vector queries. Parameter:dimensions(default 256)
configs/fts-benchmarks-arm.json (the FTS Performance Benchmarks suite)
defines the full search suite of 16 groups. Groups are selected with
--groups and individual scenarios within a group with --scenarios. Each group
begins with a write scenario that ingests its dataset, followed by the read (and
occasionally mixed) scenarios that query it.
| Group | Focus |
|---|---|
| 1 | Multi-field comprehensive (STEM enabled) |
| 2 | Proximity queries — 5-term best case (1 combination) |
| 3 | Proximity queries — 5-term worst case (100 combinations) |
| 4 | Proximity queries — 25-term worst case (100 combinations) |
| 5 | Prefix (wildcard) expansion — best case |
| 6 | Prefix (wildcard) expansion — worst case |
| 7 | Suffix expansion — best case |
| 8 | Suffix expansion — worst case |
| 9 | Hybrid queries (TEXT + NUMERIC + TAG) |
| 10 | Fuzzy matching — best case (distance 1) |
| 11 | Fuzzy matching — worst case (distance 3) |
| 12 | Posting tests — position map partitions and byte size |
| 13 | Radix tests — node growth and prefix locality |
| 14 | Misc tests — string intern, schema options, extreme case |
| 15 | Vector + text hybrid (KNN with filters) |
| 16 | Composed queries — nominal case (entries fetcher test) |
configs/fts-benchmarks-shortened-arm.json (the FTS Shortened Performance Benchmarks suite) is a smaller, CI-sized version of the full suite. It is the
default config for the continuous search_benchmark.yml workflow and is sized
to finish quickly while still exercising every major search path (text, hybrid,
and both vector index types), across 4 groups:
| Group | Focus |
|---|---|
| P1 | Text single-field baseline |
| P2 | Multi-index hybrid (TEXT + NUMERIC + TAG) |
| P3 | Vector FLAT (prefilter path) |
| P4 | Vector HNSW (UsePreFiltering ratio branch) |
Results are saved to results/search_tests/:
metrics.json- Performance metricsflamegraphs/- Profiling data (if enabled)
Sample metrics structure (module runs add test_id, group/scenario, their
descriptions, and — when supplied — module_commit / module_commit_timestamp):
{
"test_id": "1_b",
"test_phase": "read",
"group": 1,
"scenario": "b",
"group_description": "Multi-field comprehensive (STEM enabled)",
"scenario_description": "Single term all fields",
"status": "success",
"rps": 6596.42,
"avg_latency_ms": 7.535,
"p50_latency_ms": 5.367,
"p95_latency_ms": 20.975,
"cpu_avg_percent": 692.84,
"cpu_peak_percent": 751.10,
"memory_mb": 4469.87,
"dataset": "field_explosion_50k.xml",
"module_commit": "..."
}test_id is {group}_{scenario} and test_phase is the scenario type
(read, write, or mixed_read / mixed_write for mixed scenarios).
configs/tiering.json ports the Valkey data-tiering benchmarks as one group per
test. Group 1 is Zipfian 80/20: 4M keys sequentially loaded under a 1gb DRAM cap
with FlashCache tiering, then a 60 second mixed run of 40 SET clients and 160 GET
clients over a zipfian alpha 1.0 key distribution, pinned to 8 server cores and
two 24-core client ranges with per-second sampling on. The other groups change
one thing each:
| Group | Test | Change |
|---|---|---|
| 2 | Uniform 80/20 | Every key equally likely, no --zipfian |
| 3 | Balanced 50/50 | 100 GET and 100 SET clients |
| 4 | Zipfian TTL 120 | Every write sets EX 120, including the load, and the run lasts 180 seconds so keys expire during it. The 100 byte keys are written out in the commands, since --keysize only applies to built-in tests |
| 5 | Value sizes | One scenario each for 500 B and 5 KB values under a 512mb cap with a 10gb tiering file. Each keyspace is sized so about 10% of the values fit in memory |
Group 5 needs different server settings, so it is a second entry in the file.
One call runs every group, and --groups or --scenarios runs a subset.
Requirements:
- A valkey-data-tiering server build. The
ext-storage-*settings incustom-server-configsdo not exist in upstreamvalkey-server. - The patched
valkey-benchmarkfrom the same source, passed with--valkey-benchmark-path.--zipfianand--keysizeare not upstream, so a stock binary rejects the client invocation. - An NVMe-backed filesystem holding the
ext-storage-pathdirectory. - libaio, which the FlashCache backend links against.
- At least 56 CPU cores, covering the
0-7server range and the8-31and32-55client ranges.
The config sets build_args to ["BUILD_EXT_STORAGE=yes"], so a framework
build compiles the tiering code.
ext-storage-path is the directory FlashCache keeps flashcache.db in.
FlashCache creates the directory and the file at ext-storage-capacity when
they are missing. Each time the framework starts the server, it first removes
flashcache.db from that directory when custom-server-configs sets
ext-storage-enabled to yes, so every run starts with an empty tiering file.
A relative path is taken from the Valkey directory, or from dir when
custom-server-configs sets it. If the path exists and is not a directory, the
run stops with an error. Nothing is changed when you use --use-running-server.
python benchmark.py \
--mode both \
--valkey-path /path/to/valkey-data-tiering \
--valkey-benchmark-path /path/to/valkey-data-tiering/src/valkey-benchmark \
--config configs/tiering.json \
--commits HEADThe load uses populate_retries because a 4M key load under a 1gb cap is
rejected intermittently while eviction catches up, so the pass is repeated until
DBSIZE reaches the keyspace size.
metrics_sampler.py reads the server and the host once per second during a scenario's
measured benchmark process and records what each source read, unprocessed. Deriving rates,
percentiles and typed values is left to ingest. Sampling starts after any inline
warmup_inline, so elapsed_sec 0 is the first measured second. Tick k is taken at
k intervals after the first, and a tick that overruns skips the slots it missed.
| Source | Records |
|---|---|
valkey_info |
every INFO ALL field as a string |
latency_histogram |
the cumulative LATENCY HISTOGRAM reply: per command calls and histogram_usec, server-side execution time in power-of-two microsecond buckets |
process_cpu |
the cpu lines of /proc/stat as proc_stat, and per server thread its comm, utime and stime in clock ticks as threads, keyed by tid |
disk |
the backing block device and its raw /sys/block/<dev>/stat counters as stat |
process_cpu and disk read /proc and /sys, so they are collected only on Linux and
only when the server runs on the sampling machine. disk samples the whole disk backing its
path option, else custom-server-configs.ext-storage-path, else the server's
CONFIG GET dir.
It is opt-in. The root config key per_second_sampling takes either form:
"per_second_sampling": true"per_second_sampling": {
"sources": {
"valkey_info": {},
"disk": {"path": "/mnt/nvme"}
},
"cpu_range": "0-1"
}true samples every source with default options, unpinned. sources maps each selected
source to its options object, and cpu_range pins the sampler thread.
The sampler reads the server through one persistent client per scenario, with the same TLS
settings as the benchmark when tls_mode is on. Each read times out after 2 seconds.
Rows from every scenario, run and config set are appended to
results/<commit>/timeseries.jsonl, one JSON object per line. Each row holds the same
identity fields as a metrics.json row (commit, repository, cluster_mode, tls,
test_id, group, scenario, config_set, config_name, module_commit and so on),
plus run, profiling_set, sample_time (ISO 8601 UTC), elapsed_sec, and one key per
source holding its reading. As in metrics.json, timestamp is the commit time. A source
that fails a tick is absent from that row.
Each JSONL row is one line; this example is wrapped for readability:
{
"commit": "HEAD",
"repository": "valkey",
"test_id": "1_b",
"scenario": "b",
"config_set": {},
"run": 1,
"sample_time": "2026-10-06T01:26:13.915+00:00",
"elapsed_sec": 1,
"valkey_info": {"used_memory": "1950880"},
"latency_histogram": {"get": {"calls": 90887, "histogram_usec": {"1": 90873}}},
"process_cpu": {"proc_stat": {"cpu": [1768215, 220728, 1396351]}, "threads": {"22858": {"comm": "valkey-server", "utime": 121, "stime": 379}}},
"disk": {"device": "nvme0n1", "stat": [5664582, 1117339, 86138375]}
}utils/push_to_postgres.py pushes these rows together with metrics.json. With
--table X they go to benchmark_timeseries_X, so --table tiering fills the
benchmark_timeseries_tiering table used by the valkey-tiering dashboard. See
dashboards/README.md.
The framework includes a generic profiler that works with both core and FTS tests.
This framework uses The FlameGraph project by Brendan Gregg for performance visualization. FlameGraph is licensed under CDDL 1.0 (Common Development and Distribution License). The FlameGraph scripts are automatically downloaded from the source repository when profiling is enabled.
The PerformanceProfiler class can be integrated into any benchmark script:
from profiler import PerformanceProfiler
# Initialize profiler
profiler = PerformanceProfiler(results_dir, enabled=True)
# Example from search module:
profiler.start_profiling("search_1a", target_process="valkey-server")
# Run your benchmark
runner.run_benchmark_config()
# After benchmark completes
profiler.stop_profiling("search_1a")
# → Generates:
# - flamegraph: results_dir/commit_id/flamegraphs/search_1a_20251218_080245.svg
# - perf report: results_dir/commit_id/flamegraphs/search_1a_20251218_080245_report.txt
# - raw data: results_dir/commit_id/flamegraphs/search_1a_20251218_080245.perf.data- Flamegraph generation: Visual call stack analysis
- Auto-downloads scripts: Fetches flamegraph tools from GitHub on first use
- Profiling modes: cpu (cycles) or wall-time (all execution time)
- Function hotspot analysis: Identify CPU-intensive code paths
- Kernel + user space profiling: Complete stack traces with DWARF
- Generic implementation: Works with any process (valkey-server, redis-server, etc.)
- Configurable sampling: 999Hz default
Run tests with multiple profiling configurations:
{
"profiling_sets": [
{"enabled": false},
{"enabled": true, "mode": "wall-time", "sampling_freq": 999}
],
"config_sets": [
{"search.reader-threads": 1},
{"search.reader-threads": 8}
]
}Behavior:
- Iterates profiling_sets × config_sets
- Profiling OFF → Collects metrics in
metrics.json - Profiling ON → Generates flamegraphs, skips metrics
- Flamegraphs:
group{X}_{scenario}_{config_values}_{timestamp}.svg
Per-scenario override:
{"id": "a", "profiling": {"delays": {"search": {"delay": 0, "duration": 10}}}}Scenario overrides any profiling_set values.
Delay strategy pattern:
{
"profiling_sets": [{
"enabled": true,
"delays": {
"write": {"delay": 0, "duration": 10},
"read": {"delay": 30, "duration": 10}
}
}],
"scenarios": [{
"id": "a",
"type": "write",
"profiling": {"delays": {"write": {"delay": 10, "duration": 10}}}
}]
}Group 1 write scenario uses 10s delay (dataset loading), others use 0s (immediate).
Please see the LICENSE.md