Skip to content

About

A comprehensive benchmarking tool for Valkey, an in-memory data structure store. This tool allows you to measure performance across different configurations, including TLS and cluster modes.

Resources

Security policy

Stars

14 stars

Watchers

0 watching

Forks

Latest commit

 

History

60 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Valkey Performance Benchmark

A benchmarking tool for Valkey, an in-memory data store. Measures performance across different commits and configurations, including TLS and cluster modes.

Features

  • Benchmarks Valkey server with various commands (SET, GET, RPUSH, etc.)
  • Tests with different data sizes and pipeline configurations
  • Supports TLS and cluster mode testing
  • Handles automatic server setup and teardown
  • Collects detailed performance metrics and generates reports
  • Compares performance between different Valkey versions/commits
  • Provides CPU pinning via taskset using configuration file settings
  • Runs continuous benchmarking via GitHub Actions workflow
  • Tracks commits and manages progress automatically
  • Includes Grafana dashboards for visualizing performance metrics
  • Search module performance testing with valkey search, adaptable to other modules
  • Performance profiling with flamegraph generation via perf
  • Regression detection comparing the latest benchmarked commits in PostgreSQL

Prerequisites

  • Git
  • Python 3.6+
  • Linux environment (for taskset CPU pinning)
  • Build tools required by Valkey (gcc, make, etc.)

Development Setup

Use a virtual environment for development:

# Create and activate venv
python3 -m venv venv
. venv/bin/activate

# Install dependencies
pip install --require-hashes -r requirements.txt

# Install pip-tools for dependency management
pip install pip-tools

Updating Dependencies

To update dependencies, edit requirements.in and regenerate the lock file:

. venv/bin/activate
pip-compile --generate-hashes requirements.in -o requirements.txt

Additional Prerequisites for Search Tests

  • valkey-search module
  • perf tool for profiling (optional: sudo yum install perf or sudo apt-get install linux-tools-generic)
  • bunzip2 for dataset extraction (sudo yum install bzip2 or sudo apt-get install bzip2)

Project Structure

valkey-perf-benchmark/
├── .github/
│   ├── workflows/                    # GitHub Actions workflows
│   │   ├── valkey_benchmark.yml      # Continuous core benchmarking (self-hosted runners)
│   │   ├── search_benchmark.yml      # Continuous valkey-search module benchmarking
│   │   ├── benchmark_release_tags.yml # Benchmark specific Valkey release tags
│   │   ├── module-benchmark.yml      # Module framework smoke test (CI)
│   │   ├── basic.yml                 # Basic benchmark validation
│   │   ├── cluster_tls.yml           # Cluster and TLS specific tests
│   │   ├── tests.yml                 # Unit test suite (pytest)
│   │   ├── check_format.yml          # Code formatting checks (black)
│   │   └── sync-dashboards.yml       # Sync Grafana dashboards to the deployment
│   └── workflow-templates/
│       └── pr-benchmark-template.yml # Reusable PR-triggered benchmark template
├── configs/                  # Benchmark configuration files
│   ├── benchmark-configs.json           # Standard core benchmark configs
│   ├── benchmark-config-arm.json        # Core configs tuned for arm hosts
│   ├── benchmark-configs-cluster-tls.json # Cluster + TLS core configs
│   ├── benchmark-config-tag-arm.json    # Tag-index benchmark configs (arm)
│   ├── fts-benchmarks-arm.json          # Full search test suite (16 groups, arm)
│   ├── fts-benchmarks-shortened-arm.json # Shortened search suite (4 groups, arm)
│   └── module-test-arm.json             # Minimal module framework smoke test
├── dashboards/              # Grafana dashboards and AWS infrastructure
│   ├── grafana/            # Grafana dashboard definitions and Helm config
│   ├── kubernetes/         # Kubernetes manifests (ALB Ingress)
│   ├── infrastructure/     # CloudFormation templates
│   ├── scripts/            # Phase-based deployment scripts (00-06)
│   ├── schema.sql          # PostgreSQL database schema
│   └── README.md           # Deployment documentation
├── utils/                   # Utility scripts (Python package)
│   ├── compare_benchmark_results.py # Result comparison, statistics, and graphs
│   ├── postgres_track_commits.py    # Commit tracking and management (PostgreSQL)
│   ├── push_to_postgres.py          # Push metrics to PostgreSQL
│   ├── detect_regression.py         # Detect regressions between latest commits
│   ├── cpu_utils.py                 # CPU core range parsing and allocation
│   └── git_utils.py                 # Git ref resolution and commit fetching
├── benchmark.py             # Main entry point (core and modules)
├── valkey_build.py          # Builds Valkey server from source (ServerBuilder)
├── benchmark_build.py       # Builds valkey-benchmark from unstable (BenchmarkBuilder)
├── module_build.py          # Builds Valkey module .so files (ModuleBuilder)
├── valkey_server.py         # Manages Valkey server instances (ServerLauncher)
├── valkey_benchmark.py      # Client-side benchmark execution (ClientRunner) — core + modules
├── profiler.py              # Generic performance profiler (flamegraphs)
├── cpu_monitor.py           # CPU monitoring during tests
├── per_cpu_monitor.py       # Per-CPU monitoring (scheduler issue detection)
├── metrics_sampler.py       # 1 Hz per-second sampler (opt-in via per_second_sampling)
├── samplers/                # Pluggable sample sources the sampler selects between
│   ├── base.py             # SamplerContext, SampleSource, shared CLI helper
│   ├── valkey_info.py      # Raw INFO ALL fields
│   ├── latency_histogram.py # Raw LATENCY HISTOGRAM reply
│   ├── process_cpu.py      # Raw /proc/stat and per-thread server CPU ticks (local only)
│   └── disk.py             # Raw block device stat counters (local only)
├── process_metrics.py       # Parses and formats benchmark results (MetricsProcessor)
├── tests/                   # Test suite
│   ├── integration/        # Integration tests (+ README)
│   └── test_*.py           # Unit tests (pytest)
├── scripts/                 # Helper scripts
│   └── setup_datasets.py   # Search dataset + query generator
├── datasets/                # Search test datasets and queries (auto-generated)
│   ├── search_terms.csv
│   └── proximity_phrases.csv
├── requirements.in          # Direct dependencies (human-editable)
└── requirements.txt         # Locked dependencies with hashes (auto-generated, includes test deps)

Each benchmark run clones a fresh copy of the Valkey repository for the target commit. When --valkey-path is omitted, the repository is cloned into valkey_<commit> and removed after the run to maintain build isolation and repeatability.

Usage

Basic Usage

# Run server and client benchmarks together with default configuration
python benchmark.py

# Run only the client component against an existing server
python benchmark.py --mode client --target-ip 192.168.1.100

# Use a specific configuration file
python benchmark.py --config ./configs/my-custom-config.json

# Benchmark specific commit(s) or ref(s)
python benchmark.py --commits 1a2b3c4d

# Use a pre-existing Valkey dir
python benchmark.py --valkey-path /path/to/valkey

# Without --valkey-path a directory named valkey_<commit> is cloned and later removed

# Use a custom valkey-benchmark executable
# (when omitted, the latest valkey-benchmark is cloned and built from unstable)
python benchmark.py --valkey-benchmark-path /path/to/custom/valkey-benchmark

# Use a pre-running Valkey server (skips build / launch / cleanup)
python benchmark.py --valkey-path /path/to/valkey --use-running-server

--mode accepts both (default — run server and client on the same host) or client (run only the client against an existing server). There is no separate server-only mode.

Comparison Mode

# Compare against a baseline ref
python benchmark.py --commits HEAD --baseline unstable

# Run multiple benchmark runs for statistical reliability
python benchmark.py --commits HEAD --runs 5

Benchmark Comparison and Analysis

The project includes a comparison tool for analyzing benchmark results with statistical analysis and graph generation.

Compare Benchmark Results

# Basic comparison between two result files
python utils/compare_benchmark_results.py --baseline results/commit1/metrics.json --new results/commit2/metrics.json --output comparison.md

# Generate graphs along with comparison
python utils/compare_benchmark_results.py --baseline results/commit1/metrics.json --new results/commit2/metrics.json --output comparison.md --graphs --graph-dir graphs/

# Filter to show only RPS metrics
python utils/compare_benchmark_results.py --baseline results/commit1/metrics.json --new results/commit2/metrics.json --output comparison.md --metrics rps --graphs

# Filter to show only latency metrics
python utils/compare_benchmark_results.py --baseline results/commit1/metrics.json --new results/commit2/metrics.json --output comparison.md --metrics latency --graphs

Comparison Tool Features

  • Automatic Run Averaging: Groups and averages multiple benchmark runs with identical configurations
  • Statistical Analysis: Calculates means, standard deviations, and Coefficient of Variation (CV) with sample standard deviation (n-1)
  • Coefficient of Variation: Provides normalized variability metrics (CV = σ/μ × 100%) for scale-independent comparison across performance metrics
  • Graph Generation: matplotlib-based visualization including:
    • Consolidated comparison graphs for all metrics
    • Variance line graphs showing individual run values with error bars
    • RPS-focused filtering for integration purposes
  • Metrics Filtering: Supports filtering by metric type (all, rps, latency)
  • Standardized Output: Generates markdown reports with statistical information including CV
  • Module Benchmark Support: Uses the module commit as the version identifier and reports the underlying core engine commit (see below)

Module Benchmark Comparisons

When comparing module benchmarks (e.g. valkey-search), each result record carries both a module_commit (the module build under test) and a commit (the core Valkey engine it ran against). The comparison tool handles these as follows:

  • Version identifier: If module_commit is present it is used as the version shown in the report title, table headers, and graph legends. Otherwise the tool falls back to commit, then to the run timestamp.
  • Core commit line: For module comparisons the report adds a **Core commit:** line so the core engine version is visible alongside the module version:
    • **Core commit:** <sha> when both sides ran against the same core commit
    • **Core commit:** <baseline-sha> (baseline) → <new-sha> (new) when they differ

Statistical Display Format

When multiple runs are available, the comparison tool displays comprehensive statistical information:

Metric Value (n=X, σ=Y, CV=Z%)

Where:

  • n: Number of runs
  • σ: Standard deviation
  • CV: Coefficient of Variation as a percentage

The Coefficient of Variation (CV) is useful for:

  • Scale-independent comparison: Compares variability across metrics with different units (e.g., RPS vs latency)
  • Performance consistency assessment: Lower CV indicates more consistent performance
  • Benchmark reliability evaluation: High CV indicates unstable test conditions

Interpreting Significance

  • % Change: The relative difference between the two means (e.g., +3±2%). The ± is the uncertainty range — given the noise in the runs, the true change likely falls within that range (so +3±2% means somewhere between +1% and +5%).
  • Significance indicators: Determined by whether the 95% confidence intervals of the baseline and new means overlap.
    • ✅ — CIs do not overlap, change is in the favorable direction (higher RPS or lower latency)
    • ❌ — CIs do not overlap, change is in the unfavorable direction
    • ➖ — CIs overlap, the difference cannot be distinguished from run-to-run noise
    • ❔ — One or both sides have only a single run (n ≤ 1), so no CI can be computed

A large % change may still be marked ➖: if variance is high — the indicator reflects statistical confidence, not the magnitude of the change.

Graph Types

  1. Consolidated Comparison Graphs: Single graphs showing all metrics with legend format {commit}-P{pipeline}/IO{io_threads}
  2. Variance Line Graphs: Individual run values with standard deviation visualization and error bars

Advanced Options

# Use an already running Valkey server (client mode only with `--valkey-path`)
python benchmark.py --mode client --valkey-path /path/to/valkey --use-running-server

# Specify custom results directory
python benchmark.py --results-dir ./my-results

# Set logging level
python benchmark.py --log-level DEBUG

# Use a custom valkey-benchmark executable (useful for testing modified versions)
python benchmark.py --valkey-benchmark-path /path/to/custom/valkey-benchmark

# Run only a subset of test groups / scenarios (test_groups configs)
python benchmark.py --config configs/fts-benchmarks-arm.json --module search --groups 1,2 --scenarios a,b

# Run a single cluster mode from a config that lists both [false, true]
python benchmark.py --config configs/fts-benchmarks-arm.json --cluster-mode-filter true

# Skip profiling passes and config_sets for a quick single pass
python benchmark.py --skip-profiling --skip-config-set

# Provide the repository so comparison reports link to commits
python benchmark.py --commits HEAD --baseline unstable --repository valkey-io/valkey

Benchmarking Remote Servers and Pre-Running Servers

When using --use-running-server or benchmarking remote servers, restarting the server between benchmark runs is the user's responsibility. Failure to restart between runs affects test results.

Custom Valkey-Benchmark Executable

The --valkey-benchmark-path option specifies a custom path to the valkey-benchmark executable. This is useful when:

  • Testing a modified version of valkey-benchmark
  • Using a pre-built binary from a different location
  • Benchmarking with a specific version of the tool

When not specified, the tool clones and builds the latest valkey-benchmark from the Valkey unstable branch, so the benchmark client stays independent of the server commit under test.

# Example: Use a custom benchmark tool
python benchmark.py --valkey-benchmark-path /usr/local/bin/valkey-benchmark

# Example: Use with custom Valkey path
python benchmark.py --valkey-path /custom/valkey --valkey-benchmark-path /custom/valkey/src/valkey-benchmark

Configuration

Create benchmark configurations in JSON format. Each object represents a single set of options and configurations are not automatically cross-multiplied. Example:

[
  {
    "requests": [10000000],
    "keyspacelen": [10000000],
    "data_sizes": [16, 64, 256],
    "pipelines": [1, 10, 100],
    "commands": ["SET", "GET"],
    "cluster_mode": "yes",
    "tls_mode": "yes",
    "warmup": 10,
    "io-threads": [1, 4, 8],
    "server_cpu_range": "0-1",
    "client_cpu_range": "2-3",
    "custom-server-configs": {
      "maxmemory": "4gb",
      "timeout": "300"
    },
    "post_commands": ["INFO memory", "MEMORY STATS"]
  }
]

Internally, this basic format is compiled into the scenario model at load time: each combination of the list-valued fields (requests x keyspacelen x data_sizes x pipelines x clients x commands) becomes a generated test group containing a single scenario, and the runner executes only the test_groups path. This is purely an internal translation: benchmark invocations and metrics.json outputs are unchanged, and both the basic format and the test_groups format (see Module Config Structure) remain fully supported.

Custom Server Configs Examples

Add server configs the benchmark does not manage (e.g. memory limits, timeouts):

"custom-server-configs": {
  "maxmemory": "4gb",
  "timeout": "300",
  "maxclients": "10000"
}

Combine with a baseline .conf file:

"custom-server-config-file": "/etc/valkey/baseline.conf",
"custom-server-configs": {
  "hz": "100",
  "tcp-keepalive": "60"
}

Configuration Parameters

Parameter Description Data Type Multiple Values
requests Number of requests to perform Integer Yes
keyspacelen Key space size (number of distinct keys) Integer Yes
data_sizes Size of data in bytes Integer Yes
pipelines Number of commands to pipeline Integer Yes
clients Number of concurrent client connections Integer Yes
commands Valkey commands to benchmark String Yes
cluster_mode Whether to enable cluster mode String ("yes"/"no") No
tls_mode Whether to enable TLS String ("yes"/"no") No
warmup Warmup time in seconds before benchmarking Integer No
io-threads Number of I/O threads for server Integer Yes
server_cpu_range CPU cores for server (e.g. "0-3", "0,2,4", or "144-191,48-95") String No
client_cpu_range CPU cores for client (e.g. "4-7", "1,3,5", or "0-3,8-11") String No
custom-server-configs Additional server configuration options the benchmark does not manage (e.g. {"maxmemory": "4gb", "timeout": "300"}). Object (key-value pairs) No
custom-server-config-file Path to a Valkey-format .conf file passed positionally to valkey-server. Used as a baseline configuration. String (path) No
build_args Extra NAME=value variables passed to make when the framework builds Valkey (e.g. ["BUILD_EXT_STORAGE=yes"]). BUILD_TLS is not allowed because it comes from tls_mode. Each name may appear once. List of strings No

Note on custom-server-configs: This field lets you pass extra configuration options to the Valkey server at startup (e.g. maxmemory, timeout, maxclients, tcp-keepalive, hz), and to override a benchmark-managed default such as maxmemory-policy.

Precedence: Configs are applied in this order on the valkey-server command line:

  1. custom-server-config-file (positional, parsed first, lowest priority)
  2. Benchmark-managed defaults (--key value flags)
  3. custom-server-configs (--key value flags, highest priority)

A key you set in custom-server-configs wins over the benchmark default for that key: the framework skips its own default instead of appending it afterwards. For example, setting "maxmemory-policy": "noeviction" takes effect even though the default is allkeys-lru.

custom-server-configs is for settings the framework does not manage. A key the framework sets from another config field (port, tls_mode, modules, cluster_mode, cluster_nodes, cluster_config_dir) or needs for process management (daemonize, logfile, save, appendonly, protected-mode) is rejected at config validation with a message naming the reason. io-threads may be set through custom-server-configs, but not at the same time as the top-level io-threads field, which sweeps multiple values.

When warmup is provided for read commands, the benchmark performs three stages:

  1. A data injection pass using the corresponding write command with --sequential to seed the keyspace.
  2. A warmup run of the read command (without --sequential) for the specified duration.
  3. The main benchmark run of the read command.

Supported commands:

"SET", "GET", "RPUSH", "LPUSH", "LPOP", "SADD", "SPOP", "HSET", "GET", "MGET", "LRANGE", "SPOP", "ZPOPMIN"

Scenario-based Configurations (test_groups)

For module benchmarks (e.g. valkey-search) a scenario-based format is also supported. Instead of a cross-product of commands x data_sizes x ..., each scenario is an explicit test with its own command, dataset, clients, duration/requests, and warmup. Scenarios are grouped under test_groups. Sub-flags of the command can be expanded via options (e.g. NOCONTENT) into separate variants.

Two scenario types worth calling out:

  • type: "write" / type: "read" — a single benchmark process against the server. Reads may be cluster_execution: "parallel" to fan out one client per cluster node.
  • type: "mixed" — spawns concurrent write and read processes in the same test window. Each sub-scenario listed under writes: [...] and reads: [...] becomes its own valkey-benchmark process on its own CPU range and produces its own metric entry (test_phase: mixed_write / mixed_read). Options declared on a mixed scenario are applied to every read sub-scenario, not to a (non-existent) top-level command.

Datasets consumed by scenarios can be generated ahead of time from a dataset_generation block in the same config, via scripts/setup_datasets.py. Supported CSV transforms include wikipedia, inject, repeat, prefix_gen, proximity_phrase, expansion, fuzzy, numeric_range, and tag_list. The fuzzy transform pairs with a type: "fuzzy" query generator so a query term matches variant 0 of its corresponding dataset row and target_distance-edit variants (insert/delete/substitute) match under fuzzy search.

See configs/module-test-arm.json for a runnable example that includes a mixed workload and per-cluster-execution options.

Post-run Commands (post_commands)

post_commands is a list of Valkey commands that runs after a benchmark run finishes, including when that run failed. It can be used to record server-side state, such as memory fragmentation, keyspace stats, index stats, INFO sections, and so on.

It can be set on a scenario or at the config level. A scenario's own list wins; otherwise the config-level list applies.

A failing command is logged as a warning and never fails the benchmark.

In cluster mode the commands are sent to the single port given by port rather than to every node in cluster_ports.

Each command's output is captured to post_commands.json in the results directory. The file is a JSON array where each element records the command and its result:

[
  {
    "command": "INFO memory",
    "results": {
      "used_memory": 1048576,
      "used_memory_rss": 2097152,
      "mem_fragmentation_ratio": 2.0
    }
  },
  {
    "command": "DBSIZE",
    "results": 1000
  }
]

Results

Benchmark results are stored in the results/ directory, organized by commit ID:

results/
└── <commit-id>/
    ├── logs.txt                         # Benchmark logs
    ├── metrics.json                     # Performance metrics in JSON format
    └── valkey_log_cluster_disabled.log  # Valkey server logs

Sample metrics.json

[
  {
    "timestamp": "2025-05-28T01:29:42+02:00",
    "commit": "ff7135836b5d9ccaa19d5dbaf2a0b0325755c8b4",
    "command": "SET",
    "data_size": 16,
    "pipeline": 10,
    "clients": 10000000,
    "requests": 10000000,
    "rps": 556204.44,
    "avg_latency_ms": 0.813,
    "min_latency_ms": 0.28,
    "p50_latency_ms": 0.775,
    "p95_latency_ms": 1.159,
    "p99_latency_ms": 1.407,
    "max_latency_ms": 2.463,
    "cluster_mode": false,
    "tls": false
  }
]

Continuous Benchmarking & CI/CD

GitHub Actions Workflows

The project includes several GitHub Actions workflows for automated testing and deployment:

  • valkey_benchmark.yml: Continuous benchmarking workflow that runs on self-hosted EC2 runners
    • Benchmarks new commits from the Valkey unstable branch
    • Manages commit tracking via PostgreSQL
    • Uploads results to S3 and pushes metrics to PostgreSQL
    • Supports manual triggering with configurable commit limits
  • search_benchmark.yml: Continuous valkey-search module benchmarking
    • Runs on a schedule (every 4 hours) and on manual dispatch
    • Determines core and module commits, builds valkey-server + valkey-search, and benchmarks
    • Configurable module branch (default main), repeat count, cluster mode, and profiling
  • benchmark_release_tags.yml: Benchmarks a specified list of Valkey release tags
  • module-benchmark.yml: Module framework smoke test on GitHub runners
    • Validates the module benchmarking framework with a minimal test
    • Uses configs/module-test-arm.json (small, quick)
    • Builds valkey-server + valkey-search and runs a quick smoke test
  • basic.yml: Basic benchmark validation
  • cluster_tls.yml: Tests for cluster and TLS configurations
  • tests.yml: Runs the unit test suite (pytest)
  • check_format.yml: Code formatting validation (black)
  • sync-dashboards.yml: Syncs Grafana dashboard JSON changes to the deployment

Commit Tracking

The system uses PostgreSQL to track benchmarking progress. Commits are stored in the benchmark_commits table with their status, configuration, and architecture.

Status Descriptions

  • in_progress: Workflow has selected the commit and is running the benchmark
  • complete: Full workflow completed - metrics are in PostgreSQL and available in dashboards

Automatic Cleanup

The system cleans up in_progress entries on the next workflow run. This ensures:

  • Failed benchmark runs are retried
  • Commits stuck in progress do not block future runs
  • The tracking reflects completed work accurately

Config Tracking

Each commit is tracked with the actual configuration content used for benchmarking. This provides:

  • Comparison of results across different configurations by actual content
  • Tracking of exact config parameters used for each benchmark
  • Detection when config content changes even if file name stays the same
  • Benchmarking the same commit with different configs
  • Reproducibility by storing complete config data

Performance Dashboard

The project includes a complete AWS infrastructure for visualizing benchmark results using Grafana:

  • AWS EKS Fargate - Serverless Kubernetes (no EC2 nodes)
  • Amazon RDS PostgreSQL - Stores benchmark metrics and Grafana configuration
  • CloudFront CDN - Global content delivery with HTTPS
  • Application Load Balancer - Secured for CloudFront-only access

See dashboards/README.md for complete deployment guide and architecture details.

Utilities

Commit Management

  • utils/postgres_track_commits.py: Manages commit tracking, status updates, and cleanup operations using PostgreSQL
  • utils/push_to_postgres.py: Pushes benchmark metrics to PostgreSQL with dynamic schema support
  • utils/compare_benchmark_results.py: Compares benchmark results across commits (statistics + graphs)
  • utils/detect_regression.py: Detects performance regressions between the latest two benchmarked commits in PostgreSQL
  • utils/cpu_utils.py: CPU core range parsing and per-node allocation for servers/clients
  • utils/git_utils.py: Resolves git refs and fetches commits (handles shallow clones)

Configuration Files

  • configs/benchmark-configs.json: Standard benchmark configurations
  • configs/benchmark-configs-cluster-tls.json: Specialized configurations for cluster and TLS testing

Development

Local Development

For local development, simply run:

python benchmark.py

Code formatting is enforced by CI using black. To format locally:

pip install black==25.1.0
black .

Running Tests

The project includes a comprehensive test suite:

  • Unit tests: Cover core logic functions (parsing, validation, statistics, metrics processing)
  • Integration tests: Validate benchmark workflows with mock components

Tests run without requiring a Valkey server or PostgreSQL.

# Install dependencies (includes test deps)
pip install --require-hashes -r requirements.txt

# Run all tests
python -m pytest tests/ -v

# Run only unit tests
python -m pytest tests/ -v --ignore=tests/integration/

# Run only integration tests
python -m pytest tests/integration/ -v

# Run tests excluding slow tests
python -m pytest tests/ -v -m "not slow"

Integration Tests

The integration tests (tests/integration/) validate benchmark workflows end-to-end using mock components — no Valkey server, database, or network required. See tests/integration/README.md for details.

Adding New Configurations

Create new JSON configuration files in the configs/ directory following the existing format. Each configuration object represents a benchmark scenario.

Adapting the Framework for a New Module

The framework is module-agnostic: it never hard-codes anything about valkey-search. All modules run through the same path — benchmark.py builds and launches the server (loading the module .so), and ClientRunner in valkey_benchmark.py executes the scenarios described by the config's test_groups. --module is simply a label that routes results to results/{module}_tests/. No Python code changes are required to add a module.

To benchmark a new module (JSON, Bloom, TimeSeries, a custom module, …) you supply three things and reuse everything else (server lifecycle, CPU pinning, profiling, metrics, PostgreSQL upload, PR workflow):

  1. Build the module .so with its native build system, and note the path:

    cd ../valkey-json && make      # or ./build.sh / cmake — whatever the module uses
    ls -lh .build-release/libjson.so
  2. Author a config using the test_groups / scenarios structure (copy the shape of configs/fts-benchmarks-arm.json or the smaller configs/module-test-arm.json). Point the server at your module via the modules array and describe the workload:

    • setup_commands — your module's index/schema setup (search uses FT.CREATE; JSON might need none).
    • command — any command string your module accepts. Placeholders (__rand_int__, __field:NAME__, {tag}) work regardless of module.
    • dataset / dataset_generation — optional; add generated data if your workload needs it (the transforms in scripts/setup_datasets.py are generic and extensible).
  3. Run it through the same entry point search uses:

    python benchmark.py \
      --module json \
      --module-path ../valkey-json/.build-release/libjson.so \
      --valkey-path ../valkey \
      --config configs/json-benchmarks.json \
      --groups 1

Results land in results/json_tests/ and can be compared with utils/compare_benchmark_results.py and pushed to PostgreSQL with utils/push_to_postgres.py exactly like search results. The generic infrastructure (profiler.py, cpu_monitor.py / per_cpu_monitor.py, process_metrics.py) applies automatically.

To wire a module into continuous CI benchmarking, see Onboarding a New Module to Continuous Benchmarking.

Module Testing (FTS, JSON, etc.)

The framework supports generic module testing through a unified ClientRunner.

Module Config Structure

Module tests use structured test_groups with scenarios:

{
  "test_groups": [{
    "group": 1,
    "scenarios": [
      {
        "type": "write",
        "cluster_execution": "single",
        "setup_commands": ["FT.CREATE idx ..."],
        "command": "HSET ...",
        "dataset": "data.xml",
        "clients": 1000,
        "maxdocs": 50000
      },
      {
        "type": "read",
        "cluster_execution": "parallel",
        "command": "FT.SEARCH idx __field:term__",
        "dataset": "queries.csv",
        "clients": 1000,
        "duration": 60,
        "warmup": 20,
        "post_commands": ["INFO memory", "FT.INFO idx"]
      }
    ]
  }],
  "cluster_mode": false,
  "tls_mode": false
}

Scenario fields: Each scenario names its workload with exactly one of test or command (except type: mixed scenarios, which use writes/reads sub-scenarios instead):

Field Description
test Predefined valkey-benchmark test name, run as -t NAME. Used by compiled basic-format configs.
command Arbitrary command line, run after --. Supports __rand_int__ placeholders.
data_size Payload size in bytes, passed as -d N when present.
keyspacelen Per-scenario key space size, passed as -r N. Falls back to the config-level keyspacelen[0].
warmup_inline Adds --warmup N to the main benchmark run. Distinct from warmup, which performs a separate warm-up run first.
restart_before Restarts the managed server before the scenario runs (flushes the database instead when using a running server).
populate_with Write workload that seeds the keyspace before the measured run. For a test scenario it is a predefined write name (e.g. SET), run as -t NAME. For a command scenario it is an arbitrary write command string (e.g. SET key:__rand_int__ __data__), run after --. On a type: mixed scenario a single word must be a predefined write name and runs as -t NAME, and a longer string runs as an arbitrary write command after --. The populate pass runs sequentially and shares the main run's seed, so the measured run hits the seeded keys. It produces no metrics row and no sampled rows.
populate_clients Client count for the populate pass only. Requires populate_with. Defaults to the scenario's clients, or 1 when the scenario has none (a mixed parent carries its client counts on its children).
populate_benchmark_args benchmark_args-shaped list applied to the populate pass only, validated identically. Requires populate_with. The populate pass does not inherit the scenario's benchmark_args, so a load-only flag such as --keysize belongs here and a measured-only flag such as --zipfian stays out of the load.
populate_retries Additional populate attempts when the keyspace comes up short, default 0. Requires populate_with. At 0 the pass runs once and its key count is never checked. Above 0 the summed DBSIZE across the active ports is compared against min(requests, keyspacelen) after every attempt, and a short count (or a nonzero exit, which valkey-benchmark returns when the server rejects writes) sleeps two seconds and runs the pass again. Exhausting the attempts raises.
benchmark_args List of strings passed straight through to valkey-benchmark itself, spliced into the argv before --csv and therefore always left of the -- separator. Each string is shlex-split, so "--zipfian 1.0" becomes two argv elements. Use it for client flags the framework does not model, including flags that only exist in a patched valkey-benchmark supplied via --valkey-benchmark-path. This targets the benchmark client, unlike custom-server-configs, which passes settings to valkey-server. On a type: mixed parent the value is inherited by every writes/reads child that does not set its own. Flags the framework emits itself (-h -p -c -P -d -n -r -t --duration --seed --sequential --threads --warmup --cluster --tls --csv, the TLS certificate flags and the dataset flags) are rejected at config validation; use the corresponding scenario field instead.
benchmark_args List of strings passed straight through to valkey-benchmark itself, spliced into the argv before --csv and therefore always left of the -- separator. Each string is shlex-split, so "--zipfian 1.0" becomes two argv elements. Use it for client flags the framework does not model, including flags that only exist in a patched valkey-benchmark supplied via --valkey-benchmark-path. This targets the benchmark client, unlike custom-server-configs, which passes settings to valkey-server. On a type: mixed parent the value is inherited by every writes/reads child that does not set its own. Flags the framework emits itself (-h -p -c -P -d -n -r -t --duration --seed --sequential --threads --warmup --cluster --tls --csv, the TLS certificate flags and the dataset flags) are rejected at config validation; use the corresponding scenario field instead.

Cluster Mode Support

Multi-node clusters: Config with cluster_mode array runs both single-node and distributed tests:

{
  "cluster_mode": [false, true],
  "cluster_nodes": 5,
  "cluster_ports": [6379, 6380, 6381, 6382, 6383],
  "cpu_allocation": {
    "cores_per_server": 8,
    "cores_per_client": 8,
    "servers": ["0-7", "8-15", "16-23", "24-31", "32-39"],
    "clients": ["40-47", "48-55", "56-63", "64-71", "72-79"]
  },
  "modules": [
    {
      "path": "../valkey-search/.build-release/libsearch.so",
      "startup_args": ["--use-coordinator"]
    }
  ]
}

Module Loading: Supports loading multiple modules with per-module startup arguments:

{
  "modules": [
    {
      "path": "../valkey-search/.build-release/libsearch.so",
      "startup_args": ["--use-coordinator", "--timeout", "30"]
    },
    {
      "path": "../valkey-json/.build-release/libjson.so",
      "startup_args": []
    }
  ]
}

CPU Allocation: Two methods (mutually exclusive):

  1. New (recommended): cpu_allocation object

    • Automatic: Provide cores_per_server + cores_per_client

      "cpu_allocation": {
        "cores_per_server": 8,
        "cores_per_client": 8
      }

      Framework calculates ranges: servers [0-7, 8-15, 16-23...], clients [40-47, 48-55...]

      The client pool holds one range per server node, or one range per mixed client process when a type: mixed scenario launches more processes than there are nodes. Each writes/reads child is its own process wanting a full cores_per_client slice, so a single-node config with one write and one read child gets two client ranges: cores_per_server 8 with cores_per_client 24 gives servers 0-7 and clients 8-31 and 32-55. The first ranges stay the per-node ones, so cluster pinning is unchanged.

    • Manual override: Provide servers + clients arrays

      "cpu_allocation": {
        "servers": ["0-7", "8-15", "16-23", "24-31", "32-39"],
        "clients": ["40-47", "48-55", "56-63", "64-71", "72-79"]
      }

      Explicit ranges used as-is (cores_per_* ignored if present)

      A core appearing in both arrays is rejected at config validation.

  2. Old: server_cpu_range + client_cpu_range (single-node only)

CME Parallel Execution: Scenarios can specify execution strategy:

  • "cluster_execution": "single" - One client with cluster routing (default)
  • "cluster_execution": "parallel" - N clients (one per node), aggregated metrics
  • Optional: "parallel_clients": 10 - Custom client count

Filter modes: --cluster-mode-filter [false|true] runs specific mode only.
Key pattern: Use {tag} in HSET for cluster routing: HSET rd0-{tag}:__rand_int__

Onboarding a New Module to Continuous Benchmarking

A module team (valkey-json, valkey-bloom, a custom module, …) can reuse this framework's continuous benchmarking workflow without forking it. The reference implementation is .github/workflows/search_benchmark.yml: it runs on a schedule (and manual dispatch), benchmarks new module commits over time, stores results in PostgreSQL, and surfaces them in Grafana. Model your module's workflow on it.

What the workflow does, step by step (adapt each to your module):

  1. Checks out the module repo, valkey core (unstable), and this benchmark repo.
  2. Builds valkey-server, the valkey-benchmark client, and the module .so (search uses ./build.sh → .build-release/libsearch.so; use your module's build system and .so path).
  3. Determines which module commits still need benchmarking via utils/postgres_track_commits.py determine and marks them in_progress.
  4. For each commit: checkout + build the module, then run python benchmark.py --module <name> --module-path <.so> --valkey-path <core> --config <config> ....
  5. Pushes results to PostgreSQL with utils/push_to_postgres.py and marks the commits complete.

Infrastructure it needs:

  • A self-hosted runner (the search workflow uses [self-hosted, valkey-search-arm]) sized for your benchmark, with build tools and perf installed.
  • PostgreSQL + AWS creds (RDS with IAM auth in the search setup) provided as repo secrets.
  • A config using the test_groups / scenarios structure with your module's setup_commands, command templates, and datasets — see Adapting the Framework for a New Module.
  • A Grafana dashboard (optional) — add a JSON under dashboards/grafana/ and it is picked up by sync-dashboards.yml.

PostgreSQL Table Selection (--table)

Both utils/push_to_postgres.py and utils/postgres_track_commits.py take a --table identifier that decides which metrics table your results go into — this is how each module keeps its data separate. resolve_table_name() maps it:

--table value Resulting table
core (default) benchmark_metrics
tag benchmark_tags_metrics
any other id (e.g. search, json) benchmark_metrics_{table} (e.g. benchmark_metrics_search)

The identifier must match ^[a-z][a-z0-9_]{0,30}$. So a new module typically passes its own name, e.g. --table json → results land in benchmark_metrics_json.

A separate --test-type flag (e.g. core, fts) is not used for table naming — it is stored on each row as a tag for filtering/grouping in dashboards.

python utils/push_to_postgres.py \
  --results-dir ./results/<core_sha>_<module_sha> \
  --table json --test-type json \
  --host <rds-endpoint> --port 5432 \
  --database <db> --username <user> --password "$PGPASSWORD"

Use the same --table value in postgres_track_commits.py (for commit tracking) and push_to_postgres.py (for results) so tracking and metrics stay aligned.

A module that only wants local/manual runs needs none of this CI infrastructure — just build the .so and run benchmark.py as shown in Running Module Tests Locally.

Setting Up PR Benchmarks

Both valkey core and module repositories can set up automated PR benchmarking using our unified workflow template.

Quick Start

  1. Copy the template from this repo:

    cp .github/workflow-templates/pr-benchmark-template.yml \
       .github/workflows/benchmark-on-label.yml
  2. Customize for your repository:

    • Runner label: Update runs-on with your self-hosted runner label
    • For module repos: Update MODULE_NAME, .so path, and build commands
    • For core repo: Remove or comment out module-specific steps
  3. Trigger benchmarks by adding the run-benchmark label to any PR

Template Customization Guide

The template includes clear CUSTOMIZE markers for:

Module Repositories (valkey-search, valkey-json, etc.):

  • Module build command (build.sh, make, cmake)
  • Path to .so file (e.g., .build-release/libsearch.so)
  • Module name for --module parameter
  • Benchmark config file path

Core Repository (valkey/valkey):

  • Uses conditional steps based on github.repository
  • Most sections work without modification

See .github/workflow-templates/pr-benchmark-template.yml for detailed inline documentation.

Workflow Features

  • Triggered by run-benchmark label on PRs
  • Compares PR branch against base branch
  • Posts results as PR comment
  • Uploads artifacts for detailed analysis
  • Automatic cleanup and label removal

Running Module Tests Locally

--module-path requires a pre-built .so file (not source directory) since modules use different build systems (make, cmake, build.sh) and may need specific compilers.

Build module first:

cd valkey-search
make BUILD_TLS=yes  # or ./build.sh, cmake, etc.
ls -lh .build-release/libsearch.so

Run benchmarks:

python benchmark.py \
  --module search \
  --module-path ../valkey-search/.build-release/libsearch.so \
  --valkey-path ../valkey \
  --config configs/fts-benchmarks-arm.json \
  --groups 1

Framework manages server lifecycle automatically.

Running FTS Tests

Note: Datasets are automatically generated on first run if missing. The initial run may take 30-60 minutes to download Wikipedia and generate datasets. Subsequent runs use cached datasets and start immediately.

Unified Entry Point (Recommended)

Use benchmark.py with --module search:

# Run FTS test Group 1 (datasets auto-generated if missing)
python benchmark.py \
  --module search \
  --valkey-path /path/to/valkey \
  --config configs/fts-benchmarks-arm.json \
  --groups 1

# Run specific scenarios within groups
python benchmark.py \
  --module search \
  --valkey-path /path/to/valkey \
  --config configs/fts-benchmarks-arm.json \
  --scenarios a,b

# Filter by groups (works with any config using test_groups structure)
python benchmark.py \
  --module search \
  --valkey-path /path/to/valkey \
  --config configs/fts-benchmarks-arm.json \
  --groups 1,2

Results saved to results/search_tests/ with optional flamegraphs if profiling enabled in config.

Note: --groups and --scenarios work with any configuration using the test_groups structure (core or module tests).

CPU Pinning Configuration

CPU pinning is configured in the config file:

{
  "server_cpu_range": "0-7",   // Pin server to cores 0-7 (when using --mode both)
  "client_cpu_range": "8-15"   // Pin benchmark client to cores 8-15
}

With --mode both (recommended): Framework manages server automatically with CPU pinning from config.

python benchmark.py \
  --module search \
  --module-path ../valkey-search/.build-release/libsearch.so \
  --valkey-path ../valkey \
  --config configs/fts-benchmarks-arm.json \
  --groups 1

With --use-running-server (manual server management): Start server yourself with desired CPU pinning.

# Start server manually
taskset -c 0-7 /path/to/valkey-server --loadmodule libsearch.so ...

# Run benchmarks
python benchmark.py \
  --module search \
  --valkey-path /path/to/valkey \
  --use-running-server \
  --config configs/fts-benchmarks-arm.json \
  --groups 1

Dataset Generation System

The framework uses a transform-based dataset generation system that supports multiple testing strategies:

Config Format:

"dataset_generation": {
  "dataset_name.xml": {
    "doc_count": 50000,
    "fields": [
      {
        "name": "field0",
        "size": 100,
        "transforms": [
          {"type": "wikipedia"},
          {"type": "inject", "term": "MARKER_TERM", "percentage": 0.5}
        ]
      }
    ]
  }
}

Supported Transforms:

  • wikipedia: Extract Wikipedia content (base text)
  • inject: Add marker terms at specified percentage
  • repeat: Duplicate terms N times (for term_repetition tests)
  • prefix_gen: Generate prefix variations (for prefix_explosion tests)
  • proximity_phrase: Generate N-term phrases for proximity testing
    • Parameters: term_count, combinations (1=best case, 100=worst case), repeats (copies per pattern)
    • Supports CSV output (no Wikipedia needed)
  • expansion: Generate wildcard-expansion variants (term001_a, term001_aa, ...) grouped as expansion_count × docs_per_expansion copies × term_count base terms
  • fuzzy: Generate misspelled variants for fuzzy-match tests
    • Parameters: variant_count, docs_per_variant, term_count, min_word_length, max_word_length, target_distance
    • Variant 0 is the correctly-spelled base word (matches type: "fuzzy" queries); variants ≥1 apply target_distance Levenshtein edits (insert/delete/substitute)
  • numeric_range: Random numeric value in [min, max] (for NUMERIC index tests)
  • tag_list: Pipe-separated random tags from a supplied list (for TAG index tests)
  • vector: Marks a vector field. Vector data lives in a structured .npy file (not CSV); the presence of this transform routes generation to the structured NPY writer. Used for KNN / HNSW vector-search tests, including hybrid configs that combine a vector field with text/numeric/tag fields.

Compact Format (for field explosion):

"field_explosion_50k.xml": {
  "doc_count": 50000,
  "generate_fields": {
    "count": 50,
    "size": 1000,
    "transforms": [{"type": "wikipedia"}]
  }
}

Query Generation:

"query_generation": {
  "proximity_5term_queries.csv": {
    "type": "proximity_phrase",
    "doc_count": 100,
    "term_count": 5
  }
}
  • Auto-generates query CSVs matching ingestion datasets
  • Supports type-based generation (extensible for future query types)

Supported query types:

  • proximity_phrase — multi-column phrase terms matching a proximity_phrase transform
  • prefix / suffix — substring queries extracted from an existing source CSV
  • expansion — zero-padded base terms (e.g. term001) matching an expansion transform
  • fuzzy — deterministic base words matching variant 0 of a fuzzy transform, so every query has known matches in the dataset (parameters: min_word_length, max_word_length)
  • tag_only — rotates through a supplied tags list to produce category filters for composed TAG queries
  • vector — generates a structured .npy of (search_term, query_vector) pairs (L2-normalized random vectors, filename-seeded for reproducibility) plus a companion CSV of the search terms, for KNN / hybrid vector queries. Parameter: dimensions (default 256)

Search Test Groups

configs/fts-benchmarks-arm.json (the FTS Performance Benchmarks suite) defines the full search suite of 16 groups. Groups are selected with --groups and individual scenarios within a group with --scenarios. Each group begins with a write scenario that ingests its dataset, followed by the read (and occasionally mixed) scenarios that query it.

Group Focus
1 Multi-field comprehensive (STEM enabled)
2 Proximity queries — 5-term best case (1 combination)
3 Proximity queries — 5-term worst case (100 combinations)
4 Proximity queries — 25-term worst case (100 combinations)
5 Prefix (wildcard) expansion — best case
6 Prefix (wildcard) expansion — worst case
7 Suffix expansion — best case
8 Suffix expansion — worst case
9 Hybrid queries (TEXT + NUMERIC + TAG)
10 Fuzzy matching — best case (distance 1)
11 Fuzzy matching — worst case (distance 3)
12 Posting tests — position map partitions and byte size
13 Radix tests — node growth and prefix locality
14 Misc tests — string intern, schema options, extreme case
15 Vector + text hybrid (KNN with filters)
16 Composed queries — nominal case (entries fetcher test)

Shortened Search Suite

configs/fts-benchmarks-shortened-arm.json (the FTS Shortened Performance Benchmarks suite) is a smaller, CI-sized version of the full suite. It is the default config for the continuous search_benchmark.yml workflow and is sized to finish quickly while still exercising every major search path (text, hybrid, and both vector index types), across 4 groups:

Group Focus
P1 Text single-field baseline
P2 Multi-index hybrid (TEXT + NUMERIC + TAG)
P3 Vector FLAT (prefilter path)
P4 Vector HNSW (UsePreFiltering ratio branch)

Search Results

Results are saved to results/search_tests/:

  • metrics.json - Performance metrics
  • flamegraphs/ - Profiling data (if enabled)

Sample metrics structure (module runs add test_id, group/scenario, their descriptions, and — when supplied — module_commit / module_commit_timestamp):

{
  "test_id": "1_b",
  "test_phase": "read",
  "group": 1,
  "scenario": "b",
  "group_description": "Multi-field comprehensive (STEM enabled)",
  "scenario_description": "Single term all fields",
  "status": "success",
  "rps": 6596.42,
  "avg_latency_ms": 7.535,
  "p50_latency_ms": 5.367,
  "p95_latency_ms": 20.975,
  "cpu_avg_percent": 692.84,
  "cpu_peak_percent": 751.10,
  "memory_mb": 4469.87,
  "dataset": "field_explosion_50k.xml",
  "module_commit": "..."
}

test_id is {group}_{scenario} and test_phase is the scenario type (read, write, or mixed_read / mixed_write for mixed scenarios).

Data tiering benchmarks

configs/tiering.json ports the Valkey data-tiering benchmarks as one group per test. Group 1 is Zipfian 80/20: 4M keys sequentially loaded under a 1gb DRAM cap with FlashCache tiering, then a 60 second mixed run of 40 SET clients and 160 GET clients over a zipfian alpha 1.0 key distribution, pinned to 8 server cores and two 24-core client ranges with per-second sampling on. The other groups change one thing each:

Group Test Change
2 Uniform 80/20 Every key equally likely, no --zipfian
3 Balanced 50/50 100 GET and 100 SET clients
4 Zipfian TTL 120 Every write sets EX 120, including the load, and the run lasts 180 seconds so keys expire during it. The 100 byte keys are written out in the commands, since --keysize only applies to built-in tests
5 Value sizes One scenario each for 500 B and 5 KB values under a 512mb cap with a 10gb tiering file. Each keyspace is sized so about 10% of the values fit in memory

Group 5 needs different server settings, so it is a second entry in the file. One call runs every group, and --groups or --scenarios runs a subset.

Requirements:

  • A valkey-data-tiering server build. The ext-storage-* settings in custom-server-configs do not exist in upstream valkey-server.
  • The patched valkey-benchmark from the same source, passed with --valkey-benchmark-path. --zipfian and --keysize are not upstream, so a stock binary rejects the client invocation.
  • An NVMe-backed filesystem holding the ext-storage-path directory.
  • libaio, which the FlashCache backend links against.
  • At least 56 CPU cores, covering the 0-7 server range and the 8-31 and 32-55 client ranges.

The config sets build_args to ["BUILD_EXT_STORAGE=yes"], so a framework build compiles the tiering code.

ext-storage-path is the directory FlashCache keeps flashcache.db in. FlashCache creates the directory and the file at ext-storage-capacity when they are missing. Each time the framework starts the server, it first removes flashcache.db from that directory when custom-server-configs sets ext-storage-enabled to yes, so every run starts with an empty tiering file. A relative path is taken from the Valkey directory, or from dir when custom-server-configs sets it. If the path exists and is not a directory, the run stops with an error. Nothing is changed when you use --use-running-server.

python benchmark.py \
  --mode both \
  --valkey-path /path/to/valkey-data-tiering \
  --valkey-benchmark-path /path/to/valkey-data-tiering/src/valkey-benchmark \
  --config configs/tiering.json \
  --commits HEAD

The load uses populate_retries because a 4M key load under a 1gb cap is rejected intermittently while eviction catches up, so the pass is repeated until DBSIZE reaches the keyspace size.

Per-Second Metrics Sampler

metrics_sampler.py reads the server and the host once per second during a scenario's measured benchmark process and records what each source read, unprocessed. Deriving rates, percentiles and typed values is left to ingest. Sampling starts after any inline warmup_inline, so elapsed_sec 0 is the first measured second. Tick k is taken at k intervals after the first, and a tick that overruns skips the slots it missed.

Source Records
valkey_info every INFO ALL field as a string
latency_histogram the cumulative LATENCY HISTOGRAM reply: per command calls and histogram_usec, server-side execution time in power-of-two microsecond buckets
process_cpu the cpu lines of /proc/stat as proc_stat, and per server thread its comm, utime and stime in clock ticks as threads, keyed by tid
disk the backing block device and its raw /sys/block/<dev>/stat counters as stat

process_cpu and disk read /proc and /sys, so they are collected only on Linux and only when the server runs on the sampling machine. disk samples the whole disk backing its path option, else custom-server-configs.ext-storage-path, else the server's CONFIG GET dir.

It is opt-in. The root config key per_second_sampling takes either form:

"per_second_sampling": true
"per_second_sampling": {
  "sources": {
    "valkey_info": {},
    "disk": {"path": "/mnt/nvme"}
  },
  "cpu_range": "0-1"
}

true samples every source with default options, unpinned. sources maps each selected source to its options object, and cpu_range pins the sampler thread.

The sampler reads the server through one persistent client per scenario, with the same TLS settings as the benchmark when tls_mode is on. Each read times out after 2 seconds.

Rows from every scenario, run and config set are appended to results/<commit>/timeseries.jsonl, one JSON object per line. Each row holds the same identity fields as a metrics.json row (commit, repository, cluster_mode, tls, test_id, group, scenario, config_set, config_name, module_commit and so on), plus run, profiling_set, sample_time (ISO 8601 UTC), elapsed_sec, and one key per source holding its reading. As in metrics.json, timestamp is the commit time. A source that fails a tick is absent from that row.

Each JSONL row is one line; this example is wrapped for readability:

{
  "commit": "HEAD",
  "repository": "valkey",
  "test_id": "1_b",
  "scenario": "b",
  "config_set": {},
  "run": 1,
  "sample_time": "2026-10-06T01:26:13.915+00:00",
  "elapsed_sec": 1,
  "valkey_info": {"used_memory": "1950880"},
  "latency_histogram": {"get": {"calls": 90887, "histogram_usec": {"1": 90873}}},
  "process_cpu": {"proc_stat": {"cpu": [1768215, 220728, 1396351]}, "threads": {"22858": {"comm": "valkey-server", "utime": 121, "stime": 379}}},
  "disk": {"device": "nvme0n1", "stat": [5664582, 1117339, 86138375]}
}

utils/push_to_postgres.py pushes these rows together with metrics.json. With --table X they go to benchmark_timeseries_X, so --table tiering fills the benchmark_timeseries_tiering table used by the valkey-tiering dashboard. See dashboards/README.md.

Performance Profiling

The framework includes a generic profiler that works with both core and FTS tests.

Attribution

This framework uses The FlameGraph project by Brendan Gregg for performance visualization. FlameGraph is licensed under CDDL 1.0 (Common Development and Distribution License). The FlameGraph scripts are automatically downloaded from the source repository when profiling is enabled.

Using Profiler in Core Tests

The PerformanceProfiler class can be integrated into any benchmark script:

from profiler import PerformanceProfiler

# Initialize profiler
profiler = PerformanceProfiler(results_dir, enabled=True)

# Example from search module:
profiler.start_profiling("search_1a", target_process="valkey-server")

# Run your benchmark
runner.run_benchmark_config()

# After benchmark completes
profiler.stop_profiling("search_1a")
# → Generates:
#    - flamegraph: results_dir/commit_id/flamegraphs/search_1a_20251218_080245.svg
#    - perf report: results_dir/commit_id/flamegraphs/search_1a_20251218_080245_report.txt
#    - raw data: results_dir/commit_id/flamegraphs/search_1a_20251218_080245.perf.data

Profiler Features

  • Flamegraph generation: Visual call stack analysis
  • Auto-downloads scripts: Fetches flamegraph tools from GitHub on first use
  • Profiling modes: cpu (cycles) or wall-time (all execution time)
  • Function hotspot analysis: Identify CPU-intensive code paths
  • Kernel + user space profiling: Complete stack traces with DWARF
  • Generic implementation: Works with any process (valkey-server, redis-server, etc.)
  • Configurable sampling: 999Hz default

profiling_sets Configuration

Run tests with multiple profiling configurations:

{
  "profiling_sets": [
    {"enabled": false},
    {"enabled": true, "mode": "wall-time", "sampling_freq": 999}
  ],
  "config_sets": [
    {"search.reader-threads": 1},
    {"search.reader-threads": 8}
  ]
}

Behavior:

  • Iterates profiling_sets × config_sets
  • Profiling OFF → Collects metrics in metrics.json
  • Profiling ON → Generates flamegraphs, skips metrics
  • Flamegraphs: group{X}_{scenario}_{config_values}_{timestamp}.svg

Per-scenario override:

{"id": "a", "profiling": {"delays": {"search": {"delay": 0, "duration": 10}}}}

Scenario overrides any profiling_set values.

Delay strategy pattern:

{
  "profiling_sets": [{
    "enabled": true,
    "delays": {
      "write": {"delay": 0, "duration": 10},
      "read": {"delay": 30, "duration": 10}
    }
  }],
  "scenarios": [{
    "id": "a",
    "type": "write",
    "profiling": {"delays": {"write": {"delay": 10, "duration": 10}}}
  }]
}

Group 1 write scenario uses 10s delay (dataset loading), others use 0s (immediate).

License

Please see the LICENSE.md

About

A comprehensive benchmarking tool for Valkey, an in-memory data structure store. This tool allows you to measure performance across different configurations, including TLS and cluster modes.

Resources

Security policy

Stars

14 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages