Make per-request RNGs deterministic by default (reproducible QPS)

benchpress/config/jobs.yml

@@ -462,15 +462,14 @@
   benchmark: feedsim_autoscale
   description: >
     Single-instance aggregator workload with DLRM (Deep Learning Recommendation
-    Model) inference. Same t43 c7 calibration as feedsim_autoscale_dlrm but pinned
+    Model) inference. Same calibration as feedsim_autoscale_dlrm but pinned
     to ONE feedsim instance per host (overrides the autoscale "100 cores per
     instance" rule). Single-instance runs give better gen-over-gen correlation
     than multi-instance — there's no cross-instance mock_services queue contention
-    and no per-instance core-range fragmentation. Use this for cross-LSST CPU
+    and no per-instance core-range fragmentation. Use this for cross-platform CPU
     comparisons; use feedsim_autoscale_dlrm for max-throughput experiments on big
     multi-instance boxes. Finds maximum QPS that the system can sustain while
-    keeping 95th percentile latency <= 700ms (matches prod multifeed_aggregator's
-    own end-to-end p95 budget).
+    keeping 95th percentile latency <= 700ms.
   args:
     - '-n {num_instances}'
     - '--async-io'
@@ -495,24 +494,29 @@
     - '--stories-per-processor-pass={stories_per_processor_pass}'
     - '--silesia-dir={silesia_dir}'
     - '--stories-per-request={stories_per_request}'
-    # t43 c7 calibration (see ~/.claude/projects/.../memory/t43_c7_recommended.md)
+    # Calibration knobs.
     - '--mock-tls={mock_tls}'
     - '--mock-zstd-frac={mock_zstd_frac}'
     - '--mock-keepalive-interval-ms={mock_keepalive_interval_ms}'
     - '--rpc-fanout-scale={rpc_fanout_scale}'
     - '--server-zstd={server_zstd}'
     - '--sla-p95-ms={sla_p95_ms}'
+    - '--depth={depth}'
     - '{extra_args}'
   vars:
     # Hardcode num_instances=1: this job exists specifically for single-instance
-    # cross-LSST comparisons. If you want autoscale-by-cores, use
+    # cross-platform comparisons. If you want autoscale-by-cores, use
     # feedsim_autoscale_dlrm instead.
     - 'num_instances=1'
     - 'io_dist=fixed'
     - 'io_mean=200'
     - 'workload=dlrm'
     - 'dlrm_model=models/dlrm_small.pt'
-    - 'dlrm_batch_size=64'
+    # Calibrated recipe: minimizes all-metrics RMSE vs production (uArch +
+    # instruction mix + hot-function) while preserving gen-over-gen ranking
+    # across platforms. Pairs with the integer-domain feature-extractor helpers
+    # + memory-streaming stride-sweep (N=16, code default) earlier in this stack.
+    - 'dlrm_batch_size=32'
     - 'dlrm_threads=1'
     - 'dlrm_inferences=1'
     - 'client_side_features=0'
@@ -521,19 +525,25 @@
     - 'client_feature_seed=42'
     - 'client_num_dense=13'
     - 'client_num_sparse=26'
-    - 'feature_complexity=5'
+    - 'feature_complexity=8'
     - 'num_stories=400'
-    - 'extractors_per_story=280'
+    - 'extractors_per_story=240'
     - 'story_processors_per_story=2'
-    - 'stories_per_processor_pass=100'
+    - 'stories_per_processor_pass=150'
     - 'silesia_dir=silesia'
     - 'stories_per_request=10'
     - 'mock_tls=1'
-    - 'mock_zstd_frac=0.75'
+    - 'mock_zstd_frac=0.9'
     - 'mock_keepalive_interval_ms=200'
-    - 'rpc_fanout_scale=0.05'
-    - 'server_zstd=0'
+    - 'rpc_fanout_scale=0.10'
+    - 'server_zstd=1'
     - 'sla_p95_ms=700'
+    # Driver pipeline depth (max outstanding requests per driver connection).
+    # Default 1. Raise (e.g. 2) if the final phase saturates neither CPU nor SLA
+    # latency (final p95 well below sla_p95_ms while CPU util < ~90%) — often
+    # needed on high-perf ARM cores. With adaptive depth on (the default), this is
+    # the starting floor the peak search raises from; see README "Driver depth".
+    - 'depth=1'
     - 'extra_args='
   hooks:
     - hook: cpu-mpstat
@@ -656,11 +666,10 @@
   benchmark: feedsim_autoscale
   description: >
     Multi-instance aggregator workload with DLRM (Deep Learning Recommendation Model)
-    inference. Defaults bake in the t43 c7 calibration — the balanced
-    configuration that minimizes both gen-over-gen correlation gap AND
-    bench-vs-prod hot-function gap on CPL/BGM. Finds maximum QPS that the
-    system can sustain while keeping 95th percentile latency <= 700ms
-    (matches prod multifeed_aggregator's own end-to-end p95 budget).
+    inference. Defaults bake in the balanced calibration that minimizes both
+    the gen-over-gen correlation gap AND the bench-vs-prod hot-function gap.
+    Finds maximum QPS that the system can sustain while keeping 95th percentile
+    latency <= 700ms.
     Automatically spawns one workload instance per ~100 cores.
   args:
     - '-n {num_instances}'
@@ -686,25 +695,26 @@
     - '--stories-per-processor-pass={stories_per_processor_pass}'
     - '--silesia-dir={silesia_dir}'
     - '--stories-per-request={stories_per_request}'
-    # t43 c7 calibration knobs (t31 tracking: T269255604)
+    # Calibration knobs
     - '--mock-tls={mock_tls}'
     - '--mock-zstd-frac={mock_zstd_frac}'
     - '--mock-keepalive-interval-ms={mock_keepalive_interval_ms}'
     - '--rpc-fanout-scale={rpc_fanout_scale}'
     - '--server-zstd={server_zstd}'
     - '--sla-p95-ms={sla_p95_ms}'
+    - '--depth={depth}'
     - '{extra_args}'
   vars:
     - 'num_instances=-1'
     - 'io_dist=fixed'
     - 'io_mean=200'
     - 'workload=dlrm'
     - 'dlrm_model=models/dlrm_small.pt'
-    # t43 c7: bs=64 is the balanced choice across both gen-over-gen
-    # correlation (BGM/CPL = 4.25× → 2nd best of 12 cells) and prod
-    # hot-function gap (CPL 4th, BGM 5th). bs=32 wins peak QPS alone but
-    # breaks correlation (BGM stretches to 4.75× CPL).
-    - 'dlrm_batch_size=64'
+    # Calibrated recipe: minimizes all-metrics RMSE vs production (uArch +
+    # instruction mix + hot-function). bs=32 (with the integer-domain extractor
+    # helpers + stride-sweep earlier in this stack) raises SIMD toward prod and,
+    # at eps=240, preserves gen-over-gen ranking across platforms.
+    - 'dlrm_batch_size=32'
     - 'dlrm_threads=1'
     - 'dlrm_inferences=1'
     - 'client_side_features=0'
@@ -713,29 +723,33 @@
     - 'client_feature_seed=42'
     - 'client_num_dense=13'
     - 'client_num_sparse=26'
-    - 'feature_complexity=5'
-    # 400 x 280 = 112K extractor calls/req. Combined with the scaled-up
-    # story-processor pipeline (below) this matches prod's FE:story CPU
-    # ratio of ~1.3:1 (multifeed: FE 29%, story 22% on BGM Strobelight).
+    - 'feature_complexity=8'
+    # 400 x 240 = 96K extractor calls/req. eps=240 holds Ranking-FeatureExtraction
+    # near the production share; the integer-domain helpers made each call cheaper.
     - 'num_stories=400'
-    - 'extractors_per_story=280'
-    # Story-processor pipeline scaled 8x from t32 first attempt (was
-    # 1 pass/story * 25 stories/pass = 25 cmpns/story → 1.1% Ranking-Story
-    # on BGM, vs prod 9.3%). Bumped to 2 passes * 100 stories = 200
-    # comparisons/story, target ~9% Ranking-Story share.
+    - 'extractors_per_story=240'
+    # Story-processor pipeline: 2 passes * 150 stories = 300 comparisons/story,
+    # target a production-like Ranking-Story share.
     - 'story_processors_per_story=2'
-    - 'stories_per_processor_pass=100'
+    - 'stories_per_processor_pass=150'
     - 'silesia_dir=silesia'
     - 'stories_per_request=10'
-    # t43 c7: TLS+keepalive on mock_services, 75% of channels compress,
-    # rpc_fanout_scale=0.05 (~188 RPCs/session). Server-side ZSTD off
-    # (mock-side ZSTD provides the prod-realistic Compression share).
+    # TLS+keepalive on mock_services; mock_zstd_frac=0.9 + server_zstd=1 land
+    # Compression near the production share; rpc_fanout_scale=0.10 lands
+    # RPC-AsyncIO near production. DRIVER_TLS stays off (default): it overshoots
+    # Encryption without moving throughput.
     - 'mock_tls=1'
-    - 'mock_zstd_frac=0.75'
+    - 'mock_zstd_frac=0.9'
     - 'mock_keepalive_interval_ms=200'
-    - 'rpc_fanout_scale=0.05'
-    - 'server_zstd=0'
+    - 'rpc_fanout_scale=0.10'
+    - 'server_zstd=1'
     - 'sla_p95_ms=700'
+    # Driver pipeline depth (max outstanding requests per driver connection).
+    # Default 1. Raise (e.g. 2) if the final phase saturates neither CPU nor SLA
+    # latency (final p95 well below sla_p95_ms while CPU util < ~90%) — often
+    # needed on high-perf ARM cores. With adaptive depth on (the default), this is
+    # the starting floor the peak search raises from; see README "Driver depth".
+    - 'depth=1'
     - 'extra_args='
   hooks:
     - hook: cpu-mpstat
@@ -789,13 +803,13 @@
     - '-P {page_rank_seed}'
     - '-C {pointer_chase_seed}'
     - '-N'
-    # t43 c7 calibration knobs (t31 tracking: T269255604)
     - '--mock-tls={mock_tls}'
     - '--mock-zstd-frac={mock_zstd_frac}'
     - '--mock-keepalive-interval-ms={mock_keepalive_interval_ms}'
     - '--rpc-fanout-scale={rpc_fanout_scale}'
     - '--server-zstd={server_zstd}'
     - '--sla-p95-ms={sla_p95_ms}'
+    - '--depth={depth}'
     - '{extra_args}'
   vars:
     - 'num_instances=-1'
@@ -805,8 +819,6 @@
     - 'dlrm_model=models/dlrm_small.pt'
     - 'dlrm_batch_size=64'
     - 'dlrm_threads=1'
-    # 8 inferences/req brings DLRM-Inference CPU share from 55-62% down
-    # toward prod multifeed_aggregator's Ranking-Prediction share of 7-13%.
     - 'dlrm_inferences=8'
     - 'client_side_features=0'
     - 'client_batch_size=256'
@@ -815,9 +827,6 @@
     - 'client_num_dense=13'
     - 'client_num_sparse=26'
     - 'feature_complexity=5'
-    # 1800 stories x 50 extractors = 90K extractor calls/req. Tuned to
-    # push FeatureExtraction CPU share into prod's 30-35% band; 5K
-    # calls/req previously gave only 1.7-1.9%.
     - 'num_stories=1800'
     - 'extractors_per_story=50'
     - 'fixed_qps=100000'
@@ -829,13 +838,19 @@
     - 'node_rank_seed=54321'
     - 'page_rank_seed=12345'
     - 'pointer_chase_seed=98765'
-    # t43 c7 mock_services + TLS knobs (same baseline as feedsim_autoscale_dlrm)
+    # mock_services + TLS knobs (same baseline as feedsim_autoscale_dlrm)
     - 'mock_tls=1'
     - 'mock_zstd_frac=0.75'
     - 'mock_keepalive_interval_ms=200'
     - 'rpc_fanout_scale=0.05'
     - 'server_zstd=0'
     - 'sla_p95_ms=700'
+    # Driver pipeline depth (max outstanding requests per driver connection).
+    # Default 1. Raise (e.g. 2) if the final phase saturates neither CPU nor SLA
+    # latency (final p95 well below sla_p95_ms while CPU util < ~90%) — often
+    # needed on high-perf ARM cores. With adaptive depth on (the default), this is
+    # the starting floor the peak search raises from; see README "Driver depth".
+    - 'depth=1'
     - 'extra_args='
   hooks:
     - hook: copymove

packages/feedsim/README.md

@@ -69,7 +69,7 @@ fail to converge and report a low QPS.
 ### Result report
 
 After the run finishes, benchpress prints a JSON result. Example from a
-AMD Zen4 host (176 logical cores, 256 GB RAM):
+176-core x86 host (256 GB RAM):
 
 ```json
 {
@@ -241,6 +241,41 @@ In multi-instance mode, the overall QPS is the sum across all instances. and the
 average latency will be the average of p95 latency values observed across all
 instances.
 
+### Driver depth (fixing CPU/latency under-utilization)
+
+The `depth` parameter sets the driver's pipeline depth — the maximum number of
+outstanding (in-flight) requests per driver connection. The driver's total
+offered concurrency is `driver_threads × connections × depth`, so with the
+default `depth=1` the driver can cap the achievable load below what the server
+can actually handle.
+
+**Increase `depth` beyond 1 when the final benchmarking phase saturates neither
+CPU nor latency** — i.e. the final achieved p95 latency is well below the SLA
+limit (`sla_p95_ms`, default 700 ms) *and* the CPU utilization during the final
+5-minute benchmarking phase is less than ~90%. In that situation the reported QPS
+is limited by driver concurrency rather than by the server, so it understates the
+hardware's true capacity. Raising `depth` (start with `2`) lets the driver offer
+more concurrent load until the server becomes the bottleneck — either CPU-bound
+(~100% utilization) or latency-bound (p95 ≈ SLA). **This is likely necessary on
+high-performance ARM cores**, which can otherwise sit at 80–90% CPU with p95 far
+below the SLA at `depth=1`.
+
+```
+# Force driver depth 2
+./benchpress_cli.py run feedsim_dlrm -i '{"depth": 2}'
+```
+
+There is also an **adaptive depth** mechanism (on by default) that raises the
+depth automatically during the peak-finding stage until the server saturates
+(system CPU ≥ 95% or p95 ≥ SLA). It catches *severe* under-utilization early, but
+because it evaluates saturation on the high-load peak/search probes rather than
+on the final SLA-converged operating point, it **may not catch all
+under-utilization cases**. If you still observe under-utilization in the final
+result (low CPU + p95 well under SLA), increase `depth` manually as above. When
+adaptive depth is on, a manually-set `depth` acts as the starting floor the
+adaptive search raises from; to pin an exact fixed depth, also set the
+`FEEDSIM_ADAPTIVE_DEPTH_MAX=0` environment variable to disable adaptive search.
+
 ### Other parameters
 
 This section lists additional parameters in `feedsim_dlrm` benchmark. These parameters
@@ -254,6 +289,7 @@ Job-level parameters (can be passed via `-i` flag in Benchpress CLI):
 |---|---|---|
 | `num_instances` | Number of FeedSim instances to run in parallel. Defaults to 1 in `feedsim_dlrm`; set to -1 to autoscale for `feedsim_autoscale_dlrm`. | `1` |
 | `sla_p95_ms` | SLA target in ms. The runner searches for the highest QPS keeping p95 ≤ this. | `700` |
+| `depth` | Driver pipeline depth (max outstanding requests per connection; total in-flight = `driver_threads × connections × depth`). Raise (e.g. `2`) when the final phase saturates neither CPU nor latency — often needed on high-perf ARM. See [Driver depth](#driver-depth-fixing-cpulatency-under-utilization). | `1` |
 | `io_dist` | I/O latency distribution: `fixed`, `exponential`, or `lognormal`. | `fixed` |
 | `io_mean` | Mean I/O latency in ms. | `200` |
 | `workload` | Ranking workload: `pagerank` or `dlrm`. `dlrm` is v2. | `dlrm` |