perf+feat: pipeline optimization, inspectr fast loading, plot UI improvements by astafan8 · Pull Request #441 · toolsforexperiments/plottr

astafan8 · 2026-04-15T11:51:33Z

Summary

Performance optimizations, faster inspectr loading, and plot UI improvements.
See PERFORMANCE_PLAN.md for detailed analysis, benchmarks, and future suggestions.

Pipeline Performance (3x faster on real data)

Optimized the data copy/validate/gridding pipeline that runs on every data update:

copy(): rewrote with targeted per-key semantics, added deep=False option (14.8x faster)
Node.process(): defer structure() to only when structure changes (50x faster steady-state)
MeshgridDataDict.validate(): direct min/max instead of sort-based unique/sign (1.5x)
largest_numtype(): dtype check instead of Python element iteration (15,000x)
_find_switches(): eliminated redundant computations in shape guessing (2.6x)
mask_invalid(): skip masking when data is clean (65,000x memory reduction)

Real-world result: QDstability (223 MB, 16 deps): 555 ms -> 189 ms (2.93x)

Inspectr Fast Loading (minutes -> milliseconds)

Fast DB overview: single SQL query instead of O(N^2) experiments/data_sets enumeration.
New module plottr/data/qcodes_db_overview.py (intended for QCoDeS contribution)
Lazy snapshot: tree built only on expand, not on every click (951 ms -> 0.3 ms)
Incremental refresh: only loads new runs since last known run_id
Loading progress: live "Loading database... (142/1496 datasets)" indicator
Contextual messages: guidance text when no data/date selected

Plot UI Improvements

Grid layout for pyqtgraph: subplots arranged in near-square grid (matching matplotlib)
instead of stacking vertically
Scrollable plot area: toggle + min-height spinbox in both backends' toolbars
Plot backend selector: combo box in inspectr toolbar (matplotlib/pyqtgraph)
Info pane UX: collapsed by default, smooth scrolling for tall rows

Bugs Fixed

copy() now properly deep-copies global mutable metadata
remove_invalid_entries() crash on inhomogeneous index arrays
NaN handling in _find_switches() sweep direction detection

Tests

221 new tests across 4 test files covering copy semantics, pipeline integrity,
various data shapes/dtypes (including hypothesis property-based testing),
and comprehensive gridder coverage. 280+ tests pass, 0 mypy errors.

Files Changed

Area	Files
Core data	`plottr/data/datadict.py`, `plottr/data/qcodes_dataset.py`
Utilities	`plottr/utils/num.py`
Nodes	`plottr/node/node.py`, `plottr/node/dim_reducer.py`, `plottr/node/grid.py`
Plot backends	`plottr/plot/base.py`, `plottr/plot/pyqtgraph/autoplot.py`, `plottr/plot/mpl/autoplot.py`, `plottr/plot/mpl/widgets.py`
Inspectr	`plottr/apps/inspectr.py`
New modules	`plottr/data/qcodes_db_overview.py`
Tests	4 new test files (221 tests)
Docs	`PERFORMANCE_PLAN.md`

Major performance improvements to plottr's data pipeline: - Rewrite copy() with targeted per-key semantics (14.8x faster for meshgrid) - Add copy(deep=False) API for shallow copies (xarray convention) - Optimize MeshgridDataDict.validate() monotonicity check (1.5x faster) - Add _build_structure() to skip redundant validation in internal callers - Fast-path mask_invalid() to skip clean data (65,000x memory reduction) - Fix cascading copies in XYSelector (was copying twice via inheritance) - Pass copy=False in DataGridder to avoid redundant array duplication - Optimize datasets_are_equal() with shape short-circuit - Fix bug: copy() now properly deep-copies global mutable metadata Adds 127 new tests covering copy semantics, pipeline integrity, various data shapes/dtypes, and edge cases (hypothesis property-based testing). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Comprehensive analysis of remaining performance improvements across: - HDF5 loading (reads full dataset for shape metadata - critical fix) - Node.process() redundant structure() call on every update - Complex plot rendering deepcopy overhead - Signal emission overhead (7 signals per node per update) - largest_numtype() iterating every array element as Python objects - Various numpy anti-patterns (np.append in loops, unnecessary copies) - Architectural improvements (change detection, memoization) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Round 2 performance improvements: - largest_numtype(): use numpy dtype instead of iterating every element as a Python object (~15,000x faster for numeric arrays) - Node.process(): defer structure() call to only when structure changes (50x faster steady-state updates for large meshgrids) - is_invalid(): skip unnecessary np.zeros allocation for non-float arrays - guess_grid_from_sweep_direction(): convert once with np.asarray, not 4x - remove_invalid_entries(): replace O(n^2) np.append with list+concatenate Also fixes crash on inhomogeneous index arrays (pre-existing bug) - meshgrid_to_datadict/datadict_to_dataframe: ravel() instead of flatten() - _splitComplexData(): dataclasses.replace instead of deepcopy Adds 32 new tests (205 total passing). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Rename loop variable 'd' to 'diffs' to avoid shadowing the outer loop variable from 'for d in self.dependents()'. Add explicit type annotations for ndarray variables to satisfy mypy's type narrowing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Benchmarked the full plottr pipeline (load -> DataSelector -> DataGridder -> XYSelector) on 23 QCodes datasets of varying shapes and sizes. Pipeline total: 1478 ms -> 1025 ms = 1.44x overall speedup. Largest gains on big datasets: stability_diagram 1.81x, large_3d_scan 1.65x. No regressions on any dataset. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Pipeline total: 6,550 ms -> 3,465 ms = 1.89x overall speedup on 8 large datasets (4M-point 1D, 800x800 2D, 100x100x80 3D, interrupted, multi-dep). Consistent ~2x speedup across 1D/2D shapes, ~1.7x on 3D. Loading times unchanged (QCodes SQLite I/O dominated). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

New benchmark measures both cold start (new flowchart) and steady state (persistent flowchart, simulating live monitoring refresh). Uses 5 repeats with warmup, reports median. Results on 31 datasets (23 small + 8 large): - Large datasets: cold 1.88x, steady 1.77x faster - Small datasets: cold 1.43x, steady 1.69x faster - Steady-state on small data shows up to 2.11x speedup Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Benchmarks real user actions (switch dep, swap axes, toggle subtract avg, slide dimension, toggle grid) on large datasets with per-node time breakdown. Key findings: - DataSelector: 10-17x faster (largest_numtype O(1), copy optimized) - SubtractAverage: 6-29x faster (copy 15x faster, mask_invalid skips clean) - ScaleUnits: 7-15x faster (copy 15x faster) - XYSelector: 1.5-2.3x (cascading copy removed) - DataGridder: 1.1x (dominated by actual gridding computation) Action-level: toggle_subtract_avg 9-10x, swap_xy 3.3x, switch_dep 2.3x, data_refresh 2.2x, slide_dimension 1.5-1.6x. DataGridder is now the dominant cost (58% of pipeline) and is the next frontier. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The shape-guessing algorithm (_find_switches -> find_direction_period -> guess_grid_from_sweep_direction) was the #1 bottleneck after rounds 1-2. Optimizations: - Compute is_invalid() once instead of 3 times per call - Single np.percentile([lo, hi]) call instead of two separate sorts - Direct numpy subtraction instead of MaskedArray creation - Vectorized boolean mask instead of Python list comprehension - np.nanmean for NaN-safe sweep direction detection - Cached np.std in guess_grid_from_sweep_direction Results (800x800 = 640K pts): - _find_switches: 80ms -> 31ms (2.6x) - datadict_to_meshgrid: 175ms -> 71ms (2.5x) - Cumulative pipeline speedup vs master: 2.8-3.5x Adds 62 comprehensive gridder tests covering all GridOption paths, edge cases, various shapes, noisy axes, incomplete data. All 267 tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Benchmarked on production quantum device datasets: - QDstability (223 MB, 16 deps): cold 3.56x, steady 2.93x faster - TopogapStage2 (152 MB, 21 deps, 4D): cold 2.47x, steady 2.7x faster - QDtuning (14 MB, 16 deps): steady 2.73x faster - DataSelector node: 12-13x faster on these multi-dependent datasets Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Lazy snapshot loading in RunInfo: - Snapshot tree widget items are built only when the user expands the 'QCoDeS Snapshot' section, not on every click - Saves ~951ms per click on datasets with 5.9 MB snapshots (3,554x faster) - Info pane shows collapsed by default instead of expandAll() Incremental DB refresh: - refreshDB() now loads only new runs since last refresh using the start parameter of get_runs_from_db() - Merges incremental results into existing dataframe - First load still loads everything All 267 tests pass. No mypy errors introduced. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Add get_runs_from_db_fast() which uses load_by_id() directly per run, bypassing the O(N^2) experiments() + data_sets() enumeration. For 1496 runs: old approach takes 15+ minutes, new takes ~5 seconds. Incremental refresh loads only new runs since last known run_id. LoadDBProcess now uses get_runs_from_db_fast with start_run_id parameter for both initial load and incremental refresh. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

RunList now shows contextual overlay messages: - 'Loading database... (N/M datasets)' with live progress during load - 'Select a date on the left to browse datasets.' when idle - 'No datasets found in this database.' for empty DBs - 'No datasets match the current filter.' when star/cross filters hide all Progress is reported from the worker thread via progressUpdated signal, updated every 10 datasets for smooth display without overhead. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Check actual RunList widget state (topLevelItemCount) instead of _selected_dates to decide whether to show the hint text. This handles same-file reload, empty date selection, and filter edge cases. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- Data structure and Metadata sections now collapsed by default (user expands what they need) - Set ScrollPerPixel on RunInfo tree widget so tall rows (e.g., long exception tracebacks in metadata) can be scrolled smoothly instead of jumping to the next row Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Adds a combo box in the toolbar to switch between matplotlib and pyqtgraph backends. Default is matplotlib. The selection applies to all newly opened plot windows. Existing windows keep their backend. The combo box respects the --plotWidgetClass passed via constructor (e.g., from script_pyqtgraph entrypoint). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

New module plottr/data/qcodes_db_overview.py: - get_db_overview(): single SQL JOIN query for all run metadata - Skips snapshot and run_description blobs entirely - Reads inspectr_tag directly as a column from runs table - 6x faster than load_by_id, ~1000x faster than experiments() enumeration - Intended for eventual contribution to QCoDeS Inspectr LoadDBProcess now uses SQL path by default with automatic fallback to qcodes API (get_runs_from_db_fast) if SQL fails. Also: default window size widened from 640x640 to 960x640. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Replace the single-column QSplitter with a QGridLayout that arranges subplots on a near-square grid, using the same formula as matplotlib: nrows = int(n ** 0.5 + 0.5) ncols = ceil(n / nrows) This makes pyqtgraph behave like matplotlib when plotting many dependents: plots are arranged in columns (e.g., 4 plots = 2x2, 6 = 2x3, 16 = 4x4) instead of stacking vertically. A scroll area wraps the grid so very many plots remain accessible. Each plot has a minimum height of 250px to stay readable. 280 tests pass, 0 mypy errors. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Add 'Scrollable' checkbox in both pyqtgraph and matplotlib toolbars: - Enabled by default - When many subplots exist, the plot area expands beyond the window and becomes scrollable, keeping each subplot readable - Can be unchecked to fit everything into the visible window PyQtGraph: min plot height reduced from 250px to 75px. Matplotlib: canvas wraps in QScrollArea, min height set per grid row. 280 tests pass, 0 mypy errors. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Both backends: - Scrollable is now OFF by default - Added a 'px' spinbox next to the Scrollable checkbox showing the minimum height per subplot row (default 75px pyqtgraph, 100px mpl) - Spinbox is only enabled when Scrollable is checked - Minimum value is 40px - Changing the spinbox value triggers replot 280 tests pass, 0 mypy errors. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The inspectr backend selector passes plotWidgetClass to autoplotQcodesDataset, but the function signature was missing this parameter on the branch. Also passes it through to QCAutoPlotMainWindow. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- Extract 'Select a date...' string into _SELECT_DATE_HINT constant - Replace string-magic backend detection with explicit _PLOT_BACKENDS mapping (display name -> class) - _backend_name_for_class() for reverse lookup - Unknown plotWidgetClass added to combo with its class name as label - _onBackendChanged uses the mapping instead of hardcoded imports 280 tests pass, 0 mypy errors. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- _split_timestamp(): proper datetime parsing instead of string slicing for splitting qcodes timestamp strings into date/time components. Applied to both get_ds_info() and _ds_to_info_dict(). - get_runs_from_db_fast(): removed unnecessary initialise_or_create_database_at call, use same read_only pattern as get_runs_from_db. - qcodes_db_overview: use conn_from_dbpath_or_conn from qcodes instead of raw sqlite3.connect. Remove unused get_last_run_id function. - mpl/widgets: remove dead _scrollable attribute and fix setScrollable which had identical code in both branches. 280 tests pass, 0 mypy errors. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

New module plottr/utils/latex.py using unicodeit (now a required dependency): - Greek letters: \alpha -> α, \Omega -> Ω - Math symbols: \hbar -> ℏ, \partial -> ∂, \int -> ∫ - Subscripts: V_{gate} -> Vgate (HTML for text, Unicode for digits) - Superscripts: x^{2} -> x² (Unicode), e^{iπ} -> eiπ - Fractions: \frac{dI}{dV} -> dI/dV - Square root: \sqrt{x} -> √x - Dollar delimiters stripped Applied to pyqtgraph axis labels in FigureMaker.formatSubPlot(). Falls through gracefully on plain text (no LaTeX = no change). 35 new tests including hypothesis property-based testing. 315 tests pass, 0 mypy errors. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Convert subscripts and superscripts to HTML tags BEFORE running unicodeit, so they become 11 and 2 instead of Unicode ₁₁ and ². HTML tags render more consistently in Qt rich text. unicodeit still converts Greek letters and symbols inside the tags. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Plain strings with underscores (e.g. gate_voltage, channel_1_amplitude) now pass through unchanged. Conversion only triggers when the string contains backslash commands (\alpha), dollar delimiters ($...$), or braced sub/superscripts (_{...}, ^{...}). Also drops single-char bare sub/sup patterns (_x, ^x) which were too aggressive on ordinary identifiers. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- Add qcodes.utils.plotting to mypy ignore_missing_imports (module removed in newer qcodes) - Add hypothesis to test_requirements.txt for CI - Fix rettype initialization in combine_datadicts (type narrowing) - Fix plotOptions unpacking when None (mpl autoplot) - Fix widgetConnection.get type mismatch (autonode) - Remove stale type: ignore comments (inspectr) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- Restore type: ignore[has-type] on inspectr starAction/crossAction (needed by PyQt5-stubs, unused with PyQt6) - Add type: ignore[arg-type] on autonode widgetConnection.get (needed with PyQt6, unused with PyQt5-stubs) - Set warn_unused_ignores = false since project targets both Qt bindings Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

qcodes moved find_scale_and_prefix from qcodes.utils.plotting to qcodes.plotting.axis_labels. Update the import chain to try the current location first, then the old one, then the local fallback. Remove qcodes.utils.plotting from mypy ignore_missing_imports since the import is now handled via try/except with proper type: ignore annotations. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- Revert rettype initialization to None + assert before use (cleaner than pre-computing a default that changes the original semantics) - Remove dead local fallback for qcodes < 0.21 (min supported is 0.54.1); keep old-path fallback with version comment for < 0.46 - Restore warn_unused_ignores = true globally; add per-module override (warn_unused_ignores = false) only for modules with cross-Qt-backend type: ignore comments (inspectr, autonode, scaleunits) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

This PR focuses on speeding up Plottr’s data pipeline and inspectr DB browsing, while improving plot UI/UX across backends.

Changes:

Optimizes core DataDict/numeric utilities and node processing to reduce copying/validation overhead.
Adds fast inspectr DB loading (direct SQL overview + incremental refresh) and lazy snapshot rendering.
Improves plotting UX: pyqtgraph subplot grid layout, scrollable plot areas, LaTeX-ish label rendering, and backend selection.

Reviewed changes

Copilot reviewed 24 out of 24 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
test_requirements.txt	Adds `hypothesis` for new property-based tests.
pyproject.toml	Adds `unicodeit` dependency and mypy overrides.
plottr/utils/num.py	Performance-oriented rewrites for numeric helpers and grid guessing.
plottr/utils/latex.py	New LaTeX-to-HTML label conversion helper (uses `unicodeit`).
plottr/plot/pyqtgraph/autoplot.py	Grid-based subplot layout, scrollable plot area, LaTeX label rendering, new toolbar options.
plottr/plot/mpl/widgets.py	Wraps MPL canvas in a scroll area.
plottr/plot/mpl/autoplot.py	Makes plotting robust to `None` plot options; adds scrollable plot controls.
plottr/plot/base.py	Avoids `deepcopy` for complex splitting via `dataclasses.replace`.
plottr/node/scaleunits.py	Updates `find_scale_and_prefix` import path/fallback logic.
plottr/node/node.py	Defers structure snapshot creation to only when structure changes.
plottr/node/grid.py	Avoids extra copies during gridding (`copy=False` into meshgrid conversion).
plottr/node/dim_reducer.py	Removes redundant copy in `XYSelector.process()`.
plottr/node/autonode.py	Adds a mypy-targeted `type: ignore` for option type handling.
plottr/data/qcodes_db_overview.py	New module: fast run overview via a single SQL JOIN query.
plottr/data/qcodes_dataset.py	Timestamp parsing improvements + incremental/fast DB enumeration helpers.
plottr/data/datadict.py	Major copy/structure/masking/validation performance improvements and bug fixes.
plottr/apps/inspectr.py	Fast loading, incremental refresh, progress/overlay UX, lazy snapshot tree, plot backend selector.
plottr/apps/autoplot.py	Allows injecting plot widget backend into autoplot windows.
test/pytest/test_round2_optimizations.py	New tests for optimized utilities and behaviors.
test/pytest/test_pipeline_coverage.py	New broad pipeline/node coverage incl. hypothesis strategies.
test/pytest/test_latex.py	New tests for LaTeX-to-HTML conversion.
test/pytest/test_gridder_comprehensive.py	Comprehensive gridder/gridding path tests.
test/pytest/test_datadict_copy_semantics.py	Extensive copy semantics + integrity regression tests.
PERFORMANCE_PLAN.md	Benchmarks, rationale, and future optimization plan.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-04-21T11:26:36Z

+            # Incremental load: merge new rows into existing dataframe
+            # Update existing rows (e.g., completed_date may have changed)
+            for idx in dbdf.index:
+                if idx in self.dbdf.index:
+                    self.dbdf.loc[idx] = dbdf.loc[idx]
+                else:
+                    self.dbdf = pandas.concat([self.dbdf, dbdf.loc[[idx]]])


The incremental merge in DBLoaded concatenates one-row DataFrames inside a loop (pandas.concat per index). This is quadratic in the number of new rows and can become very slow if self.dbdf is empty but latestRunId is set (e.g., after an initial empty load) and the next refresh brings in a full DB. Prefer a vectorized merge: update existing rows via self.dbdf.update(dbdf) and concatenate all truly-new rows in a single concat (or just assign self.dbdf = dbdf when self.dbdf is empty).

Suggested change

# Incremental load: merge new rows into existing dataframe

# Update existing rows (e.g., completed_date may have changed)

for idx in dbdf.index:

if idx in self.dbdf.index:

self.dbdf.loc[idx] = dbdf.loc[idx]

else:

self.dbdf = pandas.concat([self.dbdf, dbdf.loc[[idx]]])

# Incremental load: merge new rows into existing dataframe.

# Update existing rows in bulk (e.g., completed_date may have changed)

# and append all truly new rows in a single concat.

if self.dbdf.empty:

self.dbdf = dbdf

else:

self.dbdf.update(dbdf)

new_rows = dbdf.loc[~dbdf.index.isin(self.dbdf.index)]

if not new_rows.empty:

self.dbdf = pandas.concat([self.dbdf, new_rows])

Fixed -- replaced per-row concat loop with vectorized update() + single concat for new rows.

Copilot · 2026-04-21T11:26:37Z

+    # fallback for qcodes < 0.46 where the function lived under utils
+    from qcodes.utils.plotting import find_scale_and_prefix  # type: ignore[import-not-found, no-redef]


plottr declares qcodes as an optional dependency, but this module now always imports from qcodes.* (both the try and the fallback). If qcodes isn’t installed, importing plottr.node.scaleunits will fail at import time. Restore a non-qcodes fallback (e.g., plottr.utils.find_scale_and_prefix.find_scale_and_prefix) when qcodes is missing, and only use the qcodes variants when available.

Suggested change

# fallback for qcodes < 0.46 where the function lived under utils

from qcodes.utils.plotting import find_scale_and_prefix # type: ignore[import-not-found, no-redef]

try:

# fallback for qcodes < 0.46 where the function lived under utils

from qcodes.utils.plotting import find_scale_and_prefix # type: ignore[import-not-found, no-redef]

except ImportError:

from plottr.utils.find_scale_and_prefix import find_scale_and_prefix

Fixed -- restored the local plottr.utils.find_scale_and_prefix fallback for when qcodes is not installed.

Copilot · 2026-04-21T11:26:37Z

+    def setScrollable(self, scrollable: bool) -> None:
+        """Enable or disable scroll area around the plot grid."""
+        if scrollable:
+            self._scrollArea.setWidgetResizable(True)
+            self._gridWidget.setMinimumHeight(0)
+            # Re-apply grid min height if we have plots
+            if self.subPlots:
+                n = len(self.subPlots)
+                nrows = max(1, int(n ** 0.5 + 0.5))
+                self._gridWidget.setMinimumHeight(nrows * 75)
+        else:
+            self._scrollArea.setWidgetResizable(True)
+            self._gridWidget.setMinimumHeight(0)



FigureWidget.setScrollable() currently doesn’t actually disable scrolling: both branches call setWidgetResizable(True) and reset min height to 0, so the scrollable flag has no effect beyond the if scrollable block. Also, it hard-codes 75 instead of using self._minPlotHeight/the passed min height. Adjust the implementation so scrollable=False restores a non-scrollable layout (or at least disables scrollbars / min-height behavior) and avoid hard-coded constants.

Fixed -- setScrollable(True) now uses self._minPlotHeight instead of hard-coded 75.

Copilot · 2026-04-21T11:26:38Z

+    def _arrangeGrid(self, min_plot_height: Optional[int] = None) -> None:
+        """Arrange subplots on a near-square grid, matching matplotlib's layout."""
+        n = len(self.subPlots)
+        if n == 0:
+            return
+
+        if min_plot_height is None:
+            min_plot_height = self._minPlotHeight
+
+        nrows = max(1, int(n ** 0.5 + 0.5))
+        ncols = max(1, int(np.ceil(n / nrows)))
+
+        self._gridWidget.setMinimumHeight(nrows * min_plot_height)
+
+        for i, plot in enumerate(self.subPlots):
+            row = i // ncols
+            col = i % ncols
+            self._gridLayout.addWidget(plot, row, col)
+


_arrangeGrid() keeps calling _gridLayout.addWidget(plot, row, col) without first removing existing layout items. If _arrangeGrid() is called multiple times (e.g., on replot/option changes), this can leave stale QLayoutItems behind and/or cause widgets to be re-added/repositioned in surprising ways. Consider clearing _gridLayout at the start (e.g., takeAt() loop) before re-adding all subplots.

Fixed -- _arrangeGrid() now clears the grid layout via takeAt() loop before re-adding widgets.

Copilot · 2026-04-21T11:26:38Z

+"""
+import numpy as np
+import pytest
+from copy import deepcopy


Unused import: from copy import deepcopy is never referenced in this test module. Removing it avoids lint noise and keeps the test focused.

Suggested change

from copy import deepcopy

Fixed -- removed the unused deepcopy import.

Copilot · 2026-04-21T11:26:38Z

+    def resizeEvent(self, event: QtGui.QResizeEvent) -> None:
+        super().resizeEvent(event)
+        self._overlayLabel.setGeometry(self.viewport().rect())
+
        self.setContextMenuPolicy(QtCore.Qt.CustomContextMenu)
        self.customContextMenuRequested.connect(self.showContextMenu)


setContextMenuPolicy and the customContextMenuRequested.connect(...) call are inside resizeEvent, so every resize will add another signal connection (leading to multiple context menus firing and a growing number of connected slots). Move the context menu policy + signal connection setup into __init__, and keep resizeEvent limited to geometry updates.

Fixed -- moved setContextMenuPolicy and customContextMenuRequested.connect from resizeEvent into init.

- Restore local fallback for find_scale_and_prefix when qcodes is not installed (qcodes is an optional dependency) - Move context menu setup from resizeEvent to __init__ to avoid accumulating signal connections on every resize - Replace per-row concat loop in DBLoaded with vectorized update() + single concat for new rows - Clear grid layout in _arrangeGrid before re-adding to avoid stale layout items on repeated calls - Fix setScrollable to use _minPlotHeight instead of hard-coded 75 - Remove unused deepcopy import from test Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Mikhail Astafev and others added 17 commits April 15, 2026 13:51

astafan8 force-pushed the perf/datadict-copy-optimization branch from a9843a4 to 4263563 Compare April 20, 2026 12:21

Mikhail Astafev and others added 5 commits April 20, 2026 14:27

fix: reduce matplotlib scrollable min height to 100px per row

f21196f

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

docs: reorganize PERFORMANCE_PLAN.md into implemented vs future

270e6cd

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

astafan8 changed the title ~~perf: optimize DataDict copy, validate, and pipeline data flow~~ perf+feat: pipeline optimization, inspectr fast loading, plot UI improvements Apr 20, 2026

Mikhail Astafev and others added 6 commits April 20, 2026 16:01

astafan8 marked this pull request as ready for review April 21, 2026 07:06

Mikhail Astafev and others added 4 commits April 21, 2026 09:41

jenshnielsen requested a review from Copilot April 21, 2026 11:19

Copilot started reviewing on behalf of jenshnielsen April 21, 2026 11:19 View session

Copilot AI reviewed Apr 21, 2026

View reviewed changes

-            # Incremental load: merge new rows into existing dataframe
-            # Update existing rows (e.g., completed_date may have changed)
-            for idx in dbdf.index:
-                if idx in self.dbdf.index:
-                    self.dbdf.loc[idx] = dbdf.loc[idx]
-                else:
-                    self.dbdf = pandas.concat([self.dbdf, dbdf.loc[[idx]]])
+            # Incremental load: merge new rows into existing dataframe.
+            # Update existing rows in bulk (e.g., completed_date may have changed)
+            # and append all truly new rows in a single concat.
+            if self.dbdf.empty:
+                self.dbdf = dbdf
+            else:
+                self.dbdf.update(dbdf)
+                new_rows = dbdf.loc[~dbdf.index.isin(self.dbdf.index)]
+                if not new_rows.empty:
+                    self.dbdf = pandas.concat([self.dbdf, new_rows])

		# fallback for qcodes < 0.46 where the function lived under utils
		from qcodes.utils.plotting import find_scale_and_prefix # type: ignore[import-not-found, no-redef]

Conversation

astafan8 commented Apr 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Pipeline Performance (3x faster on real data)

Inspectr Fast Loading (minutes -> milliseconds)

Plot UI Improvements

Bugs Fixed

Tests

Files Changed

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Apr 21, 2026

Choose a reason for hiding this comment

Uh oh!

astafan8 Apr 21, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 21, 2026

Choose a reason for hiding this comment

Uh oh!

astafan8 Apr 21, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 21, 2026

Choose a reason for hiding this comment

Uh oh!

astafan8 Apr 21, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 21, 2026

Choose a reason for hiding this comment

Uh oh!

astafan8 Apr 21, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 21, 2026

Choose a reason for hiding this comment

Uh oh!

astafan8 Apr 21, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 21, 2026

Choose a reason for hiding this comment

Uh oh!

astafan8 Apr 21, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

astafan8 commented Apr 15, 2026 •

edited

Loading