Skip to content

refactor: de-normalize per-query method classification to a varchar column #48

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
TheWitness merged 4 commits into from
Oct 1, 2026
Merged

refactor: de-normalize per-query method classification to a varchar column #48

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
a182d75
refactor: de-normalize method classification to a varchar column
TheWitness Oct 1, 2026
6ccbad5
refactor: remove write-only plugin_slowlog_tables and drop the redund…
TheWitness Oct 1, 2026
c727c51
test: correct setup create-call count after dropping plugin_slowlog_t…
TheWitness Oct 1, 2026
22de269
fix: migrate existing plugin_slowlog_details_methods to method varcha…
TheWitness Oct 1, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,10 @@

* perf: Stream the slow query log a line at a time during import (`fgets()`) instead of slurping the whole file into memory with `file()`, so importing a multi-gigabyte slow query log no longer costs many times its own size in PHP array overhead - the detail rows were already flushed to the database in `SLOWLOG_IMPORT_BATCH_SIZE` batches as parsing proceeds
* bug: Distinguish a read error from end-of-file when streaming the import - `fgets()` returns `false` for both, so a mid-file I/O error was being treated as a clean EOF and the partial import was finalized (and post-processed) as successful. `import_logfile()` now checks `feof()` before closing the handle and fails the import (status 3, "Read Error - Import Aborted, Log May Be Truncated") instead of ingesting a truncated log
* perf: Drop the `SELECT DISTINCT sld.*` from the details list query - the association tables' composite primary keys already make every `plugin_slowlog_details` x method x table join row unique, so the DISTINCT never removed a row and only forced a filesort/hash over the `mediumtext` query columns on every page of the Details view
* refactor: Remove the write-only `plugin_slowlog_tables` table - it was inserted into during import and deleted on remove/reprocess but never read (the details list, charts, and filters all use `plugin_slowlog_details_tables`); it is no longer created, and the uninstall drop is kept so existing installs clean it up
* refactor: De-normalize the per-query method classification - `plugin_slowlog_details_methods` now stores the method name directly in a `method` varchar column instead of a `methodid` foreign key into `plugin_slowlog_methods`, and the classifier fragment dictionary moved into a PHP constant (`SLOWLOG_METHOD_FRAGMENTS`). The stats, By Method chart, details list, method filter dropdown, and chart-scope queries no longer join `plugin_slowlog_methods`, removing a join from every one of those read paths. The now-unused `plugin_slowlog_methods` dictionary table is left in place (a clean install drops it)
* bug: Migrate an existing `plugin_slowlog_details_methods` table in place during upgrade - `api_plugin_db_table_create()` never retrofits a table that already exists, so on an existing install the de-normalization above would otherwise leave the old `methodid` column (and no `method` column), breaking every import/detail/stat/chart query with an unknown-column error. `slowlog_check_upgrade()` now runs an idempotent `slowlog_migrate_details_methods_to_method()` that adds the `method` column, backfills the names from the `plugin_slowlog_methods` dictionary, then swaps the primary key to `(logid, logentry, method)` and drops `methodid`. It runs on every page (self-gating on the `methodid` column) because this schema change ships without a version bump, so the version-drift upgrade branch would never fire for it
* refactor: Move the slowlog_functions.php library file into includes/ and switch every file inclusion from include/include_once to require/require_once for fail-fast consistency (references updated across setup.php, slowlog.php, import_log.php, and the test suite)
* refactor: Move all schema management (table definitions, create, upgrade, and drop helpers) into includes/database.php (the thold model) with the Cacti copyright header; setup.php's install/uninstall/upgrade paths now require that file and delegate to it, keeping the api_plugin_db_table_create() create path and the db_update_table() upgrade refresh unchanged
* dev: Enforce patch coverage of changed lines in CI and remove the inert COMPOSER_ROOT_VERSION env from the Pest step
Expand Down
59 changes: 44 additions & 15 deletions includes/database.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -145,8 +145,8 @@ function slowlog_setup_table_new(): void {
$data['columns'][] = ['name' => 'id', 'type' => 'int(10)', 'unsigned' => true, 'NULL' => false, 'auto_increment' => true];
$data['columns'][] = ['name' => 'logid', 'type' => 'int(10)', 'unsigned' => true, 'NULL' => false];
$data['columns'][] = ['name' => 'logentry', 'type' => 'int(10)', 'unsigned' => true, 'NULL' => false];
$data['columns'][] = ['name' => 'methodid', 'type' => 'int(10)', 'unsigned' => true, 'NULL' => false];
$data['primary'] = ['logid', 'logentry', 'methodid'];
$data['columns'][] = ['name' => 'method', 'type' => 'varchar(45)', 'NULL' => false];
$data['primary'] = ['logid', 'logentry', 'method'];
Comment thread
TheWitness marked this conversation as resolved.
$data['keys'][] = ['name' => 'id', 'columns' => ['id']];
$engine = slowlog_get_storage_engine();
$data['type'] = $engine;
Expand Down Expand Up @@ -183,20 +183,11 @@ function slowlog_setup_table_new(): void {
// inserting the longer ANALYZES/OPTIMIZES/CREATES seed fragments.
db_execute('ALTER TABLE plugin_slowlog_methods MODIFY COLUMN `query` varchar(96) NOT NULL');

$data = [];
$data['columns'][] = ['name' => 'logid', 'type' => 'int(10)', 'unsigned' => true, 'NULL' => false];
$data['columns'][] = ['name' => 'table_name', 'type' => 'varchar(45)', 'NULL' => false];
$data['primary'] = ['logid', 'table_name'];
$data['type'] = 'InnoDB';
$data['row_format'] = 'Dynamic';

api_plugin_db_table_create('slowlog', 'plugin_slowlog_tables', $data);

// Schema groundwork: dictionary of every distinct table name seen across imports, plus
// whether it's a known Cacti table. Per-logentry table associations are still written
// directly to plugin_slowlog_tables/plugin_slowlog_details_tables by import_post_process();
// this table only caches the is_cacti_table lookup so OTHER TABLES classification doesn't
// re-derive it per logentry (see slowlog_sync_table_dictionary()/slowlog_classify_other_tables()).
// whether it's a known Cacti table. Per-logentry table associations are written directly
// to plugin_slowlog_details_tables by import_post_process(); this table only caches the
// is_cacti_table lookup so OTHER TABLES classification doesn't re-derive it per logentry
// (see slowlog_sync_table_dictionary()/slowlog_classify_other_tables()).
$data = [];
$data['columns'][] = ['name' => 'tableid', 'type' => 'int(10)', 'unsigned' => true, 'NULL' => false, 'auto_increment' => true];
$data['columns'][] = ['name' => 'table_name', 'type' => 'varchar(45)', 'NULL' => false, 'default' => ''];
Expand Down Expand Up @@ -315,6 +306,44 @@ function slowlog_upgrade_tables(): void {
}
}

/**
* Idempotently migrates an existing plugin_slowlog_details_methods table from the
* pre-2.6 (logid, logentry, methodid) layout to the de-normalized (logid, logentry,
* method varchar) layout: adds the method column, backfills the names from the
* plugin_slowlog_methods dictionary, then swaps the primary key and drops methodid.
* A no-op on fresh installs (which already receive the new schema from the create
* path) and on installs that have already been migrated, so it is safe to call on
* every page. Runs independently of the plugin version check in
* slowlog_check_upgrade() because this schema change ships without a version bump,
* and api_plugin_db_table_create() never retrofits an existing table. Called from
* slowlog_check_upgrade().
*
* @return void
*/
function slowlog_migrate_details_methods_to_method(): void {
// Only pre-2.6 installs still carry methodid; once it's gone the migration is complete.
if (!db_table_exists('plugin_slowlog_details_methods') || !db_column_exists('plugin_slowlog_details_methods', 'methodid')) {
return;
}

if (!db_column_exists('plugin_slowlog_details_methods', 'method')) {
db_execute('ALTER TABLE plugin_slowlog_details_methods
ADD COLUMN `method` varchar(45) NOT NULL DEFAULT \'\' AFTER `logentry`');
}

// Backfill the human-readable name from the dictionary for any row still missing it.
db_execute('UPDATE plugin_slowlog_details_methods AS sldm
INNER JOIN plugin_slowlog_methods AS sm ON sm.methodid = sldm.methodid
SET sldm.method = sm.method
WHERE sldm.method = \'\'');

// Swap (logid, logentry, methodid) -> (logid, logentry, method) and retire methodid.
db_execute('ALTER TABLE plugin_slowlog_details_methods
DROP PRIMARY KEY,
ADD PRIMARY KEY (`logid`, `logentry`, `method`),
DROP COLUMN `methodid`');
}

/**
* Drops every table this plugin owns. Called from
* plugin_slowlog_uninstall() when the plugin is removed.
Expand Down
114 changes: 54 additions & 60 deletions includes/slowlog_functions.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,6 @@
function api_slowlog_remove(int $logid): void {
db_execute_prepared('DELETE FROM plugin_slowlog WHERE logid = ?', [$logid]);
db_execute_prepared('DELETE FROM plugin_slowlog_details WHERE logid = ?', [$logid]);
db_execute_prepared('DELETE FROM plugin_slowlog_tables WHERE logid = ?', [$logid]);
db_execute_prepared('DELETE FROM plugin_slowlog_details_tables WHERE logid = ?', [$logid]);
db_execute_prepared('DELETE FROM plugin_slowlog_details_methods WHERE logid = ?', [$logid]);
db_execute_prepared('DELETE FROM plugin_slowlog_stats WHERE logid = ?', [$logid]);
Expand Down Expand Up @@ -107,8 +106,8 @@ function slowlog_bulk_insert_method_rows(array $rows): void {
return;
}

$sql_prefix = 'INSERT INTO plugin_slowlog_details_methods (logid, logentry, methodid) VALUES ';
$sql_suffix = ' ON DUPLICATE KEY UPDATE methodid=VALUES(methodid)';
$sql_prefix = 'INSERT INTO plugin_slowlog_details_methods (logid, logentry, method) VALUES ';
$sql_suffix = ' ON DUPLICATE KEY UPDATE method=VALUES(method)';

foreach (array_chunk($rows, 500) as $chunk) {
$placeholders = [];
Expand All @@ -118,7 +117,7 @@ function slowlog_bulk_insert_method_rows(array $rows): void {
$placeholders[] = '(?, ?, ?)';
$params[] = (int) $row[0];
$params[] = (int) $row[1];
$params[] = (int) $row[2];
$params[] = (string) $row[2];
}

db_execute_prepared($sql_prefix . implode(', ', $placeholders) . $sql_suffix, $params);
Expand Down Expand Up @@ -168,6 +167,45 @@ function slowlog_bulk_insert_table_rows(array $rows): void {
// number of plugin_slowlog_details rows accumulated per bulk INSERT during import
const SLOWLOG_IMPORT_BATCH_SIZE = 1000;

// The SQL-method classifier dictionary: method name => query-text fragments matched
// case-insensitively (stripos) when classifying each imported query. Replaces the former
// plugin_slowlog_methods table lookup now that the method name is stored directly in
// plugin_slowlog_details_methods.method. OTHERS (matched-nothing fallback) and OTHER TABLES
// (classified by table recognition, not a text fragment) are handled separately.
const SLOWLOG_METHOD_FRAGMENTS = [
'INSERTS' => ['INSERT INTO', 'INSERT IGNORE INTO'],
'REPLACES' => ['REPLACE INTO', 'REPLACE IGNORE INTO'],
'DELETES' => ['DELETE '],
'SELECTS' => ['SELECT '],
'DISTINCTS' => ['SELECT DISTINCT'],
'UNIONS' => ['UNION'],
'JOINS' => ['JOIN '],
'UPDATES' => ['UPDATE '],
'RENAMES' => ['RENAME TABLE'],
'FLUSHES' => ['FLUSH TABLE'],
'TRUNCATES' => ['TRUNCATE '],
'LOAD DATA' => ['LOAD DATA INFILE '],
'OUTFILES' => ['INTO OUTFILE '],
'INFILES' => ['INFILE '],
'GROUP BY' => ['GROUP BY '],
'COUNTS' => ['COUNT('],
'SHOWS' => ['SHOW '],
'UNION ALLS' => ['UNION ALL'],
'MAX_EXECUTION_TIME' => ['MAX_EXECUTION_TIME('],
'MAX_STATEMENT_TIME' => ['MAX_STATEMENT_TIME'],
'FORCE INDEX' => ['FORCE INDEX'],
'ALTERS' => ['ALTER TABLE'],
'DROPS' => ['DROP TABLE', 'DROP TEMPORARY TABLE'],
'ANALYZES' => ['ANALYZE TABLE', 'ANALYZE NO_WRITE_TO_BINLOG TABLE', 'ANALYZE LOCAL TABLE'],
'OPTIMIZES' => ['OPTIMIZE TABLE', 'OPTIMIZE NO_WRITE_TO_BINLOG TABLE', 'OPTIMIZE LOCAL TABLE'],
'CREATES' => ['create table'],
'CREATE TEMPS' => ['create temporary table'],
];

// Bucket method names stored directly in plugin_slowlog_details_methods.method.
const SLOWLOG_METHOD_OTHERS = 'OTHERS';
const SLOWLOG_METHOD_OTHER_TABLES = 'OTHER TABLES';

// the By Method/By Table charts' 'Top N' selectmenu options
const SLOWLOG_CHART_TOP_OPTIONS = ['2', '10', '15', '20', '25', '30'];

Expand Down Expand Up @@ -436,10 +474,9 @@ function slowlog_collect_stats_by_method(int $logid, array &$values, int $chunk_
$last_id = 0;

do {
$rows = db_fetch_assoc_prepared('SELECT sldm.id, sm.method AS scope_key,
$rows = db_fetch_assoc_prepared('SELECT sldm.id, sldm.method AS scope_key,
d.query_time, d.rows_sent, d.rows_examined, d.rows_affected, d.bytes_sent
FROM plugin_slowlog_details_methods AS sldm
INNER JOIN plugin_slowlog_methods AS sm ON sm.methodid = sldm.methodid
INNER JOIN plugin_slowlog_details AS d ON d.logid = sldm.logid AND d.logentry = sldm.logentry
WHERE sldm.logid = ?
AND sldm.id > ?
Expand Down Expand Up @@ -744,15 +781,6 @@ function slowlog_sync_table_dictionary(int $logid, ?array $known_tables = null):
* @return void
*/
function slowlog_classify_other_tables(int $logid): void {
$methodid = db_fetch_cell_prepared("SELECT methodid
FROM plugin_slowlog_methods
WHERE method = 'OTHER TABLES'",
[]);

if (!$methodid) {
return;
}

$rows = db_fetch_assoc_prepared('SELECT DISTINCT dt.logentry
FROM plugin_slowlog_details_tables AS dt
INNER JOIN plugin_slowlog_table_names AS tn
Expand All @@ -768,7 +796,7 @@ function slowlog_classify_other_tables(int $logid): void {
$method_rows = [];

foreach ($rows as $row) {
$method_rows[] = [$logid, $row['logentry'], $methodid];
$method_rows[] = [$logid, $row['logentry'], SLOWLOG_METHOD_OTHER_TABLES];
}

slowlog_bulk_insert_method_rows($method_rows);
Expand All @@ -791,15 +819,6 @@ function slowlog_classify_other_tables(int $logid): void {
* @return void
*/
function slowlog_classify_other_tables_against_list(int $logid, array $reference_tables): void {
$methodid = db_fetch_cell_prepared("SELECT methodid
FROM plugin_slowlog_methods
WHERE method = 'OTHER TABLES'",
[]);

if (!$methodid) {
return;
}

$tables = db_fetch_assoc_prepared('SELECT DISTINCT table_name
FROM plugin_slowlog_details_tables
WHERE logid = ?',
Expand Down Expand Up @@ -837,7 +856,7 @@ function slowlog_classify_other_tables_against_list(int $logid, array $reference
$method_rows = [];

foreach ($rows as $row) {
$method_rows[] = [$logid, $row['logentry'], $methodid];
$method_rows[] = [$logid, $row['logentry'], SLOWLOG_METHOD_OTHER_TABLES];
}

slowlog_bulk_insert_method_rows($method_rows);
Expand Down Expand Up @@ -1358,23 +1377,7 @@ function import_post_process(int $logid, string $table_names = '', bool $usecact
* default too), instead of one LIKE/NOT LIKE table scan per method (~20 round trips
* previously for the default method dictionary).
*/
$methods = db_fetch_assoc_prepared('SELECT *
FROM plugin_slowlog_methods
ORDER BY method',
[]);

$method_fragments = [];
$others_methodid = null;

foreach ($methods as $row) {
// OTHERS is the "matched nothing else" bucket; OTHER TABLES is classified
// separately below (by table recognition, not a query text fragment).
if ($row['method'] == 'OTHERS') {
$others_methodid = $row['methodid'];
} elseif ($row['method'] != 'OTHER TABLES') {
$method_fragments[$row['methodid']] = explode(',', $row['query']);
}
}
$method_fragments = SLOWLOG_METHOD_FRAGMENTS;

$method_chunk_size = 2000;
$last_logentry = 0;
Expand All @@ -1395,19 +1398,19 @@ function import_post_process(int $logid, string $table_names = '', bool $usecact
$last_logentry = $row['logentry'];
$matched = false;

foreach ($method_fragments as $methodid => $fragments) {
foreach ($method_fragments as $method => $fragments) {
foreach ($fragments as $fragment) {
if (stripos($row['query'], $fragment) !== false) {
$method_rows[] = [$logid, $row['logentry'], $methodid];
$method_rows[] = [$logid, $row['logentry'], $method];
$matched = true;

break;
}
}
}

if (!$matched && $others_methodid !== null) {
$method_rows[] = [$logid, $row['logentry'], $others_methodid];
if (!$matched) {
$method_rows[] = [$logid, $row['logentry'], SLOWLOG_METHOD_OTHERS];
}
}

Expand Down Expand Up @@ -1447,11 +1450,6 @@ function import_post_process(int $logid, string $table_names = '', bool $usecact

if ($total_tables > 0) {
foreach ($tables as $t) {
db_execute_prepared('INSERT INTO plugin_slowlog_tables
(logid, table_name)
VALUES (?, ?)',
[$logid, $t]);

db_execute_prepared('INSERT INTO plugin_slowlog_details_tables (logid, logentry, table_name)
SELECT ? AS logid, logentry, ? AS table_name
FROM plugin_slowlog_details
Expand Down Expand Up @@ -1532,7 +1530,6 @@ function import_post_process(int $logid, string $table_names = '', bool $usecact
function slowlog_reprocess(int $logid, string $table_names = '', bool $usecacti = false, ?string $table_mode = null): void {
db_execute_prepared('DELETE FROM plugin_slowlog_details_methods WHERE logid = ?', [$logid]);
db_execute_prepared('DELETE FROM plugin_slowlog_details_tables WHERE logid = ?', [$logid]);
db_execute_prepared('DELETE FROM plugin_slowlog_tables WHERE logid = ?', [$logid]);
db_execute_prepared('DELETE FROM plugin_slowlog_stats WHERE logid = ?', [$logid]);
db_execute_prepared('UPDATE plugin_slowlog_details SET timeout = 0 WHERE logid = ?', [$logid]);

Expand Down Expand Up @@ -2894,16 +2891,14 @@ function slowlog_get_chart_object_live(int $logid, string $scope, string $measur
$agg = ($measure == 'count') ? 'COUNT(*)' : "SUM($measure)";

if ($scope == 'method') {
return db_fetch_assoc_prepared("SELECT sm.method AS scope_key,
return db_fetch_assoc_prepared("SELECT dm.method AS scope_key,
$agg AS value
FROM plugin_slowlog_details_methods AS dm
INNER JOIN plugin_slowlog_details AS d
ON dm.logid = d.logid
AND dm.logentry = d.logentry
INNER JOIN plugin_slowlog_methods AS sm
ON dm.methodid = sm.methodid
WHERE d.logid = ?
GROUP BY sm.methodid
GROUP BY dm.method
ORDER BY value DESC" . $limit,
[$logid]);
} else {
Expand Down Expand Up @@ -3092,11 +3087,10 @@ function slowlog_get_chart_scope_items(string $chart_type, int $id): array {

$scope_items[] = slowlog_others_bucket_key($id);
} else {
$scope_items = array_column(db_fetch_assoc_prepared('SELECT DISTINCT sm.method AS value
$scope_items = array_column(db_fetch_assoc_prepared('SELECT DISTINCT dm.method AS value
FROM plugin_slowlog_details_methods AS dm
INNER JOIN plugin_slowlog_methods AS sm ON sm.methodid = dm.methodid
WHERE dm.logid = ?
ORDER BY sm.method',
ORDER BY dm.method',
[$id]), 'value');
}

Expand Down
5 changes: 5 additions & 0 deletions setup.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -186,6 +186,11 @@ function slowlog_check_upgrade(): void {
return;
}

// Runs on every load (idempotent, self-gating on the methodid column) because the 2.6
// methodid -> method de-normalization ships without a version bump, so the drift branch
// below would otherwise never fix an existing plugin_slowlog_details_methods table.
slowlog_migrate_details_methods_to_method();

$current = $info['version'];
$old = db_fetch_cell_prepared('SELECT version
FROM plugin_config
Expand Down
Loading
Loading