feat(server): add plikd migrate command for live backend-to-backend migration

AGENTS.md

@@ -113,6 +113,9 @@ make vuln                   # govulncheck (report only)
 | `webapp/src/components/UploadControls.vue` | Shared sort/order/badge-filter control bar used by AdminView and HomeView |
 | `server/common/feature_flags.go` | Feature flag types (`disabled`/`enabled`/`default`/`forced`) |
 | `server/server/server.go` | `ensureDefaultAdmin()` — idempotent bootstrap of `DefaultAdminLogin`/`DefaultAdminPassword` on startup |
+| `server/cmd/migrate.go` | `plikd migrate` cobra command — orchestrates backend-to-backend migration |
+| `server/metadata/migrator.go` | `Migrate(src, dst, opts)` — streams all metadata from one backend to another |
+| `server/data/migrator.go` | `MigrateFiles(src, dst, meta, opts)` — parallel file blob copy worker pool |
 
 ## Conventions
 

docs/.vitepress/config.js

@@ -74,6 +74,7 @@ export default withMermaid(defineConfig({
                         { text: 'Reverse Proxy', link: '/operations/reverse-proxy' },
                         { text: 'Server CLI', link: '/operations/server-cli' },
                         { text: 'Import / Export', link: '/operations/import-export' },
+                        { text: 'Migration', link: '/operations/migration' },
                         { text: 'Cross Compilation', link: '/operations/cross-compilation' },
                     ],
                 },

docs/backends/metadata.md

@@ -62,4 +62,7 @@ Plik uses [gormigrate](https://github.com/go-gormigrate/gormigrate) for automati
 
 ## Migrating Between Backends
 
-To migrate data between different metadata backends (e.g. SQLite → PostgreSQL), use the `plikd export` and `plikd import` commands. See the [Import / Export](/operations/import-export) guide for details.
+Plik provides two approaches for migrating between metadata backends:
+
+- **`plikd export` / `plikd import`** — Export metadata to a portable file and import it into a new backend. Good for offline backups or scheduled migrations. See the [Import / Export](/operations/import-export) guide.
+- **`plikd migrate`** — Live, direct backend-to-backend migration that also handles file data blobs in parallel. No intermediate files needed. See the [Migration](/operations/migration) guide.

docs/operations/migration.md

@@ -0,0 +1,120 @@
+# Migration Guide
+
+Plik supports live, direct backend-to-backend migration via the `plikd migrate` command. This lets you move all data and/or metadata from one backend configuration to another **without creating intermediate archive files**. File data is streamed directly from the source backend to the target backend.
+
+## Overview
+
+The `plikd migrate` command:
+
+- Reads the source backends from the current `plikd.cfg` (same config used to start the server)
+- Reads the **target** backends from a second config file (`--to` flag)
+- Migrates metadata (users, tokens, uploads, files, settings) in order, respecting foreign key dependencies
+- Migrates file blobs in parallel (configurable workers, default: 4)
+- Supports dry-run mode, error tolerance, and selective migration
+
+> [!IMPORTANT]
+> Stop the Plik server before running `plikd migrate`. Running a migration against a live server may produce inconsistent results.
+
+## Basic Syntax
+
+```sh
+plikd migrate --to /path/to/target.cfg [options]
+```
+
+| Flag | Default | Description |
+|------|---------|-------------|
+| `--to` | *(required)* | Path to the target `plikd.cfg` |
+| `--metadata-only` | false | Only migrate metadata (skip file blobs) |
+| `--data-only` | false | Only migrate file blobs (skip metadata) |
+| `--workers` | 4 | Number of parallel file copy workers |
+| `--dry-run` | false | Print every item that would be migrated, write nothing |
+| `--ignore-errors` | false | Log errors per item and continue (don't abort) |
+
+## Common Scenarios
+
+### 1. SQLite → PostgreSQL (metadata only)
+
+Your current `plikd.cfg`:
+```toml
+MetadataBackend = "sqlite3"
+[MetadataBackendConfig]
+  ConnectionString = "/home/plik/server/db/plik.db"
+```
+
+Create a `plikd-pg.cfg` with just the fields that differ:
+```toml
+MetadataBackend = "postgres"
+[MetadataBackendConfig]
+  ConnectionString = "host=localhost user=plik password=secret dbname=plik sslmode=disable"
+```
+
+Run:
+```sh
+plikd migrate --to plikd-pg.cfg --metadata-only
+```
+
+### 2. Local Files → S3 (data only)
+
+You already have PostgreSQL running. Create `plikd-s3.cfg`:
+```toml
+DataBackend = "s3"
+[DataBackendConfig]
+  Bucket     = "my-plik-bucket"
+  Region     = "us-east-1"
+  # Credentials via AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY env vars
+```
+
+Run:
+```sh
+plikd migrate --to plikd-s3.cfg --data-only --workers 8
+```
+
+### 3. Full Migration (SQLite + Local → PostgreSQL + S3)
+
+1. Prepare `plikd-new.cfg` with both new backends configured.
+2. First preview with dry-run:
+   ```sh
+   plikd migrate --to plikd-new.cfg --dry-run
+   ```
+3. Run the full migration:
+   ```sh
+   plikd migrate --to plikd-new.cfg
+   ```
+4. Update your main `plikd.cfg` to use the new backends and start the server.
+
+### 4. Resuming a Failed Migration
+
+If a migration is interrupted, re-run with `--ignore-errors` to skip already-migrated records (unique constraint violations) and continue:
+```sh
+plikd migrate --to plikd-new.cfg --ignore-errors
+```
+
+> [!WARNING]
+> With `--ignore-errors`, migration errors are silently skipped and logged to stdout. Always review the output to ensure no critical data was lost.
+
+## Migration Order
+
+Metadata is migrated in dependency order to avoid foreign key violations:
+
+1. **Users**
+2. **Tokens** (depend on users)
+3. **Uploads** (including soft-deleted, to maintain FK integrity with files)
+4. **Files** (metadata records)
+5. **Settings**
+
+File **data blobs** are migrated after metadata, in parallel. Files with status `uploaded` or `removed` are copied (both still exist in the data backend). Files with status `missing`, `uploading`, or `deleted` are skipped.
+
+## Verifying After Migration
+
+After migration, verify the target by querying the HTTP API (with the server running against the new config):
+
+```sh
+# Check server stats via API
+curl http://localhost:8080/version
+```
+
+Or query record counts directly in your database. For example with SQLite:
+
+```sh
+sqlite3 /path/to/new/plik.db "SELECT COUNT(*) FROM uploads; SELECT COUNT(*) FROM files;"
+```

server/ARCHITECTURE.md

@@ -35,6 +35,7 @@ The server binary `plikd` uses [cobra](https://github.com/spf13/cobra) for CLI m
 | `clean.go` | `plikd clean` | Run metadata cleanup |
 | `import.go` | `plikd import [input-file]` | Import metadata from gob + Snappy binary |
 | `export.go` | `plikd export [output-file]` | Export metadata to gob + Snappy binary |
+| `migrate.go` | `plikd migrate --to <cfg>` | Live backend-to-backend migration (metadata + data) |
 
 Config loading order: `--config` flag → `PLIKD_CONFIG` env → `./plikd.cfg` → `/etc/plikd.cfg`.
 
@@ -143,6 +144,7 @@ Uses GORM with gormigrate for schema management across SQLite3, PostgreSQL, and
 | `stats.go` | Aggregate statistics queries |
 | `exporter.go` | gob + Snappy export of all metadata |
 | `importer.go` | gob + Snappy import |
+| `migrator.go` | Live metadata migration: `Migrate(src, dst, opts)` — streams all records using `ForEach*`/`Create*`, supports `--ignore-errors` |
 
 ### Import / Export
 
@@ -151,10 +153,19 @@ The `plikd export` and `plikd import` commands dump and restore all metadata (us
 - **Format**: Go [gob](https://pkg.go.dev/encoding/gob) encoding compressed with [Snappy](https://github.com/golang/snappy). Architecture-independent (portable across `amd64`/`arm64`), streaming (constant memory), Go-specific (not human-readable).
 - **Export order**: users → tokens → uploads (including soft-deleted) → files → settings. CLI auth sessions are intentionally excluded (ephemeral).
 - **Import**: decodes sequentially, calls `Create*` on the metadata backend. Supports `--ignore-errors` to skip problematic records.
-- **Use cases**: backend migration (e.g. SQLite → PostgreSQL), backups, disaster recovery.
+- **Use cases**: backend backups, disaster recovery.
 
 > **Note**: Only metadata is exported — file data in the data backend must be migrated separately.
 
+### Live Migration (`plikd migrate`)
+
+The `plikd migrate --to <target.cfg>` command performs a **direct, live backend-to-backend migration** of both metadata and file data. No intermediate files are created.
+
+- **Metadata** (`metadata/migrator.go`): Streaming `ForEach*` + `Create*` pipeline, same migration order as export/import. CLI auth sessions are excluded (ephemeral).
+- **File data** (`data/migrator.go`): Parallel worker pool (`--workers`, default: 4) reading from `src.GetFile()` and writing to `dst.AddFile()`. Copies files with status `uploaded` or `removed` (both still exist in storage); skips `missing`, `uploading`, `deleted`.
+- **Dry-run**: Lists every individual item that would be migrated, writes nothing.
+- **Target config**: Full `plikd.cfg` via `--to`; only the backend fields need to differ from the source config.
+
 ### Migration Dump Tests
 
 When adding a new migration, `TestGenerateSQLDump` and `TestGenerateExport` in `migrations_test.go` auto-generate dump files on first run. Each dump captures the full DB state after all migrations and test data have been applied. `TestMigrationsFromSQLDumps` and `TestLoadExports` then load **all** existing dumps and verify migrations can be replayed forward to the current schema.

server/cmd/migrate.go

@@ -0,0 +1,194 @@
+package cmd
+
+import (
+	"fmt"
+	"os"
+	"time"
+
+	"github.com/spf13/cobra"
+
+	"github.com/root-gg/plik/server/common"
+	"github.com/root-gg/plik/server/data"
+	"github.com/root-gg/plik/server/metadata"
+	"github.com/root-gg/plik/server/server"
+)
+
+type migrateFlagParams struct {
+	to           string
+	dataOnly     bool
+	metadataOnly bool
+	ignoreErrors bool
+	workers      int
+	dryRun       bool
+}
+
+var migrateParams = migrateFlagParams{}
+
+// migrateCmd migrates metadata and/or data from one backend to another
+var migrateCmd = &cobra.Command{
+	Use:   "migrate",
+	Short: "Migrate metadata and/or data backend",
+	Long: `Migrate metadata (users, tokens, uploads, files, settings) and file data
+from the configured source backends to the backends configured in the target config file.
+
+Both the source plikd.cfg (--config flag or default location) and the target plikd.cfg
+(--to flag) are required. The target config only needs to specify the backend fields
+that differ from the source.
+
+Typical use cases:
+  - SQLite → PostgreSQL (metadata only):  plikd migrate --to new.cfg --metadata-only
+  - Local files → S3 (data only):        plikd migrate --to new.cfg --data-only
+  - Full migration:                       plikd migrate --to new.cfg`,
+	Example: `  plikd migrate --to /etc/plikd-new.cfg
+  plikd migrate --to /etc/plikd-new.cfg --metadata-only
+  plikd migrate --to /etc/plikd-new.cfg --data-only --workers 8
+  plikd migrate --to /etc/plikd-new.cfg --dry-run
+  plikd migrate --to /etc/plikd-new.cfg --ignore-errors`,
+	Run: migrateBackends,
+}
+
+func init() {
+	migrateCmd.Flags().StringVar(&migrateParams.to, "to", "", "path to target plikd.cfg (required)")
+	migrateCmd.Flags().BoolVar(&migrateParams.dataOnly, "data-only", false, "migrate file data only (skip metadata)")
+	migrateCmd.Flags().BoolVar(&migrateParams.metadataOnly, "metadata-only", false, "migrate metadata only (skip file data)")
+	migrateCmd.Flags().BoolVar(&migrateParams.ignoreErrors, "ignore-errors", false, "continue on individual record/file errors")
+	migrateCmd.Flags().IntVar(&migrateParams.workers, "workers", 4, "number of parallel file copy workers")
+	migrateCmd.Flags().BoolVar(&migrateParams.dryRun, "dry-run", false, "print what would be migrated without writing anything")
+	_ = migrateCmd.MarkFlagRequired("to")
+	rootCmd.AddCommand(migrateCmd)
+}
+
+func migrateBackends(cmd *cobra.Command, args []string) {
+	if migrateParams.dataOnly && migrateParams.metadataOnly {
+		fmt.Println("--data-only and --metadata-only are mutually exclusive")
+		os.Exit(1)
+	}
+
+	start := time.Now()
+
+	if migrateParams.dryRun {
+		fmt.Println("[dry-run mode — nothing will be written]")
+	}
+
+	// Load target config
+	targetConfig, err := common.LoadConfiguration(migrateParams.to)
+	if err != nil {
+		fmt.Printf("unable to load target config %s: %s\n", migrateParams.to, err)
+		os.Exit(1)
+	}
+
+	// Initialize source backends (from primary config, already loaded by initConfig)
+	initializeMetadataBackend()
+	srcMeta := metadataBackend
+
+	// Initialize target metadata backend
+	var dstMeta *metadata.Backend
+	if !migrateParams.dataOnly {
+		dstMeta, err = server.NewMetadataBackend(targetConfig.MetadataBackendConfig, targetConfig.NewLogger())
+		if err != nil {
+			fmt.Printf("unable to initialize target metadata backend: %s\n", err)
+			os.Exit(1)
+		}
+
+		fmt.Printf("Migrating metadata: %s → %s\n",
+			config.MetadataBackendConfig["Driver"],
+			targetConfig.MetadataBackendConfig["Driver"])
+
+		if !migrateParams.dryRun {
+			metaOpts := &metadata.MigrateOptions{
+				IgnoreErrors: migrateParams.ignoreErrors,
+			}
+			metaStats, err := metadata.Migrate(srcMeta, dstMeta, metaOpts)
+			if err != nil {
+				fmt.Printf("metadata migration failed: %s\n", err)
+				os.Exit(1)
+			}
+			totalMetaErrors := metaStats.UserErrors + metaStats.TokenErrors +
+				metaStats.UploadErrors + metaStats.FileErrors + metaStats.SettingErrors
+			fmt.Printf("metadata migration complete: %d users, %d tokens, %d uploads, %d files, %d settings (%d errors)\n",
+				metaStats.Users, metaStats.Tokens, metaStats.Uploads, metaStats.Files, metaStats.Settings,
+				totalMetaErrors)
+		} else {
+			// Dry-run: enumerate and print all metadata items that would be migrated
+			fmt.Println("[dry-run] would migrate:")
+			var dryRunErr error
+			dryRunErr = srcMeta.ForEachUsers(func(u *common.User) error {
+				fmt.Printf("  user     %s (%s)\n", u.ID, u.Login)
+				return nil
+			})
+			if dryRunErr != nil {
+				fmt.Printf("error enumerating users: %s\n", dryRunErr)
+				os.Exit(1)
+			}
+			dryRunErr = srcMeta.ForEachToken(func(t *common.Token) error {
+				fmt.Printf("  token    %s (user: %s)\n", t.Token, t.UserID)
+				return nil
+			})
+			if dryRunErr != nil {
+				fmt.Printf("error enumerating tokens: %s\n", dryRunErr)
+				os.Exit(1)
+			}
+			dryRunErr = srcMeta.ForEachUploadUnscoped(func(u *common.Upload) error {
+				fmt.Printf("  upload   %s\n", u.ID)
+				return nil
+			})
+			if dryRunErr != nil {
+				fmt.Printf("error enumerating uploads: %s\n", dryRunErr)
+				os.Exit(1)
+			}
+			dryRunErr = srcMeta.ForEachFile(func(f *common.File) error {
+				fmt.Printf("  file     %s (%s, %s)\n", f.ID, f.Name, f.Status)
+				return nil
+			})
+			if dryRunErr != nil {
+				fmt.Printf("error enumerating files: %s\n", dryRunErr)
+				os.Exit(1)
+			}
+			dryRunErr = srcMeta.ForEachSetting(func(s *common.Setting) error {
+				fmt.Printf("  setting  %s\n", s.Key)
+				return nil
+			})
+			if dryRunErr != nil {
+				fmt.Printf("error enumerating settings: %s\n", dryRunErr)
+				os.Exit(1)
+			}
+		}
+	}
+
+	// Migrate data
+	if !migrateParams.metadataOnly {
+		initializeDataBackend()
+		srcData := dataBackend
+
+		dstDataBackend, err := server.NewDataBackend(targetConfig.DataBackend, targetConfig.DataBackendConfig)
+		if err != nil {
+			fmt.Printf("unable to initialize target data backend: %s\n", err)
+			os.Exit(1)
+		}
+
+		fmt.Printf("Migrating file data: %s → %s\n", config.DataBackend, targetConfig.DataBackend)
+
+		dataOpts := &data.MigrateOptions{
+			IgnoreErrors: migrateParams.ignoreErrors,
+			Workers:      migrateParams.workers,
+			DryRun:       migrateParams.dryRun,
+		}
+
+		// Use source metadata to enumerate files (even if we just migrated metadata,
+		// the source metadata backend is always the canonical file list)
+		dataStats, err := data.MigrateFiles(srcData, dstDataBackend, srcMeta, dataOpts)
+		if err != nil {
+			fmt.Printf("data migration failed: %s\n", err)
+			os.Exit(1)
+		}
+
+		prefix := ""
+		if migrateParams.dryRun {
+			prefix = "[dry-run] "
+		}
+		fmt.Printf("%sdata migration complete: %d files copied, %d skipped, %d errors, %d bytes\n",
+			prefix, dataStats.Copied, dataStats.Skipped, dataStats.Errors, dataStats.Bytes)
+	}
+
+	fmt.Printf("migration completed in %s\n", time.Since(start).Round(time.Millisecond))
+}