push/scrub secret gate never converges on base64 (Aiven) false positives

[dacorvo]

Summary

funes push and funes scrub disagree about which rows contain secrets, so the intended
push → scrub → push remediation loop never converges for high-entropy base64 blobs (inline
images) that trufflehog's Aiven detector fires on as a false positive. Each scrub redacts a
few blocks and each push ships a handful of freshly-cleaned chunks, but the same ~3.3k rows are
held back on every push.

Reproduction (observed on dacorvo/funes-memory)

$ funes push dacorvo/funes-memory
scanning 13177 chunk(s) for secrets…
uploading 9809 chunk(s) …
⚠ held back 3368 row(s) containing secrets (Aiven×3) — run `funes scrub` …

$ funes scrub
scanning 26865 block(s) for secrets…
re-embedding 14891 redacted chunk(s)…
scrubbed 84552 rows: redacted 8 secret(s) in 6 block(s)

$ funes push dacorvo/funes-memory
scanning 3365 chunk(s) for secrets…
nothing published — held back 3365 row(s) with secrets (Aiven×3); run `funes scrub` …

$ funes scrub
re-embedding 7178 redacted chunk(s)…
scrubbed 84548 rows: redacted 4 secret(s) in 4 block(s)

$ funes push dacorvo/funes-memory
scanning 3365 chunk(s) for secrets…
uploading 10 chunk(s) …
⚠ held back 3355 row(s) containing secrets (Aiven×2) …

Held-back count drifts 3368 → 3365 → 3355, always Aiven, pushing only 3–10 chunks per round —
it never reaches zero.

Root cause

Confirmed from the code:

• Both paths share the scanner and scan::scan_blocks, which does
  trufflehog(texts.join("\n")) — it joins all input block texts into one blob and scans once,
  mapping findings back by line number.
• But they feed it different inputs: push (drop_secret_rows, src/push.rs) joins only the
  delta (local rows not yet on the remote); scrub (src/scrub.rs) joins the entire local
  memory. Different blobs → trufflehog can return a different Aiven set.
• Hold-back is block-granularity: any dirty chunk marks the whole block. The held-back blocks
  here are giant base64 blobs, so Aiven×2 (2 blocks) ≈ 3355 chunk-rows.
• scrub only removes a block if excise reports fully_redacted (or drops it whole, which it
  didn't here — no "dropped" clause), so the blocks it flags get redacted while the blocks push
  flags do not.
• There is no allowlist wired in (the scan.rs module comments reference one, but I don't see
  it implemented), so there is no way to exempt a known false positive.

Likely (not yet directly reproduced): trufflehog returns a different Aiven set on the delta
blob (push flags 2–3 blocks) vs the full-memory blob (scrub redacts 6 disjoint blocks), so scrub
never touches the specific blocks push holds back. This is the concrete mechanism to confirm.

Underlying: Aiven is a false positive — the flagged rows are inline base64 PNGs (largely
one codex session); trufflehog's Aiven detector matches high-entropy base64. There is no real secret.

Why the log numbers look inconsistent (not a bug, just different denominators)

• push scanning N chunk(s) = delta chunk-rows; held back M row(s) = chunk-rows (block-granular).
• scrub scanning B block(s) = all blocks; scrubbed T rows = all chunk-rows (T/B ≈ 3.1, multi-tier).
• re-embedding 14891 for 6 blocks = those blocks are huge base64, re-chunking yields ~14891 rows.

Proposed fixes (any subset)

1. Suppress the false positive — an allowlist / entropy+size guard so base64 image blobs don't
   trip the secret gate (or downweight Aiven/unverified detectors on obvious base64).
2. Make push and scrub scan identically so scrub → push is guaranteed to converge (e.g. both
   scan per-block, or both over the same row set — don't join a delta differently than the full set).
3. Consider not storing/redacting giant inline-image blobs at all (they aren't useful recall content).

Code pointers

• src/push.rs — drop_secret_rows (the push gate).
• src/scrub.rs — run (redaction).
• src/scan.rs — scan_blocks (join-and-scan-once), excise.